标题： BeAts：使用多模态注意力融合的孟加拉语言语行为识别 显示英文标题

标题： BeAts: Bengali Speech Acts Recognition using Multimodal Attention Fusion

摘要： 口语语言常常利用语调、节奏、强度和结构来传达意图，而这些意图会根据话语的节奏不同而有不同的解释。 这些言语行为构成了交流的基础，并在表达上因语言而异。 最近基于注意力的模型取得了进展，显示出它们从多语言数据集中学习强大表示的能力，在语音任务中表现良好，非常适合在资源较少的语言中建模特定任务。 在这里，我们开发了一种新颖的多模态方法，结合了wav2vec2.0音频模型和MarianMT文本翻译模型，通过多模态注意力融合来预测我们准备的孟加拉语语音语料库中的言语行为。 我们还表明，我们的模型BeAts（$\underline{\textbf{Be}}$ngali speech acts recognition using Multimodal$\underline{\textbf{At}}$tention Fu$\underline{\textbf{s}}$ion）显著优于仅使用语音数据的单模态基线以及使用语音和文本数据的简单双模态融合。 项目页面：https://soumitri2001.github.io/BeAts 显示英文摘要

摘要： Spoken languages often utilise intonation, rhythm, intensity, and structure, to communicate intention, which can be interpreted differently depending on the rhythm of speech of their utterance. These speech acts provide the foundation of communication and are unique in expression to the language. Recent advancements in attention-based models, demonstrating their ability to learn powerful representations from multilingual datasets, have performed well in speech tasks and are ideal to model specific tasks in low resource languages. Here, we develop a novel multimodal approach combining two models, wav2vec2.0 for audio and MarianMT for text translation, by using multimodal attention fusion to predict speech acts in our prepared Bengali speech corpus. We also show that our model BeAts ($\underline{\textbf{Be}}$ngali speech acts recognition using Multimodal $\underline{\textbf{At}}$tention Fu$\underline{\textbf{s}}$ion) significantly outperforms both the unimodal baseline using only speech data and a simpler bimodal fusion using both speech and text data. Project page: https://soumitri2001.github.io/BeAts