标题： 面向包容性交流：从手语、口型和音频生成口语语言的统一框架 显示英文标题

标题： Towards Inclusive Communication: A Unified Framework for Generating Spoken Language from Sign, Lip, and Audio

摘要： 音频是人类交流的主要方式，并推动了自动语音识别（ASR）技术的成功。 然而，这种以音频为中心的系统本质上排除了聋人或听力障碍的人。 视觉替代方案，如手语和读唇术，提供了有效的替代方案，近年来手语翻译（SLT）和视觉语音识别（VSR）的进步改善了无音频的交流。 然而，这些模式在很大程度上是在孤立的情况下研究的，它们在一个统一框架内的集成仍然研究不足。 在本文中，我们提出了第一个能够处理手语、唇部运动和音频的各种组合的统一框架，用于口语语言文本生成。 我们专注于三个主要目标：(i) 设计一个统一的、模态无关的架构，能够有效处理异构输入；(ii) 探索模态之间的未充分研究的协同作用，特别是唇部运动在手语理解中作为非手动线索的作用；以及(iii) 实现与专门针对单个任务的最先进模型相当或更优的性能。 基于这个框架，我们在SLT、VSR、ASR和视听语音识别方面实现了与特定任务的最先进模型相当或更好的性能。 此外，我们的分析揭示了一个关键的语言学见解：将唇部运动明确建模为一个独立的模态，通过捕捉关键的非手动线索，显著提高了SLT性能。 显示英文摘要

摘要： Audio is the primary modality for human communication and has driven the success of Automatic Speech Recognition (ASR) technologies. However, such audio-centric systems inherently exclude individuals who are deaf or hard of hearing. Visual alternatives such as sign language and lip reading offer effective substitutes, and recent advances in Sign Language Translation (SLT) and Visual Speech Recognition (VSR) have improved audio-less communication. Yet, these modalities have largely been studied in isolation, and their integration within a unified framework remains underexplored. In this paper, we propose the first unified framework capable of handling diverse combinations of sign language, lip movements, and audio for spoken-language text generation. We focus on three main objectives: (i) designing a unified, modality-agnostic architecture capable of effectively processing heterogeneous inputs; (ii) exploring the underexamined synergy among modalities, particularly the role of lip movements as non-manual cues in sign language comprehension; and (iii) achieving performance on par with or superior to state-of-the-art models specialized for individual tasks. Building on this framework, we achieve performance on par with or better than task-specific state-of-the-art models across SLT, VSR, ASR, and Audio-Visual Speech Recognition. Furthermore, our analysis reveals a key linguistic insight: explicitly modeling lip movements as a distinct modality significantly improves SLT performance by capturing critical non-manual cues.