标题： 视频-LLaMA：用于视频理解的指令调优的视听语言模型 显示英文标题

标题： Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

摘要： 我们提出了Video-LLaMA，这是一个多模态框架，使大型语言模型（LLMs）具备理解视频中视觉和听觉内容的能力。Video-LLaMA从冻结的预训练视觉和音频编码器以及冻结的LLMs中进行跨模态训练。与之前仅补充LLMs来处理视觉或音频信号的工作不同，Video-LLaMA通过解决两个挑战来实现视频理解：(1) 捕捉视觉场景的时间变化，(2) 整合视听信号。为应对第一个挑战，我们提出了一种Video Q-former，将预训练图像编码器整合到我们的视频编码器中，并引入了视频到文本生成任务以学习视频与语言的对应关系。对于第二个挑战，我们利用ImageBind，一个对齐多种模态的通用嵌入模型，作为预训练音频编码器，并在ImageBind之上引入了一个Audio Q-former，以学习合理的听觉查询嵌入用于LLM模块。为了将视觉和音频编码器的输出与LLM的嵌入空间对齐，我们首先在大量视频/图像-标题对上训练Video-LLaMA，然后使用中等数量但质量更高的视觉指令数据集来微调我们的模型。我们发现Video-LLaMA表现出感知和理解视频内容的能力，并能基于视频中呈现的视觉和听觉信息生成有意义的响应。 显示英文摘要

摘要： We present Video-LLaMA a multi-modal framework that empowers Large Language Models (LLMs) with the capability of understanding both visual and auditory content in the video. Video-LLaMA bootstraps cross-modal training from the frozen pre-trained visual and audio encoders and the frozen LLMs. Unlike previous works that complement LLMs to process the visual or audio signals only, Video-LLaMA enables video comprehension by tackling two challenges: (1) capturing the temporal changes in visual scenes, (2) integrating audio-visual signals. To counter the first challenge, we propose a Video Q-former to assemble a pre-trained image encoder into our video encoder and introduce a video-to-text generation task to learn video-language correspondence. For the second challenge, we leverage ImageBind, a universal embedding model aligning multiple modalities, as the pre-trained audio encoder and introduce an Audio Q-former on top of ImageBind to learn reasonable auditory query embeddings for the LLM module. To align the output of both visual and audio encoders with LLM's embedding space, we first train Video-LLaMA on massive video/image-caption pairs and then tune our model with visual-instruction datasets of moderate amount but higher quality. We found Video-LLaMA shows the ability to perceive and comprehend video content and generate meaningful responses grounded in the visual and auditory information presented in the videos.