标题： 嵌入中的迷失：视觉-语言模型中的信息丢失 显示英文标题

标题： Lost in Embeddings: Information Loss in Vision-Language Models

摘要： 视觉-语言模型（VLMs）通常通过预训练的视觉编码器处理视觉输入，然后通过连接组件将输入投影到语言模型的嵌入空间中。 尽管对于模态融合至关重要，但这一投影步骤引起的潜在信息丢失及其对模型能力的直接影响仍研究不足。 我们引入了两种互补的方法，通过分析潜在表示空间来检验和量化这种损失。 首先，我们通过分析图像表示在投影前后之间的k最近邻关系变化来评估语义信息的保留情况。 其次，我们通过从投影表示中重建视觉嵌入来直接测量信息丢失，并在图像块级别定位损失。 实验表明，连接组件显著扭曲了视觉表示的局部几何结构，投影后k最近邻的差异达到40%-60%，这与检索性能的下降相关。 在图像块级别的嵌入重建为模型在视觉基础问答任务中的行为提供了可解释的见解，发现高信息丢失区域能够可靠地预测模型遇到困难的实例。 显示英文摘要

摘要： Vision--language models (VLMs) often process visual inputs through a pretrained vision encoder, followed by a projection into the language model's embedding space via a connector component. While crucial for modality fusion, the potential information loss induced by this projection step and its direct impact on model capabilities remain understudied. We introduce two complementary approaches to examine and quantify this loss by analyzing the latent representation space. First, we evaluate semantic information preservation by analyzing changes in k-nearest neighbor relationships between image representations, before and after projection. Second, we directly measure information loss by reconstructing visual embeddings from the projected representation, localizing loss at an image patch level. Experiments reveal that connectors substantially distort the local geometry of visual representations, with k-nearest neighbors diverging by 40--60\% post-projection, correlating with degradation in retrieval performance. The patch-level embedding reconstruction provides interpretable insights for model behavior on visually grounded question-answering tasks, finding that areas of high information loss reliably predict instances where models struggle.