标题： 治愈还是毒药？ 嵌入指令通过视觉影响视觉-语言模型中的幻觉 显示英文标题

标题： Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Models

摘要： 视觉-语言模型（VLMs）常常出现幻觉，部分原因是对多模态信息对齐的挑战。我们提出了图像内提示，这是一种简单的方法，将文本指令直接嵌入到图像中。这消除了对单独文本输入的需求，并迫使模型通过视觉通道处理所有内容。我们在三个流行的开源VLM上评估了这种方法：Qwen2.5-VL、LLaVA-1.5和InstructBLIP。结果显示出明显的差异。图像内提示提高了Qwen2.5-VL的性能，POPE准确率提高了4.1个百分点（从80.2个百分点提高到84.3个百分点），并在MS-COCO上减少了幻觉率。相反，LLaVA-1.5和InstructBLIP的性能严重下降，准确率从大约84个百分点降至接近随机水平。通过详细分析，我们发现LLaVA和InstructBLIP中的基于CLIP的编码器对嵌入文本区域表现出过度的关注偏差，破坏了视觉理解。相比之下，Qwen的视觉编码器能够稳健地处理嵌入文本的图像。至关重要的是，图像内提示减少了Qwen的模态差距，通过单一模态统一信息处理来增强跨模态对齐。 显示英文摘要

摘要： Vision-Language Models (VLMs) often suffer from hallucination, partly due to challenges in aligning multimodal information. We propose Prompt-in-Image, a simple method that embeds textual instructions directly into images. This removes the need for separate text inputs and forces the model to process all content through the visual channel. We evaluate this method on three popular open-source VLMs: Qwen2.5-VL, LLaVA-1.5, and InstructBLIP. The results reveal sharp differences. Prompt-in-Image improves Qwen2.5-VL's performance, increasing POPE accuracy by 4.1 percent (from 80.2 percent to 84.3 percent) and also reducing hallucination rates on MS-COCO. In contrast, LLaVA-1.5 and InstructBLIP experience a severe performance drop, with accuracy falling from around 84 percent to near-random levels. Through detailed analysis, we found that CLIP-based encoders in LLaVA and InstructBLIP exhibit excessive attention bias toward embedded text regions, disrupting visual understanding. In contrast, Qwen's vision encoder handles text-embedded images robustly. Crucially, Prompt-in-Image reduces Qwen's modality gap, enhancing cross-modal alignment by unifying information processing through a single modality.