标题： 解耦后再对齐：视觉解缠增强提示调优 显示英文标题

标题： Decouple before Align: Visual Disentanglement Enhances Prompt Tuning

摘要： 提示微调（PT），作为一种新兴的资源高效微调范式，已展示了在提高视觉-语言模型任务特定迁移能力方面的显著效果。 本文深入研究了PT中一个之前被忽视的信息不对称问题，其中视觉模态通常比面向对象的文本模态传达更多上下文。 相应地，粗略对齐这两种模态可能导致偏差注意力，使模型仅关注上下文区域。 为了解决这个问题，我们提出了DAPT，一种基于直观的“解耦后再对齐”概念的有效PT框架。 首先，我们提出通过利用粗粒度和细粒度的视觉分割线索，显式地将视觉模态分解为前景和背景表示，然后将这些解耦模式与原始前景文本和手工设计的背景类别对齐，从而对称地增强模态对齐。 为了进一步增强视觉聚焦，我们提出了一种针对前景-背景模式的视觉拉推正则化，引导原始视觉表示朝着对感兴趣区域对象的无偏注意。 我们通过少样本学习、从基础到新颖的泛化以及数据高效学习展示了无架构DAPT的强大性能，所有这些都在主流基准上取得了优越的表现。 我们的代码将在https://github.com/Ferenas/DAPT发布。 显示英文摘要

摘要： Prompt tuning (PT), as an emerging resource-efficient fine-tuning paradigm, has showcased remarkable effectiveness in improving the task-specific transferability of vision-language models. This paper delves into a previously overlooked information asymmetry issue in PT, where the visual modality mostly conveys more context than the object-oriented textual modality. Correspondingly, coarsely aligning these two modalities could result in the biased attention, driving the model to merely focus on the context area. To address this, we propose DAPT, an effective PT framework based on an intuitive decouple-before-align concept. First, we propose to explicitly decouple the visual modality into the foreground and background representation via exploiting coarse-and-fine visual segmenting cues, and then both of these decoupled patterns are aligned with the original foreground texts and the hand-crafted background classes, thereby symmetrically strengthening the modal alignment. To further enhance the visual concentration, we propose a visual pull-push regularization tailored for the foreground-background patterns, directing the original visual representation towards unbiased attention on the region-of-interest object. We demonstrate the power of architecture-free DAPT through few-shot learning, base-to-novel generalization, and data-efficient learning, all of which yield superior performance across prevailing benchmarks. Our code will be released at https://github.com/Ferenas/DAPT.