标题： 冻结与揭示：揭示视觉-语言模型中的模态偏差 显示英文标题

标题： Freeze and Reveal: Exposing Modality Bias in Vision-Language Models

摘要： 视觉语言模型在多模态性能上表现出色，但常常从训练数据中继承性别偏见。 这种偏见可能来自视觉和文本模态两方面。 在本工作中，我们通过应用反事实数据增强和任务向量方法进行有针对性的去偏，来剖析视觉和文本主干对这些偏见的贡献。 受仇恨言论分类中数据高效方法的启发，我们引入了一个新指标，刻板印象程度，以及相应的去偏方法，使用刻板印象程度的数据增强 - DAUDoS，以最小的计算成本减少偏见。 我们整理了一个带有性别注释的数据集，并在VisoGender基准上评估所有方法，以量化改进并确定偏见的主要来源。 我们的结果表明，CDA将性别差距减少了6%，DAUDoS减少了3%，但仅使用了三分之一的数据。 这两种方法还使模型正确识别图像中性别的能力提高了3%，其中DAUDoS仅使用几乎三分之一的训练数据就实现了这一改进。 从我们的实验中，我们观察到CLIP的视觉编码器更具偏见，而PaliGemma2的文本编码器更具偏见。 通过确定偏见更多来自视觉还是文本编码器，我们的工作为未来多模态系统提供了更针对性和有效的偏见缓解策略。 显示英文摘要

摘要： Vision Language Models achieve impressive multi-modal performance but often inherit gender biases from their training data. This bias might be coming from both the vision and text modalities. In this work, we dissect the contributions of vision and text backbones to these biases by applying targeted debiasing using Counterfactual Data Augmentation and Task Vector methods. Inspired by data-efficient approaches in hate-speech classification, we introduce a novel metric, Degree of Stereotypicality and a corresponding debiasing method, Data Augmentation Using Degree of Stereotypicality - DAUDoS, to reduce bias with minimal computational cost. We curate a gender annotated dataset and evaluate all methods on VisoGender benchmark to quantify improvements and identify dominant source of bias. Our results show that CDA reduces the gender gap by 6% and DAUDoS by 3% but using only one-third of the data. Both methods also improve the model's ability to correctly identify gender in images by 3%, with DAUDoS achieving this improvement using only almost one-third of training data. From our experiment's, we observed that CLIP's vision encoder is more biased whereas PaliGemma2's text encoder is more biased. By identifying whether bias stems more from vision or text encoders, our work enables more targeted and effective bias mitigation strategies in future multi-modal systems.