标题： 一块滚动的石头不长青苔：大型多模态模型中稳定自我评估的自适应策略优化 显示英文标题

标题： A Rolling Stone Gathers No Moss: Adaptive Policy Optimization for Stable Self-Evaluation in Large Multimodal Models

摘要： 自我评估，即模型评估自身输出正确性的能力，对于大型多模态模型（LMMs）在多轮对话中实现自我改进至关重要，但在基础模型中却 largely 缺失。 最近的工作已采用强化学习（RL）来增强自我评估；然而，其固定的奖励机制在优化多个训练目标时容易受到奖励劫持的影响，导致模型崩溃。 在本文中，我们提出了AdaPO，这是一种在线强化学习框架，能够根据每个任务的当前训练状态实时自适应调整训练目标。 具体而言，为了缓解奖励劫持，AdaPO引入了一个自适应奖励模型（ARM）和一个奖励感知动态KL正则化机制。 ARM从模型生成的多轮轨迹性能分布中评估任务的训练状态。 奖励感知动态KL用由不同多轮情境之间的奖励差距调节的动态系数替代了固定的惩罚项。 值得注意的是，我们的方法无需人工干预，即可根据子任务的训练进度自动且平滑地调整其学习重点。 在8个基准测试和各种模型上的广泛实验表明，我们的方法显著增强了直接推理和自我评估能力。 我们将发布我们的代码以贡献给社区。 显示英文摘要

摘要： Self-evaluation, a model's ability to assess the correctness of its own output, is crucial for Large Multimodal Models (LMMs) to achieve self-improvement in multi-turn conversations, yet largely absent in foundation models. Recent work has employed reinforcement learning (RL) to enhance self-evaluation; however, its fixed reward mechanism suffers from reward hacking when optimizing multiple training objectives, leading to model collapse. In this paper we propose AdaPO, an online reinforcement learning framework capable of adaptively adjusting training objective in real time according to the current training state for each task. Specifically, to mitigate reward hacking , AdaPO introduces an Adaptive Reward Model (ARM) and a Reward Aware Dynamic KL Regularization mechanism. ARM assesses the task's training state from the distribution of model generated multi-turn trajectories' performance. Reward Aware Dynamic KL replaces a fixed penalty with dynamic coefficients which is modulated by the reward gap between different multi-turn situations. Notably, our method automatically and smoothly adjusts its learning focus based on sub-tasks' training progress without manual intervention. Extensive experiments over 8 benchmarks and various models show that our method significantly enhances both direct reasoning and self-evaluation capability. We will release our code to contribute to the community.