标题： GDPO：多奖励RL优化的组奖励解耦归一化策略优化 显示英文标题

标题： GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization

摘要： 随着语言模型的能力不断增强，用户期望它们不仅能提供准确的回答，还能在各种场景中表现出与多样化人类偏好一致的行为。 为了实现这一目标，强化学习（RL）流程开始结合多个奖励，每个奖励捕捉一种不同的偏好，以引导模型朝着这些期望的行为发展。 然而，最近的研究在多奖励设置下默认应用了组相对策略优化（GRPO），而没有检验其适用性。 在本文中，我们证明了直接将GRPO应用于不同的回放奖励组合会导致它们坍缩为相同的优势值，降低了训练信号的分辨率，导致训练信号次优收敛，有时甚至导致早期训练失败。 随后，我们引入了组奖励解耦归一化策略优化（GDPO），这是一种新的策略优化方法，通过解耦单个奖励的归一化来解决这些问题，更忠实地保留它们的相对差异，并实现更精确的多奖励优化，同时显著提高了训练稳定性。 我们在三个任务中将GDPO与GRPO进行了比较：工具调用、数学推理和编码推理，评估了正确性指标（准确性、错误率）和约束遵循指标（格式、长度）。 在所有设置中，GDPO始终优于GRPO，证明了其在多奖励强化学习优化中的有效性和通用性。 显示英文摘要

摘要： As language models become increasingly capable, users expect them to provide not only accurate responses but also behaviors aligned with diverse human preferences across a variety of scenarios. To achieve this, Reinforcement learning (RL) pipelines have begun incorporating multiple rewards, each capturing a distinct preference, to guide models toward these desired behaviors. However, recent work has defaulted to apply Group Relative Policy Optimization (GRPO) under multi-reward setting without examining its suitability. In this paper, we demonstrate that directly applying GRPO to normalize distinct rollout reward combinations causes them to collapse into identical advantage values, reducing the resolution of the training signal and resulting in suboptimal convergence and, in some cases, early training failure. We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues by decoupling the normalization of individual rewards, more faithfully preserving their relative differences and enabling more accurate multi-reward optimization, along with substantially improved training stability. We compare GDPO with GRPO across three tasks: tool calling, math reasoning, and coding reasoning, evaluating both correctness metrics (accuracy, bug ratio) and constraint adherence metrics (format, length). Across all settings, GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.