标题： ExGRPO：从经验中学习推理 显示英文标题

标题： ExGRPO: Learning to Reason from Experience

摘要： 从可验证奖励中进行强化学习（RLVR）是一种新兴的范式，用于提高大型语言模型的推理能力。 然而，标准的策略梯度训练在一次更新后会丢弃滚动体验，导致计算效率低下和不稳定。 尽管之前关于强化学习的研究强调了重用过去经验的好处，但经验特征在塑造大型推理模型学习动态中的作用仍鲜有研究。 在本文中，我们首次研究了什么使推理经验有价值，并将滚动正确性和熵作为经验价值的有效指标。 基于这些见解，我们提出了 ExGRPO（经验组相对策略优化），一种组织和优先处理有价值经验的框架，并采用混合策略目标来平衡探索与经验利用。 在五个主干模型（1.5B-8B参数）上的实验表明，ExGRPO在数学/通用基准测试中持续提高了推理性能，相对于策略梯度RLVR平均提升了+3.5/7.6分。 此外，ExGRPO在更强和更弱的模型上稳定了训练，而策略梯度方法在此类模型上失败。 这些结果突显了有原则的经验管理是高效和可扩展RLVR的关键要素。 显示英文摘要

摘要： Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm for improving the reasoning ability of large language models. However, standard on-policy training discards rollout experiences after a single update, leading to computational inefficiency and instability. While prior work on RL has highlighted the benefits of reusing past experience, the role of experience characteristics in shaping learning dynamics of large reasoning models remains underexplored. In this paper, we are the first to investigate what makes a reasoning experience valuable and identify rollout correctness and entropy as effective indicators of experience value. Based on these insights, we propose ExGRPO (Experiential Group Relative Policy Optimization), a framework that organizes and prioritizes valuable experiences, and employs a mixed-policy objective to balance exploration with experience exploitation. Experiments on five backbone models (1.5B-8B parameters) show that ExGRPO consistently improves reasoning performance on mathematical/general benchmarks, with an average gain of +3.5/7.6 points over on-policy RLVR. Moreover, ExGRPO stabilizes training on both stronger and weaker models where on-policy methods fail. These results highlight principled experience management as a key ingredient for efficient and scalable RLVR.