标题： RL-PLUS：在强化学习中通过混合策略优化对抗大语言模型的能力边界崩溃 显示英文标题

标题： RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization

摘要： 基于可验证奖励的强化学习（RLVR）显著提升了大型语言模型（LLMs）的复杂推理能力。 然而，由于其本质上是在线策略，加上LLM的巨大动作空间和稀疏奖励，它难以突破基础LLM的固有能力边界。 关键的是，RLVR可能导致能力边界崩溃，缩小LLM的问题解决范围。 为了解决这个问题，我们提出了RL-PLUS，这是一种针对LLMs的新颖混合策略优化方法，通过结合内部利用与外部数据来实现更强的推理能力并超越基础模型的边界。 RL-PLUS集成了两个核心组件，即多重重要性采样，以解决来自外部数据的分布不匹配问题，以及基于探索的优势函数，以引导模型走向高价值、未探索的推理路径。 我们提供了理论分析和大量实验来证明我们方法的优势和通用性。 与现有的RLVR方法相比，RL-PLUS在六个数学推理基准测试中实现了1）最先进的性能；2）在六个分布外推理任务中表现更优；3）在多种模型家族中实现了持续且显著的提升，平均相对改进高达69.2%。 此外，对Pass@k曲线的分析表明，RL-PLUS有效解决了能力边界崩溃的问题。 显示英文摘要

摘要： Reinforcement Learning with Verifiable Reward (RLVR) has significantly advanced the complex reasoning abilities of Large Language Models (LLMs). However, it struggles to break through the inherent capability boundaries of the base LLM, due to its essentially on-policy strategy coupled with LLM's immense action space and sparse reward. Critically, RLVR can lead to the capability boundary collapse, narrowing the LLM's problem-solving scope. To address this problem, we propose RL-PLUS, a novel hybrid-policy optimization approach for LLMs that synergizes internal exploitation with external data to achieve stronger reasoning capabilities and surpass the boundaries of base models. RL-PLUS integrates two core components, i.e., Multiple Importance Sampling to address distributional mismatch from external data, and Exploration-Based Advantage Function to guide the model towards high-value, unexplored reasoning paths. We provide both theoretical analysis and extensive experiments to demonstrate the superiority and generalizability of our approach. Compared with existing RLVR methods, RL-PLUS achieves 1) state-of-the-art performance on six math reasoning benchmarks; 2) superior performance on six out-of-distribution reasoning tasks; 3) consistent and significant gains across diverse model families, with average relative improvements up to 69.2\%. Moreover, the analysis of Pass@k curves indicates that RL-PLUS effectively resolves the capability boundary collapse problem.