标题： 注意差距：稳定离策略监督微调的数据重写 显示英文标题

标题： Mind the Gap: Data Rewriting for Stable Off-Policy Supervised Fine-Tuning

摘要： 监督微调（SFT）大型语言模型可以被视为一个非策略学习问题，其中专家示范来自一个固定的行动策略，而训练的目标是优化一个目标策略。 重要性采样是校正这种分布不匹配的标准工具，但大的策略差距会导致权重偏斜、方差高和优化不稳定。 现有方法通过KL惩罚或截断来缓解这个问题，这些方法被动地限制更新，而不是主动减少差距。 我们提出了一种简单而有效的数据重写框架，在训练前主动缩小策略差距。 对于每个问题，正确的模型生成的解决方案被保留为策略内数据，而错误的解决方案则通过引导重新求解进行重写，仅在需要时回退到专家示范。 这使训练分布与目标策略对齐，减少方差并提高稳定性。 为了处理重写后的剩余不匹配，我们在训练期间额外应用重要性采样，形成一种两阶段的方法，结合数据级对齐与轻量级优化级修正。 在五个数学推理基准上的实验表明，相对于原始SFT和最先进的动态微调（DFT）方法，取得了持续且显著的提升。 数据和代码将在 https://github.com/NKU-HLT/Off-Policy-SFT 发布。 显示英文摘要

摘要： Supervised fine-tuning (SFT) of large language models can be viewed as an off-policy learning problem, where expert demonstrations come from a fixed behavior policy while training aims to optimize a target policy. Importance sampling is the standard tool for correcting this distribution mismatch, but large policy gaps lead to skewed weights, high variance, and unstable optimization. Existing methods mitigate this issue with KL penalties or clipping, which passively restrict updates rather than actively reducing the gap. We propose a simple yet effective data rewriting framework that proactively shrinks the policy gap before training. For each problem, correct model-generated solutions are kept as on-policy data, while incorrect ones are rewritten through guided re-solving, falling back to expert demonstrations only when needed. This aligns the training distribution with the target policy, reducing variance and improving stability. To handle residual mismatch after rewriting, we additionally apply importance sampling during training, forming a two-stage approach that combines data-level alignment with lightweight optimization-level correction. Experiments on five mathematical reasoning benchmarks show consistent and significant gains over both vanilla SFT and the state-of-the-art Dynamic Fine-Tuning (DFT) approach. Data and code will be released at https://github.com/NKU-HLT/Off-Policy-SFT.