标题： ThinkTuning：在不进行知识蒸馏的情况下灌输认知反思 显示英文标题

标题： ThinkTuning: Instilling Cognitive Reflections without Distillation

摘要： 最近在测试时缩放方面的进展导致了具有自我反思行为和多步骤推理的思维大语言模型的出现。虽然强化学习推动了这种自我改进范式，但最近的一项研究（Gandhi等，2025）表明，仅靠强化学习并不能真正培养这些新的推理能力——它只是揭示了基础模型中已有的行为。这引发了一个问题：我们如何训练那些不表现出这种思维行为的模型，使其首先发展出这种能力？为此，我们提出了ThinkTuning，一种基于GRPO的交互式训练方法，我们在其中通过教师模型的指导来增强学生模型的 rollout。来自课堂实践的一个简单想法启发了我们的方法：教师提出一个问题，让学生尝试回答，然后给予纠正性反馈——足以指出正确的方向，然后展示解决方案。每一条反馈都会重塑学生的思维，引导他们得出正确答案。同样，我们发现，通过同规模教师模型的反馈进行这种隐式监督可以提高学生模型的推理能力。特别是，平均而言，我们的方法在基准测试中比零样本基线提高了3.85%，在MATH-500、AIME和GPQA-Diamond上分别比原始GRPO基线提高了2.08%、2.23%和3.99%。源代码可在https://github.com/3rdAT/ThinkTuning获取。 显示英文摘要

摘要： Recent advances in test-time scaling have led to the emergence of thinking LLMs that exhibit self-reflective behaviors and multi-step reasoning. While RL drives this self-improvement paradigm, a recent study (Gandhi et al., 2025) shows that RL alone does not truly instill these new reasoning abilities - it merely draws out behaviors already present in the base models. This raises a question: How can we train the models that don't exhibit such thinking behavior to develop it in the first place? To this end, we propose ThinkTuning, a GRPO-based interactive training approach where we augment the rollouts of a student model with the guidance from a teacher model. A simple idea from classroom practice inspires our method: a teacher poses a problem, lets the student try an answer, then gives corrective feedback -- enough to point the mind in the right direction and then show the solution. Each piece of feedback reshapes the student's thoughts, leading them to arrive at the correct solution. Similarly, we find that this type of implicit supervision through feedback from a teacher model of the same size improves the reasoning capabilities of the student model. In particular, on average, our method shows a 3.85% improvement over zero-shot baselines across benchmarks, and on MATH-500, AIME and GPQA-Diamond it shows 2.08%, 2.23% and 3.99% improvements over the vanilla-GRPO baseline. Source code is available at https://github.com/3rdAT/ThinkTuning.