标题： RewardMap：通过多阶段强化学习解决细粒度视觉推理中的稀疏奖励问题 显示英文标题

标题： RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning

摘要： 细粒度视觉推理仍然是多模态大语言模型（MLLMs）的核心挑战。 最近引入的ReasonMap通过显示即使先进的MLLMs在结构化和信息丰富的环境（如交通图）中的空间推理上也存在困难，突显了这一差距，这是一个具有明确实际和科学重要性的问题。 然而，在此类任务上的标准强化学习（RL）受到稀疏奖励和不稳定优化的阻碍。 为了解决这个问题，我们首先构建了ReasonMap-Plus，一个扩展的数据集，通过视觉问答（VQA）任务引入密集奖励信号，从而实现细粒度视觉理解技能的有效冷启动训练。 接下来，我们提出了RewardMap，一个多层次的RL框架，旨在提升MLLMs的视觉理解和推理能力。 RewardMap结合了两个关键设计。 首先，我们引入了一种难度感知的奖励设计，包括细节奖励，直接解决稀疏奖励问题，同时提供更丰富的监督。 其次，我们提出了一种多层次的RL方案，从简单的感知任务开始进行训练，逐步过渡到复杂的推理任务，相比传统的监督微调（SFT），提供了更有效的冷启动策略。 在ReasonMap和ReasonMap-Plus上的实验表明，RewardMap的每个组件都带来了持续的性能提升，而它们的结合则取得了最佳结果。 此外，使用RewardMap训练的模型在跨越空间推理、细粒度视觉推理和超越交通图的一般任务的6个基准测试中平均提升了3.47%，凸显了增强的视觉理解和推理能力。 显示英文摘要

摘要： Fine-grained visual reasoning remains a core challenge for multimodal large language models (MLLMs). The recently introduced ReasonMap highlights this gap by showing that even advanced MLLMs struggle with spatial reasoning in structured and information-rich settings such as transit maps, a task of clear practical and scientific importance. However, standard reinforcement learning (RL) on such tasks is impeded by sparse rewards and unstable optimization. To address this, we first construct ReasonMap-Plus, an extended dataset that introduces dense reward signals through Visual Question Answering (VQA) tasks, enabling effective cold-start training of fine-grained visual understanding skills. Next, we propose RewardMap, a multi-stage RL framework designed to improve both visual understanding and reasoning capabilities of MLLMs. RewardMap incorporates two key designs. First, we introduce a difficulty-aware reward design that incorporates detail rewards, directly tackling the sparse rewards while providing richer supervision. Second, we propose a multi-stage RL scheme that bootstraps training from simple perception to complex reasoning tasks, offering a more effective cold-start strategy than conventional Supervised Fine-Tuning (SFT). Experiments on ReasonMap and ReasonMap-Plus demonstrate that each component of RewardMap contributes to consistent performance gains, while their combination yields the best results. Moreover, models trained with RewardMap achieve an average improvement of 3.47% across 6 benchmarks spanning spatial reasoning, fine-grained visual reasoning, and general tasks beyond transit maps, underscoring enhanced visual understanding and reasoning capabilities.