标题： 通过融合启发式方法改进离线强化学习 显示英文标题

标题： Improving Offline RL by Blending Heuristics

摘要： 我们提出启发式混合（HUBL），这是一种基于值引导的离线强化学习算法的广泛类别的简单性能提升技术。 HUBL修改了这些算法中使用的贝尔曼算子，部分用通过蒙特卡洛回报估计的启发式值替换引导值。 对于回报较高的轨迹，HUBL更依赖启发式值，而较少依赖引导；否则，它更依赖引导。 HUBL可以通过用调整后的奖励和折扣因子重新标记离线数据集，很容易地与许多现有的离线强化学习实现结合。 我们推导出一种理论，解释HUBL对离线强化学习的影响，即降低离线强化学习的复杂性，从而提高其有限样本性能。 此外，我们通过实证证明，HUBL在D4RL和Meta-World基准的27个数据集上，平均提高了四种最先进的基于引导的离线强化学习算法（ATAC、CQL、TD3+BC和IQL）的策略质量，提高幅度为9%。 显示英文摘要

摘要： We propose Heuristic Blending (HUBL), a simple performance-improving technique for a broad class of offline RL algorithms based on value bootstrapping. HUBL modifies the Bellman operators used in these algorithms, partially replacing the bootstrapped values with heuristic ones that are estimated with Monte-Carlo returns. For trajectories with higher returns, HUBL relies more on the heuristic values and less on bootstrapping; otherwise, it leans more heavily on bootstrapping. HUBL is very easy to combine with many existing offline RL implementations by relabeling the offline datasets with adjusted rewards and discount factors. We derive a theory that explains HUBL's effect on offline RL as reducing offline RL's complexity and thus increasing its finite-sample performance. Furthermore, we empirically demonstrate that HUBL consistently improves the policy quality of four state-of-the-art bootstrapping-based offline RL algorithms (ATAC, CQL, TD3+BC, and IQL), by 9% on average over 27 datasets of the D4RL and Meta-World benchmarks.