标题： RePo：通过正则化后验可预测性实现的弹性基于模型的强化学习 显示英文标题

标题： RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictability

摘要： 基于视觉模型的强化学习方法通常以不消除冗余信息的方式将图像观测编码为低维表示。这使得它们容易受到虚假变化的影响——任务无关组件的变化，例如背景干扰或光照条件的变化。在本文中，我们提出了一种基于视觉模型的强化学习方法，该方法学习一种对这种虚假变化具有鲁棒性的潜在表示。我们的训练目标鼓励表示尽可能预测动态和奖励，同时限制从观测到潜在表示的信息流。我们证明，这个目标显著增强了基于视觉模型的强化学习方法对视觉干扰的鲁棒性，使其能够在动态环境中运行。然后我们表明，虽然学习到的编码器对虚假变化具有鲁棒性，但在显著分布偏移下并不具有不变性。为了解决这个问题，我们提出了一种简单的无奖励对齐过程，使编码器能够在测试时进行适应。这允许在不需要重新学习动态和策略的情况下快速适应广泛不同的环境。我们的工作是使基于模型的强化学习成为动态、多样化领域实用且有用的工具迈出的一步。我们在具有显著虚假变化的仿真基准以及具有背景噪声电视的真实世界自我中心导航任务中展示了其有效性。视频和代码见https://zchuning.github.io/repo-website/。 显示英文摘要

摘要： Visual model-based RL methods typically encode image observations into low-dimensional representations in a manner that does not eliminate redundant information. This leaves them susceptible to spurious variations -- changes in task-irrelevant components such as background distractors or lighting conditions. In this paper, we propose a visual model-based RL method that learns a latent representation resilient to such spurious variations. Our training objective encourages the representation to be maximally predictive of dynamics and reward, while constraining the information flow from the observation to the latent representation. We demonstrate that this objective significantly bolsters the resilience of visual model-based RL methods to visual distractors, allowing them to operate in dynamic environments. We then show that while the learned encoder is resilient to spirious variations, it is not invariant under significant distribution shift. To address this, we propose a simple reward-free alignment procedure that enables test time adaptation of the encoder. This allows for quick adaptation to widely differing environments without having to relearn the dynamics and policy. Our effort is a step towards making model-based RL a practical and useful tool for dynamic, diverse domains. We show its effectiveness in simulation benchmarks with significant spurious variations as well as a real-world egocentric navigation task with noisy TVs in the background. Videos and code at https://zchuning.github.io/repo-website/.