标题： ROSARL：仅奖励安全强化学习 显示英文标题

标题： ROSARL: Reward-Only Safe Reinforcement Learning

摘要： 在强化学习中，一个重要的问题是设计能够在环境中安全地解决问题的智能体。 常见的解决方案是让人类专家在奖励函数中定义惩罚，或在到达不安全状态时定义需要最小化的成本。 然而，这并不容易，因为惩罚过小可能导致智能体到达不安全状态，而惩罚过大则会增加收敛时间。 此外，设计奖励或成本函数的难度可能随着问题复杂性的增加而增加。 因此，对于给定环境和一组不安全状态，我们感兴趣的是找到不安全状态下奖励的上界，其最优策略能够最小化到达这些不安全状态的概率，而不管任务奖励如何。 我们将这个精确的上界称为“最小最大惩罚”，并证明可以通过考虑环境的可控性和直径来获得它。 我们提供了一个简单实用的无模型算法，使智能体在学习任务策略的同时学习这个最小最大惩罚，并证明使用它可以使智能体在高维连续控制环境中学习到安全策略。 显示英文摘要

摘要： An important problem in reinforcement learning is designing agents that learn to solve tasks safely in an environment. A common solution is for a human expert to define either a penalty in the reward function or a cost to be minimised when reaching unsafe states. However, this is non-trivial, since too small a penalty may lead to agents that reach unsafe states, while too large a penalty increases the time to convergence. Additionally, the difficulty in designing reward or cost functions can increase with the complexity of the problem. Hence, for a given environment with a given set of unsafe states, we are interested in finding the upper bound of rewards at unsafe states whose optimal policies minimise the probability of reaching those unsafe states, irrespective of task rewards. We refer to this exact upper bound as the "Minmax penalty", and show that it can be obtained by taking into account both the controllability and diameter of an environment. We provide a simple practical model-free algorithm for an agent to learn this Minmax penalty while learning the task policy, and demonstrate that using it leads to agents that learn safe policies in high-dimensional continuous control environments.