标题： 样本高效的非平稳环境经验回放 显示英文标题

标题： Sample Efficient Experience Replay in Non-stationary Environments

摘要： 强化学习（RL）在非平稳环境中具有挑战性，因为动态变化和奖励会迅速使过去的经历过时。 传统的经验回放（ER）方法，特别是那些使用TD误差优先级的方法，在区分代理策略引起的变化和环境引起的变化方面存在困难，导致在动态条件下学习效率低下。 为了解决这一挑战，我们提出了环境动态差异（DoE），这是一种将环境变化对价值函数的影响隔离的度量标准。 在此基础上，我们引入了环境优先经验回放差异（DEER），这是一种自适应的ER框架，根据策略更新和环境变化对转换进行优先级排序。 DEER使用二元分类器检测环境变化，并在每次变化前后应用不同的优先级策略，从而实现更高效的样本学习。 在四个非平稳基准测试中的实验表明，与最先进的ER方法相比，DEER进一步将离策略算法的性能提高了11.54%。 显示英文摘要

摘要： Reinforcement learning (RL) in non-stationary environments is challenging, as changing dynamics and rewards quickly make past experiences outdated. Traditional experience replay (ER) methods, especially those using TD-error prioritization, struggle to distinguish between changes caused by the agent's policy and those from the environment, resulting in inefficient learning under dynamic conditions. To address this challenge, we propose the Discrepancy of Environment Dynamics (DoE), a metric that isolates the effects of environment shifts on value functions. Building on this, we introduce Discrepancy of Environment Prioritized Experience Replay (DEER), an adaptive ER framework that prioritizes transitions based on both policy updates and environmental changes. DEER uses a binary classifier to detect environment changes and applies distinct prioritization strategies before and after each shift, enabling more sample-efficient learning. Experiments on four non-stationary benchmarks demonstrate that DEER further improves the performance of off-policy algorithms by 11.54 percent compared to the best-performing state-of-the-art ER methods.