标题： 使用强化学习在多智能体马尔可夫决策过程中的公平性实现 显示英文标题

标题： Achieving Fairness in Multi-Agent Markov Decision Processes Using Reinforcement Learning

摘要： 公平在各种多智能体系统（例如，通信网络、金融市场等）中起着关键作用。 许多多智能体动态交互可以表示为马尔可夫决策过程（MDPs）。 尽管现有研究集中在已知环境中的公平性研究，但此类系统在未知环境中的公平性探索仍是一个开放问题。 在本文中，我们提出一种强化学习（RL）方法，以在多智能体有限时间范围的周期性MDPs中实现公平性。 我们引入一个公平函数，而不是最大化个体智能体的价值函数之和，以确保 across agents 的公平奖励。 由于当个体价值函数之和不是最大时，经典的贝尔曼方程不成立，因此我们不能使用传统方法。 相反，为了探索，我们维护未知环境的置信度边界，然后提出一种基于在线凸优化的方法，以获得受限于该置信区域的策略。 我们证明这种方法在周期数方面实现了次线性遗憾。 此外，我们基于获得的遗憾界提供了可能近似正确（PAC）保证。 我们还提出了一种离线RL算法，并与最优公平解相比界定了最优性差距。 为了减轻计算复杂性，我们引入了一种针对公平目标的策略梯度方法。 模拟实验也证明了我们方法的有效性。 显示英文摘要

摘要： Fairness plays a crucial role in various multi-agent systems (e.g., communication networks, financial markets, etc.). Many multi-agent dynamical interactions can be cast as Markov Decision Processes (MDPs). While existing research has focused on studying fairness in known environments, the exploration of fairness in such systems for unknown environments remains open. In this paper, we propose a Reinforcement Learning (RL) approach to achieve fairness in multi-agent finite-horizon episodic MDPs. Instead of maximizing the sum of individual agents' value functions, we introduce a fairness function that ensures equitable rewards across agents. Since the classical Bellman's equation does not hold when the sum of individual value functions is not maximized, we cannot use traditional approaches. Instead, in order to explore, we maintain a confidence bound of the unknown environment and then propose an online convex optimization based approach to obtain a policy constrained to this confidence region. We show that such an approach achieves sub-linear regret in terms of the number of episodes. Additionally, we provide a probably approximately correct (PAC) guarantee based on the obtained regret bound. We also propose an offline RL algorithm and bound the optimality gap with respect to the optimal fair solution. To mitigate computational complexity, we introduce a policy-gradient type method for the fair objective. Simulation experiments also demonstrate the efficacy of our approach.