标题： 向有限时间范围内噪声线性二次强化学习中的最优统计推断迈进 显示英文标题

标题： Toward Optimal Statistical Inference in Noisy Linear Quadratic Reinforcement Learning over a Finite Horizon

摘要： 强化学习的最新进展显著提高了在不确定环境中的序列决策能力。 尽管现有工作在提升学习控制策略的操作精度和学习算法的收敛速度方面取得了显著成效，但对不确定性量化和统计推断的关注相对较少。 然而，这些方面对于评估控制策略的可靠性和变异性至关重要，特别是在高风险应用中。 在本文中，我们研究了在有限时间范围内噪声线性二次强化学习（LQ RL）中策略梯度（PG）方法的统计推断问题，其中线性动力学在已知和未知漂移参数的情况下受到二次成本的约束。 我们建立了LQ RL中统计推断的理论基础，推导了PG估计量及其相应目标损失的精确渐近结果。 此外，我们引入了一个合理的推断框架，利用在线自助法为学习到的最优策略及其相应的目标损失构建置信区间。 该方法在新观测数据到达时更新PG估计量以及一组随机扰动的PG估计量。 我们证明了所提出的自助程序在分布上是一致的，并且得到的置信区间在渐近和非渐近意义上都有效。 值得注意的是，我们的结果表明，精确分布的分位数可以以$n^{-1/4}$的速率进行近似，其中$n$是该过程中使用的样本数量。 所提出的方法易于实现，适用于离线和完全在线设置。 数值实验展示了我们在一系列噪声线性动态系统中方法的有效性。 显示英文摘要

摘要： Recent developments in Reinforcement learning have significantly enhanced sequential decision-making in uncertain environments. Despite their strong performance guarantees, most existing work has focused primarily on improving the operational accuracy of learned control policies and the convergence rates of learning algorithms, with comparatively little attention to uncertainty quantification and statistical inference. Yet, these aspects are essential for assessing the reliability and variability of control policies, especially in high-stakes applications. In this paper, we study statistical inference for the policy gradient (PG) method for noisy Linear Quadratic Reinforcement learning (LQ RL) over a finite time horizon, where linear dynamics with both known and unknown drift parameters are controlled subject to a quadratic cost. We establish the theoretical foundations for statistical inference in LQ RL, deriving exact asymptotics for both the PG estimators and the corresponding objective loss. Furthermore, we introduce a principled inference framework that leverages online bootstrapping to construct confidence intervals for both the learned optimal policy and the corresponding objective losses. The method updates the PG estimates along with a set of randomly perturbed PG estimates as new observations arrive. We prove that the proposed bootstrapping procedure is distributionally consistent and that the resulting confidence intervals achieve both asymptotic and non-asymptotic validity. Notably, our results imply that the quantiles of the exact distribution can be approximated at a rate of $n^{-1/4}$, where $n$ is the number of samples used during the procedure. The proposed procedure is easy to implement and applicable to both offline and fully online settings. Numerical experiments illustrate the effectiveness of our approach across a range of noisy linear dynamical systems.