标题： 让我们逐步测量信息：超越感觉的基于LLM的评估 显示英文标题

标题： Let's Measure Information Step-by-Step: LLM-Based Evaluation Beyond Vibes

摘要： 我们开发了在没有真实数据的情况下评估AI系统的机制，这是通过利用游戏抵抗性和输出质量之间的联系来实现的。 数据处理不等式确保了事后尝试操纵指标会降低信息内容和任务性能。 我们证明，在自然条件下，f-互信息度量是唯一的抗游戏机制，监督者作为代理发挥作用。 虽然香农互信息面临指数级的样本复杂性，但像总变差距离这样的有界度量仍然易于处理。 经验上，在从翻译到同行评审的十个领域中，所有信息论机制都能在忠实代理和策略代理之间实现完美的区分（d > 0.5）。 相比之下，LLM评判者表现出系统性的评估反转，更喜欢虚构内容而非准确摘要。 我们的机制在对抗性操控方面的鲁棒性比当前实践好10到100倍。 我们还发现性能随着压缩比呈倒U型曲线变化，在10:1时达到峰值，此时代理响应表现出最优的信息多样性（3个有效维度），从偏差-方差的角度来看，说明了我们的方法何时最有效。 显示英文摘要

摘要： We develop mechanisms for evaluating AI systems without ground truth by exploiting a connection between gaming resistance and output quality. The data processing inequality ensures post-hoc attempts to game a metric degrades both information content and task performance. We prove that f-mutual information measures are the unique gaming resistant mechanisms under natural conditions, with the overseer acting as an agent. While Shannon mutual information faces exponential sample complexity, bounded measures like total variation distance remain tractable. Empirically, across ten domains from translation to peer review, all information-theoretic mechanisms achieve perfect discrimination (d > 0.5) between faithful and strategic agents. In contrast, LLM judges exhibit systematic evaluation inversion, preferring fabricated content over accurate summaries. Our mechanisms show 10-100x better robustness to adversarial manipulation than current practices. We also find performance follows an inverted-U curve with compression ratio, peaking at 10:1 where agent responses exhibit optimal information diversity (3 effective dimensions), giving a bias-variance perspective on when our approach is expected to be most effective.