标题： 民主化外交：一种用于在全面压力外交中评估任何大型语言模型的工具 显示英文标题

标题： Democratizing Diplomacy: A Harness for Evaluating Any Large Language Model on Full-Press Diplomacy

摘要： 我们提出了第一个评估工具，使任何现成的、本地的大型语言模型（LLMs）能够在不进行微调或专门训练的情况下完整地进行全压外交游戏。 以往的工作由于外交游戏状态的高复杂性和信息密度，需要前沿LLMs或微调。 结合比赛的高变异性，这些因素使得外交游戏难以研究。 在本工作中，我们通过数据驱动的迭代优化了文本游戏状态表示，使得一个24B模型能够在没有任何微调的情况下可靠地完成比赛。 我们开发了工具以促进假设测试和统计分析，并展示了关于说服力、激进玩法和不同模型性能的案例研究。 我们在许多流行的LLMs上进行了各种实验，发现更大的模型表现最好，但较小的模型仍然可以适当玩。 我们还引入了关键状态分析：一种用于快速迭代和深入分析游戏中关键时刻的实验协议。 我们的工具通过消除微调的需要，使LLMs的战略推理评估民主化，并提供了这些能力如何自然地从广泛使用的LLMs中产生的见解。 我们的代码可在补充材料中找到，并将开源。 显示英文摘要

摘要： We present the first evaluation harness that enables any out-of-the-box, local, Large Language Models (LLMs) to play full-press Diplomacy without fine-tuning or specialized training. Previous work required frontier LLMs, or fine-tuning, due to the high complexity and information density of Diplomacy's game state. Combined with the high variance of matches, these factors made Diplomacy prohibitive for study. In this work, we used data-driven iteration to optimize a textual game state representation such that a 24B model can reliably complete matches without any fine tuning. We develop tooling to facilitate hypothesis testing and statistical analysis, and we present case studies on persuasion, aggressive playstyles, and performance across a range of models. We conduct a variety of experiments across many popular LLMs, finding the larger models perform the best, but the smaller models still play adequately. We also introduce Critical State Analysis: an experimental protocol for rapidly iterating and analyzing key moments in a game at depth. Our harness democratizes the evaluation of strategic reasoning in LLMs by eliminating the need for fine-tuning, and it provides insights into how these capabilities emerge naturally from widely used LLMs. Our code is available in the supplement and will be open sourced.