标题： 从随机性到信号：一种用于LLMs可靠测量的贝叶斯潜在状态模型 显示英文标题

标题： From Stochasticity to Signal: A Bayesian Latent State Model for Reliable Measurement with LLMs

摘要： 大型语言模型（LLMs）越来越多地用于自动化商业中的分类任务，例如从文本中分析客户满意度。 然而，LLMs的固有随机性，即它们对同一输入产生不同输出的倾向，会引发一个重要的测量误差问题，通常在仅进行一次输出时被忽视，或通过诸如多数投票等临时方法来解决。 这些简单的方法无法量化不确定性，并可能导致总体水平指标的偏差估计。 在本文中，我们通过将LLM的变异性重新表述为统计测量误差问题，并引入一个贝叶斯潜在状态模型来解决这一问题，提出了一种系统性的解决方案。 我们的模型将真实的分类（例如，客户不满）视为一个未观察到的潜在变量，而多个LLM评分则作为该状态的噪声测量值。 该框架允许同时估计LLM的假阳性率和假阴性率、现象在总体中的基础率、每个个体观测值的真实状态的后验概率，以及任何业务干预对该潜在状态的因果影响。 通过模拟研究，我们证明了我们的模型能够在简单方法失败的情况下准确恢复真实参数。 我们得出结论，这种方法为将LLM的噪声、概率输出转化为科学和商业应用中准确且可操作的见解提供了一个通用且可靠的框架。 显示英文摘要

摘要： Large Language Models (LLMs) are increasingly used to automate classification tasks in business, such as analyzing customer satisfaction from text. However, the inherent stochasticity of LLMs, in terms of their tendency to produce different outputs for the same input, creates a significant measurement error problem that is often neglected with a single round of output, or addressed with ad-hoc methods like majority voting. Such naive approaches fail to quantify uncertainty and can produce biased estimates of population-level metrics. In this paper, we propose a principled solution by reframing LLM variability as a statistical measurement error problem and introducing a Bayesian latent state model to address it. Our model treats the true classification (e.g., customer dissatisfaction) as an unobserved latent variable and the multiple LLM ratings as noisy measurements of this state. This framework allows for the simultaneous estimation of the LLM's false positive and false negative error rates, the underlying base rate of the phenomenon in the population, the posterior probability of the true state for each individual observation, and the causal impact of a business intervention, if any, on the latent state. Through simulation studies, we demonstrate that our model accurately recovers true parameters where naive methods fail. We conclude that this methodology provides a general and reliable framework for converting noisy, probabilistic outputs from LLMs into accurate and actionable insights for scientific and business applications.