标题： 重要性采样就是你需要的一切：通过重用现有基准来预测大型语言模型在新基准上的表现 显示英文标题

标题： Importance Sampling is All You Need: Predict LLM's performance on new benchmark by reusing existing benchmark

摘要： 随着大型语言模型的快速发展，代码生成已成为评估LLM能力的关键基准。 然而，现有的基准测试面临两个主要挑战：(1) 构建高质量测试套件和参考解决方案的成本不断上升，以及(2) 数据污染风险增加，这削弱了基于基准的评估的可靠性。 在本文中，我们提出了BIS，一种以提示为中心的评估框架，能够在无需真实答案的情况下预测LLM在代码生成任务中的性能。 BIS不是执行生成的代码，而是仅通过分析提示分布来估计性能指标。 该方法基于重要性抽样理论，并使用重要性加权自动编码器实现，我们的方法对现有标注基准中的样本进行重新加权，以估计在新未见过的基准上的性能。 为了稳定估计，我们引入了权重截断策略，并计算拟合分布上的边缘期望。 BIS作为一种补充工具，在资源受限的情况下支持基准开发和验证，为提示选择和污染评估提供可操作且快速的反馈。 我们进行了广泛的实验，涉及4个CodeLlama模型和9个不同的基准测试，共8000个评估点。 我们的框架在代码正确性评分上的平均绝对预测误差为1.1%，最佳和最差情况下的误差分别为0.3%和1.9%。 它在其他指标上也表现良好，pass@1的平均绝对误差为2.15%。 这些结果证明了BIS的可靠性和广泛适用性，可以显著降低在与代码相关的任务中对LLM进行基准测试的成本和努力。 显示英文摘要

摘要： With the rapid advancement of large language models , code generation has become a key benchmark for evaluating LLM capabilities. However, existing benchmarks face two major challenges: (1) the escalating cost of constructing high-quality test suites and reference solutions, and (2) the increasing risk of data contamination, which undermines the reliability of benchmark-based evaluations. In this paper, we propose BIS, a prompt-centric evaluation framework that enables ground-truth-free prediction of LLM performance on code generation tasks. Rather than executing generated code, BIS estimates performance metrics by analyzing the prompt distribution alone. Built on importance sampling theory and implemented using Importance Weighted Autoencoders, our method reweights samples from existing annotated benchmarks to estimate performance on new, unseen benchmarks. To stabilize the estimation, we introduce weight truncation strategies and compute marginal expectations across the fitted distributions. BIS serves as a complementary tool that supports benchmark development and validation under constrained resources, offering actionable and quick feedback for prompt selection and contamination assessment. We conduct extensive experiments involving 8,000 evaluation points across 4 CodeLlama models and 9 diverse benchmarks. Our framework achieves an average absolute prediction error of 1.1% for code correctness scores, with best- and worst-case errors of 0.3% and 1.9%, respectively. It also generalizes well to other metrics, attaining average absolute errors of 2.15% for pass@1. These results demonstrate the reliability and broad applicability of BIS, which can significantly reduce the cost and effort of benchmarking LLMs in code-related tasks.