标题： 重新思考领域专用大语言模型基准构建：全面性与紧凑性方法 显示英文标题

标题： Rethinking Domain-Specific LLM Benchmark Construction: A Comprehensiveness-Compactness Approach

摘要： 许多基准已被构建用于评估大型语言模型（LLMs）的领域特定能力，突显了有效且高效构建基准的必要性。 现有的领域特定基准主要关注扩展定律，依赖大量语料进行监督微调或生成广泛的题集以实现广泛覆盖。 然而，语料和问题答案（QA）集设计对领域特定LLMs的精确率和召回率的影响仍未被探索。 在本文中，我们解决了这一差距，并证明扩展定律并不总是特定领域基准构建的最佳原则。 相反，我们提出了Comp-Comp，一个基于全面性-紧凑性原则的迭代基准框架。 在这里，全面性确保领域的语义召回，而紧凑性提高精确度，指导语料和QA集的构建。 为了验证我们的框架，我们在一所著名的大学进行了案例研究，从而创建了XUBench，一个大规模且全面的封闭领域基准。 尽管我们在本工作中使用学术领域作为案例，但我们的Comp-Comp框架设计为可扩展至学术以外的领域，为各个领域的基准构建提供了有价值的见解。 显示英文摘要

摘要： Numerous benchmarks have been built to evaluate the domain-specific abilities of large language models (LLMs), highlighting the need for effective and efficient benchmark construction. Existing domain-specific benchmarks primarily focus on the scaling law, relying on massive corpora for supervised fine-tuning or generating extensive question sets for broad coverage. However, the impact of corpus and question-answer (QA) set design on the precision and recall of domain-specific LLMs remains unexplored. In this paper, we address this gap and demonstrate that the scaling law is not always the optimal principle for benchmark construction in specific domains. Instead, we propose Comp-Comp, an iterative benchmarking framework based on a comprehensiveness-compactness principle. Here, comprehensiveness ensures semantic recall of the domain, while compactness enhances precision, guiding both corpus and QA set construction. To validate our framework, we conducted a case study in a well-renowned university, resulting in the creation of XUBench, a large-scale and comprehensive closed-domain benchmark. Although we use the academic domain as the case in this work, our Comp-Comp framework is designed to be extensible beyond academia, providing valuable insights for benchmark construction across various domains.