标题： MemoryBench：用于LLM系统中内存和持续学习的基准测试 显示英文标题

标题： MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems

摘要： 扩大数据、参数和测试时计算已成为改进LLM系统（LLMsys）的主流方法，但由于高质量数据的逐渐耗尽以及从更大计算资源消耗中获得的边际收益，它们的上限几乎已经到达。 受人类和传统AI系统在实践中学习的能力启发，为LLMsys构建记忆和持续学习框架已成为近年来文献中的一个重要且流行的研究方向。 然而，现有的LLM记忆基准通常专注于在同质阅读理解任务上评估系统，而不是测试它们在服务时间内从累积用户反馈中学习的能力。 因此，我们提出了一种用户反馈模拟框架和一个涵盖多个领域、语言和任务类型的综合基准，以评估LLMsys的持续学习能力。 实验表明，最先进的基线的有效性和效率远未满足要求，我们希望这个基准能为未来关于LLM记忆和优化算法的研究铺平道路。 显示英文摘要

摘要： Scaling up data, parameters, and test-time computation has been the mainstream methods to improve LLM systems (LLMsys), but their upper bounds are almost reached due to the gradual depletion of high-quality data and marginal gains obtained from larger computational resource consumption. Inspired by the abilities of human and traditional AI systems in learning from practice, constructing memory and continual learning frameworks for LLMsys has become an important and popular research direction in recent literature. Yet, existing benchmarks for LLM memory often focus on evaluating the system on homogeneous reading comprehension tasks with long-form inputs rather than testing their abilities to learn from accumulated user feedback in service time. Therefore, we propose a user feedback simulation framework and a comprehensive benchmark covering multiple domains, languages, and types of tasks to evaluate the continual learning abilities of LLMsys. Experiments show that the effectiveness and efficiency of state-of-the-art baselines are far from satisfying, and we hope this benchmark could pave the way for future studies on LLM memory and optimization algorithms.