2026-10-14T15:43:49Z https://www.scidb.cn/oai

10.57760/sciencedb.010v1 2026-08-31T08:54:13Z

2026-08-31 RACE doi:10.57760/sciencedb.010v1 en In recent years, researchers have proposed numerous benchmarks to evaluate the impressive coding capabilities of large language models (LLMs). However, current benchmarks primarily assess the accuracy of generated code while neglecting the LLMs' multi-dimensional instruction-following abilities, resulting in a gap between benchmark performance and real-world development requirements. Therefore, this paper proposes the RACE benchmark, which focuses on evaluating LLMs' instruction-following ability across four dimensions: Readability, mAintainability, Correctness, and Efficiency. We select specific, actionable attributes for each dimension and establish verifiable instructions grounded in common software development practices. We then develop deterministic evaluation methods to ensure objective and reliable assessment. We analyze 24 representative LLMs based on RACE and find that: 1) Using correctness as the only guiding indicator is insufficient to steer code LLMs towards comprehensive advancement. 2) RACE not only provides valuable insights that reveal the defects of current LLMs across multiple dimensions, but also serves as an effective tool for mitigating the risks of data contamination. These findings highlight the need for a multi-dimensional instruction-following evaluation for LLMs' code ability, emphasizing metrics beyond correctness for real-world applications. large language models; code generation; instruction following; benchmark; multi-dimension 郑家胜 曹博希 马争朝 潘若彤 林鸿宇 陆垚杰 韩先培 孙乐 PUBLIC https://creativecommons.org/licenses/by/4.0/ dataset Science Data Bank