阿里巴巴开源的多语言基准测试,用于在仓库级别评估 LLM 在自动代码审查方面的表现,该基准测试采用人工智能辅助和专家验证的数据。
阿里巴巴开源的多语言基准测试,用于在仓库级别评估 LLM 在自动代码审查方面的表现,该基准测试采用人工智能辅助和专家验证的数据。
AACR-Bench is the industry's first multilingual, repository-level context-aware code review evaluation dataset, designed to assess the performance of large language models in automated code review tasks. The dataset comprises 200 real Pull Requests from 50 active open-source projects, covering 10 mainstream programming languages. Each instance includes not only code changes but also preserves complete repository context, authentically reproducing the entire code review process. Through human-LLM collaborative review combined with multi-round expert annotation, we ensure high quality and comprehensiveness of the data.
Covers 10 mainstream programming languages used in projects:
Quality Assurance Process: GitHub human comments → LLM enhancement → Expert multi-round cross-annotation → Consistency validation
AACR-Bench provides systematic evaluation capabilities across four core dimensions, supporting diverse research and application scenarios:
| 1️⃣ Multi-language Evaluation (10 Languages) | 2️⃣ Positioning Accuracy Evaluation (Line-level) |
|---|---|
| • Cross-language performance comparison: Identify model strengths and weaknesses across languages | |
| • Language-specific optimization: Improve model capabilities for specific languages | |
| • Generalization assessment: Test model's language transfer effectiveness | • Precise positioning: Assess single-line/multi-line issue location accuracy |
| • Cross-file tracking: Test cross-file reference identification capability | |
| • Context boundaries: Verify issue scope judgment accuracy | |
| 3️⃣ Issue Classification Evaluation (4 Categories) | 4️⃣ Context Understanding Evaluation (3 Levels) |
| • Classification accuracy: Assess issue type identification capability | |
| • Severity assessment: Test issue priority judgment | |
| • Specialized capabilities: Analyze detection rates for specific issue types | • Diff-level understanding: Basic code change analysis |
| • File-level understanding: Complete file logic comprehension | |
| • Repo-level understanding: Project-wide dependency analysis |
git clone https://github.com/alibaba/aacr-bench.git
cd aacr-benchThe evaluation framework is in the evaluation/ directory, which provides a unified pipeline: data loading → review execution → evaluation. It supports three reviewers: OpenCodeReview (OCR), Claude Code, and Codex. See evaluation/README.md for detailed setup and usage.
…We employ a multidimensional metric system to comprehensively evaluate code review model performance. For complete metric definitions, calculation methods, and language-specific statistics, please refer to metrics.md.
| Metric | Description | Formula |
|---|---|---|
| Precision | Proportion of valid comments generated | Valid matches / Total generated |
| Recall | Ability to discover annotated issues | Valid matches / Dataset valid count |
| Line Precision | Ability to precisely locate code lines | Line matches / Total generated |
| Noise Rate | Proportion of invalid or incorrect comments | Unmatched / Total generated |
We welcome community contributions! If you want to contribute to AACR-Bench, please follow these steps:
git checkout -b feat/add-new-prs)git commit -m 'feat: add new PRs')git push origin feat/add-new-prs)For detailed contribution guidelines, please refer to CONTRIBUTING.md
| Name | GitHub | Domain | Responsibilities |
|---|---|---|---|
| Zhengfeng Li | @lizhengfeng | Project Lead | Overall architecture design, technical direction |
| Boge Wang | @wangboge | Technical Consultant | Technical Solution Review, technical Guidance |
| Lei Zhang | @zhanglei | Data Architecture | Evaluation framework, metric system, performance optimization |
| Yongda Yu | @yuyongda | Evaluation System | Data schema design, evaluation protocol, quality standards |
| Xinxin Guo | @guoxinxin | Annotation Platform | Annotation system development, workflow design, quality assurance |
| Minghui Yu | @yuminghui | AI Enhancement | LLM annotation pipeline, model selection, prompt optimization |
| Zhengqi Zhuang | @zhuangzhengqi | Engineering | CI/CD pipeline, automated testing, deployment scripts |
This project is licensed under the Apache License 2.0. For details, please see the LICENSE file.
If you use AACR-Bench in your research, please cite our paper:
@misc{zhang2026aacrbenchevaluatingautomaticcode,
title={AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context},
author={Lei Zhang and Yongda Yu and Minghui Yu and Xinxin Guo and Zhengqi Zhuang and Guoping Rong and Dong Shao and Haifeng Shen and Hongyu Kuang and Zhengfeng Li and Boge Wang and Guoan Zhang and Bangyu Xiang and Xiaobin Xu},
year={2026},
eprint={2601.19494},
archivePrefix={arXiv},
primaryClass={cs.SE},
url={https://arxiv.org/abs/2601.19494},
}暂无开放 Issues,或尚未同步最近议题。