GitHub opens ReviewBench, a public benchmark for AI code review: 219 real pull requests, a published judge, and offline scores that track production A/B tests

Why it matters

【Key points】

On October 5 GitHub launched ReviewBench, a public research-preview benchmark for AI code-review agents. It asks how many real problems in a pull request a reviewer catches and how much noise it adds. The dataset, rubric, judge configuration and a self-serve runner are all public, so any team can score its own agent and submit to the leaderboard.

【Verifiable】

GitHub profiled 103.9 million real pull requests, then picked 219 PRs from 187 open-source repos across 19 languages, weighting toward substantive multi-file changes. Ground truth combines human review comments, issues inferred from follow-up fix commits, static analysis and several frontier LLMs, deduplicated and judged by Claude Sonnet 5 under one rubric. Senior engineers who did not build the set re-labeled every finding and agreed 96.6% of the time. Grounded metrics use only known findings; augmented metrics credit valid issues outside the golden set. GitHub says offline results have matched the direction of later production A/B tests; in one multi-model ensemble experiment, addressed rate rose 8.0%, recall 13.6%, comment volume 61%, and cost per review fell 8.0%.

【Our read】

AI now writes code faster than people can review it, and review has lacked a public, reproducible yardstick. Publishing the judge and rubric makes vendor claims checkable. Caveats: the benchmark comes from Copilot's own vendor, leaderboard entries need maintainer approval, and findings are judged by an LLM; a third-party tally of the public files says human reviewers supplied only about 1.5% of true positives. A third-party reading of the leaderboard on October 6 put the best agent at roughly a quarter of known issues found, so AI review is a precise supplement, not yet a replacement for human review.