AI PR review catch rate: how accurate is automated code review in 2026?

Buyers comparing AI pull request review tools rarely see the same number twice. One vendor reports 92% catch rate, the next reports 67% on 'real-world PRs', and a third quotes precision without naming the corpus. The published numbers do not lie, but they measure different things on different stacks. Here is what AI PR review catch rate actually measures, how benchmark corpora are built, where published numbers mislead, and the five-question checklist to run before paying for automated code review.

What 'AI pull request review catch rate' actually measures

AI PR review catch rate is the share of real defects an automated reviewer flags out of the total known defects in a labelled PR corpus, measured against a ground-truth set a human reviewer wrote first. The metric splits into precision (of everything flagged, how much was actually a defect) and recall (of everything real, how much the tool caught). A high-precision, low-recall tool shouts only when it is right; a high-recall tool catches more but burns reviewer attention on false positives. Published benchmarks usually report precision first because labelled corpora measure precision more cheaply. The metric a buyer should weight is recall per finding-type and per language — where the marginal review attention actually lives.

How benchmark corpora label ground truth, and why that matters

A labelled PR corpus is a list of merged PRs where one reviewer set marked which lines held a defect and another judged whether an AI tool caught it. The catch rate depends almost entirely on how that ground-truth set was built: who labelled it, what counts as a defect, and how disagreements were resolved. Some corpora use only deterministic-rule defects (missing await, unused imports) that linters already catch with high precision — those make any AI tool look near-perfect. Others stress cross-file logic flaws linters cannot see, where AI review actually earns its claim. Per-language stratification matters equally: a model at 80% on a TypeScript-only corpus can fall to 40% on a Python-only one. A benchmark that does not name its language mix is hiding where it loses.

The published numbers — and the baskets they sit in

Open-source benchmark suites (SWE-bench, Defects4J reproductions) report precision in the 70-90% band because they rest on labelled fix datasets and deterministic test cases. Commercial vendor reports sit closer to the 50-70% band on real-world PRs, often with recall below 30%. Model-tier differences are smaller than corpus-tier ones: moving from one frontier model to another shifts catch rate by 5-10 points, while switching from a TypeScript-heavy lab corpus to a mixed-language production corpus halves the number. Security-specific catch rate is its own sub-bucket, and the hardest to measure because labelled vulnerability corpora are scarce. Most published AI PR review numbers describe precision heavily because labelled corpora are easiest to build for precision; that is the basket every buyer should name before quoting a number.

Where benchmark numbers mislead a buyer evaluating automated code review

A catch rate number without the false-positive rate next to it is a buyer trap. A reviewer that flags 90% of real defects but also fires on 60% of stylistic differences costs an engineer twenty minutes per PR; a reviewer that flags 70% and rarely fires on stylistic differences costs two. The metric that ends up mattering is operating cost — how much reviewer attention does an automated review reclaim per PR after subtracting time spent triaging false positives. Most published benchmarks do not report false-positive rate at all because labelling FP cost is harder than labelling defect cost. The right question is what an average PR costs you in review time after the automated pass runs, not what the headline catch rate is.

The five-question checklist before you pay for automated PR review

Run five questions before committing to any AI PR review vendor. First, on what corpus was the published catch rate measured — a TypeScript-heavy corpus holds no signal for a Python shop. Second, ask for recall broken out per finding-type and per language, not a single number; precision-only benchmarks miss where the model fails. Third, ask for the false-positive burn in review-time minutes per PR, not percentage points — that is the metric that scales. Fourth, ask how the tool ships a fix: comment only, or a follow-up commit on the same branch? Fifth, ask whether webhook delivery is HMAC-verified per install — an agent reading your PR diff is a privileged surface. Tools that answer all five clearly are worth a trial; tools that answer only catch rate are selling a headline.