Measured quality

What ocra's evaluation runs have measured so far, how, and the limits. The sample is small, so read Limits before relying on a number.

How it is measuredCopy Anchor Link

ocra-eval runs the real ocra review on changes pinned at fixed commits and scores what it reports (Evaluation). Two datasets:

• Golden cases (ADR-0011) belong to ocra. Each pins a public repository at two commits and lists the issues ocra must report, other places the same issue may rightly be reported, ranges where a finding would be wrong, or that the change has no issue. Every run so far used the 8 cases the smoke tier had then, now with 16 expected issues. The smoke tier has since grown to 10 cases and the full tier to 16; neither has been run as it is now.
• AACR-Bench is an external benchmark of real pull requests whose ground truth is the comments human reviewers left; it is the number comparable with published results.

On golden cases, a finding matches an expected issue when it is in the same file, at most one line from the expected lines or one of the other places, and a judge model says it is the same concern.

• Recall is the share of expected issues found at or above the case's severity.
• Precision is the share of reported findings that match an expected issue or are labeled valid. A finding matching nothing is checked against the code once and labeled valid or invalid with the case; until then it counts as wrong.
• A critical finding in a range marked wrong, or on a change with no issue, is a failure. No run has had one.

AACR-Bench uses the benchmark's own matching: same file and side of the diff, at most one line apart, then the judge model. Its precision is matches over findings, without labels.

Every run until 2026-10-01 used Gemini on Vertex AI: gemini-3.5-flash for the reviewers and verification (gemini-3.5-flash-lite as failback), gemini-3.1-pro-preview for the judge step, gemini-3.5-flash-lite for light tasks, and gemini-flash-lite-latest to decide matches when scoring. Two later runs used a free model for every step (On a free model). Costs are ocra's own accounting: token counts at the models' list prices.

Golden casesCopy Anchor Link

The current configurationCopy Anchor Link

One run on 2026-09-29 (UTC), with ocra's default settings as of main at 664c850 plus the 30-step limit per review agent merged right after it (f966d83):

• Found 7 of the 16 expected issues (44%): 5 of the 6 the cases rate as warnings, 2 of the 10 rated as suggestions. No case expects a critical issue.
• Reported 7 findings, all correct (7 of 7, 100%), and nothing on the change with no issue.
• Cost $12.53 for the 8 changes: $1.57 each on average (median $1.15, from $0.07 to $3.98). A review took 243 seconds at the median and 437 at the 90th percentile.

Found: header declarations and callers out of step with their definitions, breaking the build (FreeCAD, C++); a test's monitor container left on the public image when its server container moved to a mirror (ASP.NET Core, C#); files created before their parent directory (Ollama, Go); an import of a name the module does not export, breaking the build (RAGFlow, TypeScript); and three bugs from ocra's history: docs promising a .gitignore no code writes, a helper registered as a plugin, and reference-style Markdown links through which model text could still post live links.

Missed: four bugs from ocra's history (a temporary git index copy with a new modification time, so a file edited in the same second can look unchanged and be skipped; a blocking verdict returned before a new coverage check; unchanged files counted as not reviewed; a run counted as reviewing nothing when one reviewer failed on files another had finished), four smaller validation, error-handling and test issues in Ollama, and missing translation keys in RAGFlow.

Where the expected issues come fromCopy Anchor Link

Four of the 16 were written from bugs fixed in ocra's history, before any run; the current configuration found 2 of them. The other 12 were added after an ocra run reported them and a check against the code confirmed them: 5 from runs on AACR-Bench pull requests, 7 from runs on these cases, one of them the run above. On those 12, recall shows whether ocra finds again what it found before; it found 5.

Every runCopy Anchor Link

All runs on those 8 cases, scored against today's cases and labels so they compare. Three earlier runs on the first 4 each found 1 of those cases' 7 expected issues and reported nothing wrong. Runs that only checked mechanics, or compared a change on a few AACR-Bench pull requests, are left out.

¹ The 16th expected issue was taken from this run.

² One more expected issue was found below the severity its case asks for.

Both configurations that ran twice found about the same number each time (6 and 6, 7 and 8), with precision one finding apart. Two runs cannot show the real spread: treat a difference of one or two issues as noise.

On a free modelCopy Anchor Link

Two runs of today's smoke tier (10 cases, 18 expected issues) on Space Bunny Alpha (stealth/space-bunny-alpha), an anonymous free preview on OpenRouter reached as a declared OpenAI-compatible endpoint. The free key reaches no other model, so every tier, verification, the judge and the scoring judge ran on it. ocra was main as of 2026-10-02 (c997e07) with default settings. OpenRouter's free models share one daily quota per account (1000 requests once the account has bought at least $10 of credits, about 50 without; see Providers) and a smoke case takes 30 to 250, so even at 1000 a day each run stretched from 2026-10-02 to 2026-10-04. OpenRouter withdrew the model on 2026-10-06, so these runs cannot be repeated.

³ One more expected issue was found below the severity its case asks for.

Both runs found the same seven issues: FreeCAD's declarations out of step with their definitions, ASP.NET Core's monitor container left on the public image, Keycloak's priority fallthrough, RAGFlow's import of a name the module does not export and its missing translation keys, the .gitignore ocra's docs promised, and the helper registered as a plugin. Both missed the same eleven: all five Ollama issues (nothing reported on that change), the Elasticsearch issue, the git index copy, and four of the five in the coverage-gaps change. Neither reported a critical finding where a case says findings are wrong; the change with no issue got one suggestion in each. Precision differs by four findings on the same seven hits: the 8 and 4 unlabeled findings count as wrong until checked, and three recur in both runs (a read cache without a bound, a FreeCAD comment naming a removed parameter, the manual's example output).

Three caveats beyond the Limits:

• Space Bunny itself decided the matches, not gemini-flash-lite-latest, so these rows compare only with each other.
• The coverage-gaps change was reviewed in part. The daily quota refused some of its tasks (12 of 19 in A, 9 of 18 in B) and the review finished incomplete; its four misses are partly code no reviewer read.
• Two tasks in A and one in B hit the 10-minute task limit on the index-copy change.

Cost was $0 at the model's price of 0. Each run sent 15 to 18 million input tokens, 88% cached, and about half a million output tokens; a change took 231 and 247 seconds at the median, 992 and 803 at the 90th percentile.

AACR-BenchCopy Anchor Link

Ten pull requests (--seed 1 --limit 10 --max-change-lines 300: at most 300 changed lines each, 44 annotated issues), with two configurations run twice each on 2026-09-28, before the current configuration:

The current configuration has not been run on AACR-Bench. When reading these numbers:

• Recall has a ceiling. Of the 530 issues annotated on 94 of its pull requests, 40% are maintainability and readability comments, which ocra does not report by design, so at most 58% are within reach (ocra-eval ceiling, which calls no model).
• Precision is understated. A real issue nobody commented on counts as a false positive. A one-off check of each finding against the code (by an AI agent working on ocra, not recorded as labels) found 6 real defects, 4 minor issues and 1 wrong finding among the 11 findings of the first two runs, where the benchmark matched 6; and 15 real defects, 5 minor issues and 1 wrong finding among the 21 of the last two, where it matched 6.
• Matching allows one line of distance. In one Electron pull request, ocra reported the annotated use-after-free ten lines from the reference, counted as a miss and a false positive. Hence ocra-eval's lenient match (same concern, same file, any line) next to the official one.

CostCopy Anchor Link

In these runs, at list prices:

• with the current configuration, a review of a golden case cost $1.57 on average (median $1.15), from $0.07 to $3.98;
• with earlier configurations, a review of one of AACR-Bench's smaller pull requests cost $0.77 to $1.02 on average;
• the most expensive single review in any run cost $4.31, for a change to 10 files.

Cost grows with the change's size and the reviewers its risk tier runs. --max-cost-usd caps a run's spend (CLI).

LimitsCopy Anchor Link

• Small. 8 changes and 16 expected issues, of which only 4 were written independently of ocra's own findings. The current configuration ran once, and the full tier has not run.
• Not independent. The cases and labels are the project's own, the labels made by an AI agent working on ocra that checked each finding against the code and recorded its reason. A second model (Claude Fable 5.1), not told the labels, re-judged 5 of the 11 labels recorded by then and agreed with all 5; no person outside the project has reviewed them. Recent changes, the 30-step limit among them, were chosen on these cases, which flatters the current configuration.
• A model decides matches: gemini-flash-lite-latest. Its answers are cached, so re-scoring a run gives the same result, but another judge could decide differently.
• Two model families, not comparable. Runs until 2026-10-01 used Gemini on Vertex AI; the two free-model runs used Space Bunny Alpha for every step, scoring included. Other models will score differently.
• Runs vary. Two runs of the same configuration differed by one finding in precision.
• Recall is the weak point. ocra missed more than half of the expected issues here, most of them rated as suggestions, and on AACR-Bench it matched under 10% of the annotated issues.
• Changes since the last run are not measured, among them that an agent stopping before finishing its files is now asked once to continue.
• Not comparable across tools. Other tools publish numbers on other datasets, matching rules and models. These say how ocra did on its own cases, not how it ranks.

What this means for youCopy Anchor Link

ocra finds real bugs and has rarely reported something wrong, but it missed more issues than it found. Use it as a second reviewer next to people, not as a gate; its verdict is advice, not a security check (How it works).

ReproduceCopy Anchor Link

The cases and labels are in evals/golden. With models configured as in Evaluation, a run on the 8 cases above cost $8 to $13 at list prices; the smoke tier now has 10:

node packages/eval/dist/main.js run --dataset golden --tier smoke --label mine --max-cost-usd 20 --pr-max-cost-usd 6

On a free OpenRouter model, add --config <file> with the recipe from Free models through OpenRouter and the judge variables from Evaluation. It costs nothing, but a smoke run exceeds a day's free quota, so it resumes across days with --retry-failed.