Cheap LLM code review is fine until it hits an authorization bug

A code review vendor's evaluation of 50 public benchmark pull requests from Cal.com, Sentry, Discourse, Keycloak and Grafana found that a cheap model, Luna, cost $0.0030 per verified bug versus $0.061 for the frontier model Astra, a 20x gap, but caught only 9 of 24 security bugs compared with Astra's 19. On the Keycloak IAM server, Luna found 6 verified bugs to Astra's 14, with only half its findings holding up against 93% for Astra. The vendor recommends routing routine changes to cheap models while sending authentication and permission logic to a more trusted model.

A code review vendor ran its own eval comparing a $1.20-per-million-output-token model Luna against a frontier one Astra across 50 public benchmark pull requests from Cal.com, Sentry, Discourse, Keycloak and Grafana. The numbers are worth reading cold, because they show exactly where the cheap model falls apart. Luna found 69 verified bugs for a total run cost of $0.20. Astra found 92 for $5.66. Per verified bug that is $0.0030 against $0.061, a 20x gap. On everyday data and logic bugs the cheap model kept up: 39 verified against Astra's 47. On concurrency, 10 against 13. Those are the stats that justify routing every PR to the cheap model and calling it a day. Then look at the security split. Luna caught 9 of the 24 security bugs. Astra caught 19. And on Keycloak, the IAM server, Luna found 6 verified bugs to Astra's 14, with only half its findings holding up against 93% for Astra. That is not a gently declining curve. It is a cliff on exactly the code you cannot afford to review wrong. The two Keycloak bugs Astra caught and Luna missed are the useful part. Federated recovery codes were never marked as used, so a recovery code could be reused indefinitely. And a new global view permission silently overrode denials set on individual clients. Neither one looks wrong on a single line. You only see them by reasoning about the permission model as a whole after the change. A small model scanning for local defects is structurally blind to that class. The other number worth quoting is precision. About one in four of Luna's findings was wrong 24 of 93 , versus 4 of 96 for Astra. Cost per bug is not the only thing that burns a team's review budget. Noise does too, because reviewers who already skim AI comments start skipping them entirely once a quarter are false positives, and then the real catches stop landing at all. Some caveats before anyone treats this as gospel. Astra is itself one of the two judges that verified the findings, so it could be grading itself with a soft touch. And the benchmark PRs all predate both models' training cutoffs, so public-history recall is a real risk the authors flag. I'd read the absolute magnitudes skeptically, but the shape of the result is robust: the cheap model is a fine triage layer on routine changes and the wrong call on authentication and permission code. The practical version of this is pretty simple. Let the cheap model flag everyday busywork bugs and cut your cost ceiling. Route anything that touches authz, authn, or permission logic to a model you trust more, and read that output like it's unpaid. The difference is not in what the model costs per token. It is in what a missed permission-bypass bug costs you downstream. That split is the real takeaway, and it is cheaper than any model routing: decide a small set of high-risk paths before you adopt an AI reviewer, and never let the price per review be the deciding input on those.