{"slug": "you-re-paying-too-much-for-code-review", "title": "You're paying too much for code review", "summary": "A comparison of five AI code-review tools across 95 pushes of 47 pull requests on Jeff Dickey's Rust cache project Mr. Boxington found that all systems surfaced material problems without excessive nitpicking, and that Mjolnir running vanilla DeepSeek Flash (high effort) matched Greptile's performance at roughly 1/20 the price. The test, run by Brokk AI, covered Greptile, CodeRabbit, Cursor Bugbot, open-source Kodus and Mjolnir's in-harness reviews; Kodus, also powered by DeepSeek Flash but with far more added context and structure, performed worse than Mjolnir's simple prompt lifted from Codex's /review command. The author concluded that paying more does not buy better reviews — DeepSeek is \"freakishly good at code review\" — and that running multiple parallel reviews and unioning the findings recovers defects individual reviewers miss, though with diminishing returns.", "body_md": "# You're paying too much for code review\n\n[I wrote in August](https://blog.brokk.ai/its-time-to-rip-off-the-band-aid-and-stop-performing-code-reviews/) that humans shouldn't spend their time reading code anymore. (TLDR: the opportunity cost isn't worth it.)\n\nThat doesn't mean that you should dispense with code review entirely. It just means that you should have agents perform the reviews (as well as other forms of agentic QA). But which agents? There are more companies offering hosted code review than you can shake a stick at.\n\nSo I took the unscientific but popular approach of examining the data I already had close at hand; specifically, PRs submitted to Jeff Dickey's Rust cache [Mr. Boxington](https://github.com/jdx/mr-boxington/?ref=blog.brokk.ai) (which [every developer working in Rust should use](https://blog.brokk.ai/infrastructure-for-agentic-rust/)). \n\nJeff has three reviewer products wired up: Greptile, CodeRabbit, and Cursor Bugbot, all of which have generous free plans for open source projects. I also tested the open source Kodus reviewer, which offers a low cost bring-your-own-keys hosted option as well as host-your-own, and [Mjolnir](https://github.com/BrokkAi/mjolnir/?ref=blog.brokk.ai)'s [in-harness reviews](https://blog.brokk.ai/mjolnir-automated-cross-vendor-adversarial-review/). \n\nHere's how they did across 95 pushes of 47 PRs (that is, on average two pushes of each PR were reviewed before merge):\n\nA \"material finding\" means primarily incorrect, user-observable behavior, but also performance regressions or tests that don't test what they purport to test. False positives include \"the problem was deemed not important to fix\" as well as actual incorrect findings.\n\nIn short: all systems are able to find material problems without drowning in nitpickery; that problem from six months ago is solved.\n\n### Simpler can be better\n\nMjolnir and Kodus are both powered by DeepSeek Flash (high effort), but Kodus is throwing in a ton of extra context and structure while Mjolnir is just using vanilla DeepSeek Harness with [a prompt lifted from Codex's /review command](https://github.com/openai/codex/blob/main/codex-rs/prompts/templates/review/rubric.md?ref=blog.brokk.ai). \n\nWhile code diving Kodus to understand how it made DSF so much worse, [Opus and I found a bunch of issues](https://github.com/kodustech/kodus-ai/issues?q=is%3Aissue+state%3Aopen+author%3Ajbellis&ref=blog.brokk.ai), but nothing that obviously looks like \"if only this were fixed, Kodus would outperform the simpler design.\"\n\nTurns out a smart model armed with bash is pretty hard to beat for code review as well as for authoring. (Mjolnir also had a more complex reviewer with multiple subagents and fancy tools until about a month ago; in early 2026 this improved review quality, but re-testing with modern models failed to find any improvement over the simple approach tested here.)\n\n### Simpler can be cheaper\n\nMjolnir-powered-by-DSF performs roughly equally to Greptile at about 1/20 of the price. Someone should wire this up as a service for teams who want reviews to run in CI instead of their coding harnesses.\n\n### No reviewer is perfect\n\nEach reviewer found defects that the others did not. It turns out that you can recover many of these with multiple passes of the same reviewer as well; here is what that looks like for the Mjolnir reviews:\n\nClearly there's diminishing returns, but \"run multiple reviews in parallel and union the findings\" seems like an underappreciated option. What if you have money to burn?\n\nI've shown DeepSeek Flash as the Mjolnir (and Kodus) reviewer model because of its unique place on [the cost:intelligence spectrum when you're paying API prices](https://slopcop.com/power-ranking?ref=blog.brokk.ai), but can you get better reviews if you're willing to pay more?\n\nIt turns out that: no, you can't: DeepSeek is freakishly good at code review. But if you're willing to accept slightly fewer bugs spotted, Terra is 6x faster as well as meaningfully less expensive. (Because DeepSeek's massive advantage in cache hit pricing is less meaningful in a relatively brief code review session, while Terra's output-token-frugality is more meaningful.)\n\n## Conclusion\n\nAdding agentic review to your development process will almost certainly catch real bugs for minimal effort; you can pay a bunch of money for a hosted service, or you can use Codex `/review` for results that are just as good and a lot cheaper. If you don't use Codex, or if you want your harness to automatically perform a review before pushing, try out [Mjolnir](https://github.com/BrokkAi/mjolnir/?ref=blog.brokk.ai). After this evaluation, we've updated its defaults to two parallel reviews and its Auto model priority order to Terra > DSF > Haiku.\n\n### Appendix: How this evolved / what I'd do differently\n\n1. I did almost all of the work before Terra was released. Then when it came out, I added it to the test mix, and it turned out to be really good. Terra is now the highest priority review model in Mjolnir, but I didn't go back and rebuild the main mj and Kodus comparisons on Terra.\n2. I was sloppy with my data mining and did all the mj testing against the 47 PRs from the (at the time) most recent 30 days of mbx activity before realizing that Bugbot only had results on some of those. (It's enabled for Jeff's account only.) So ideally I would rerun all the other models against the Bugbot set; instead, I just ran the Greptile and CodeRabbit live-data-eval code against the Bugbot ticket and used that commonality to extrapolate \"here's how Bugbot would have probably done\" on the \"main\" 47.\n3. The first mj/DSF review found 43 material defects, so that's the number I used both in the first comparison chart and the diminishing-returns chart. But 43 is unusually low, that run just got unlucky, and using the median of 52 would be more correct.", "url": "https://wpnews.pro/news/you-re-paying-too-much-for-code-review", "canonical_source": "https://blog.brokk.ai/youre-paying-too-much-for-code-review/", "published_at": "2026-11-03 18:48:20+00:00", "updated_at": "2026-11-03 18:50:16.968511+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "large-language-models", "ai-products"], "entities": ["Brokk AI", "Mjolnir", "DeepSeek Flash", "Greptile", "CodeRabbit", "Cursor Bugbot", "Kodus", "Jeff Dickey"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/you-re-paying-too-much-for-code-review", "markdown": "https://wpnews.pro/news/you-re-paying-too-much-for-code-review.md", "text": "https://wpnews.pro/news/you-re-paying-too-much-for-code-review.txt", "jsonld": "https://wpnews.pro/news/you-re-paying-too-much-for-code-review.jsonld"}}