Code has become extremely cheap to produce, if you have a 5x claude subscription, you can produce or at least try to produce any piece of software you want. And, to be honest, I actually like that, I feel more people should build their own tools and release them as open source because everyone gets to turn their ideas into real products, for their own use if not for others.
But this also means it is getting easier to copy a product, put it on public GitHub and announce you've killed the paid version. Being open source shouldn't give you a pass for shipping slop. If you are calling your tool a replacement for something people pay for, show how it compares.
In the past week, each time I opened X, I kept seeing ads about a new open source code review tool claiming to be the "CodeRabbit killer". I have used CodeRabbit in the past and it definitely has its shortcomings, but if you're claiming you've killed a product, there needs to be evidence.
I saw the advertisement so many times that I decided to evaluate how the product actually performs, it also gave me an excuse to add review capabilities into my own open source tool called shadowclone which helps me manage my agent skills and ensure that regardless of the harness or model I use, agents consistently follow my engineering preferences.
I ran an eval using Greptile, OpenQodex (the open source tool I wanted to evaluate), and my own implementation of a code reviewer in shadowclone, which is basically a code review skill, a local reviewer and a Github bot that can review on the cloud.
My bot reported a higher proportion of findings that both model judges considered real, but it missed 4 real bugs that Greptile caught and as for OpenQodex, I will let the results speak for themselves.
- Greptile found 69% of the known bugs and 68% of its findings were judged as real 2. shadowclone cloud bot found 38% and 96% of the findings were judged as real 3. shadowclone locally found 36% and 91% of the findings were judged as real 4. OpenQodex found 31% and 86% of its findings were judged as real
The local reviewers ran 3 times per PR. You can see the full eval results at https://github.com/theonly1me/shadowclone/tree/main/evals/code-review
The harness is open source too and you can re-run the eval if you like.
I still want more people to build and release their own tools. I am doing it too. But, if you advertise yours as a replacement for an existing product, I want to see what you tested and where it falls short. Mine missed bugs Greptile caught and publishing that gives me something useful to work on, calling it a Greptile killer wouldn't.
Comments URL: [https://news.ycombinator.com/item?id=50043719](https://news.ycombinator.com/item?id=50043719)
Points: 2