Comparing Jev, Gemini, and OpenAI decision models

Mjolnir, the agent tool from BrokkAi, selected Fastino Labs' GLiDE model as its GitHub event relevance classifier after a bakeoff against Jev, Google's Gemini Decisions and OpenAI, with GLiDE delivering the best results for the use case at one of the cheapest prices. Author Jonathan Ellis reported that Jev's recall fell from 0.91 against labels built from its own verdicts to 0.57 against an independent panel, and that relevant pairs make up 11% of traffic. GLiDE, which is open-ish weights with commercial use requiring a separate agreement above $10M annual revenue, was the second-slowest service tested, while Gemini and OpenAI were fastest.

- Written by - Jonathan Ellis https://blog.slopcop.com/author/jonathan/ - Published - By email - Subscribe /portal/signup One of the ways that Mjolnir https://github.com/BrokkAi/mjolnir/?ref=blog.slopcop.com makes your agents smarter is by delivering relevant github events to them so they stay looped in to what's happening with the rest of the team. Some of these can be done deterministically: activity on your own issues or PRs second entry from the top here is always considered relevant. But what if a teammate opens an issue or a PR that is related to what you're working on? We want to surface those, too. For events that we can't classify programatically, Mjolnir uses a classifier to determine relevance. Provided free out of the box via a SlopCop-hosted proxy, with a local-sized "almost as good as GLiDE" model coming soon for those who want to keep their data completely private. We started with Jev, because it was there first, but when Google released Gemini Decisions I decided to perform a bakeoff to see if we could do better. We ended up choosing GLiDE from Fastino Labs https://www.fastino.ai/?ref=blog.slopcop.com for our revision. As a bonus, not only does it give the best results for our use case, it's also one of the cheapest options: It's also open-ish weights https://huggingface.co/fastino/GLiDE-27B?ref=blog.slopcop.com . Commercial use requires a separate agreement if your annual revenue is over $10M. The only real ding against GLiDE is that it was the second-slowest service tested; this is where Gemini and OpenAI most excelled. I guess having some of the top inference-speed experts in the world on tap, with massive hardware footprints, is worth something Lessons learned I asked Fable to go through our eval notebooks for generalizable lessons: 1. Never use the incumbent's verdicts as truth, or labels built from them. In our first label set, a strong adjudicator saw a pair only when the first labeller disagreed with Jev; most pairs where both said no were never checked. Against those labels Jev's recall looked like 0.91; against an independent panel it is 0.57. Any truth that is filtered through the incumbent's answer flatters the incumbent. 2. Give the judges more context than the contenders, from more than one vendor. Our judges read the whole session transcript; the contenders see only the bounded production evidence. Two judges from different labs, plus a third model that breaks ties after seeing both rationales unattributed, keeps one model family's quirks out of the truth. 3. Sample by stratum, weight back to the population, and stratify on something that isn't competing. Relevant pairs are 11% of traffic, so a uniform sample wastes most labels on easy negatives. We stratified on the incumbent's verdict and on an independent reranker score Voyage, not a contender , then weighted each pair by its stratum's size. Stratifying on a contender's own score would bias the comparison toward it. 4. When building your dataset, record why a pair is relevant, not only whether. The judges' "own / dependency / adjacent" field showed that every model loses most of its recall on dependency items. That changed the follow-up work from choosing a model to improving evidence. My takeaway If you don't have the time or the data to perform a proper eval, Jev is a good place to start: inexpensive, smart, and reasonably fast. Just know that you can probably do better depending on your domain and your priorities, just like with coding models https://slopcop.com/power-ranking?ref=blog.slopcop.com .