# Comparing Jev, Gemini, and OpenAI decision models

> Source: <https://blog.slopcop.com/comparing-jev-gemini-and-openai-decision-models/>
> Published: 2026-11-12 18:56:45+00:00

- Written by
- [Jonathan Ellis](https://blog.slopcop.com/author/jonathan/)
- Published
- By email
- [Subscribe](#/portal/signup)

One of the ways that [Mjolnir](https://github.com/BrokkAi/mjolnir/?ref=blog.slopcop.com) makes your agents smarter is by delivering relevant github events to them so they stay looped in to what's happening with the rest of the team.

Some of these can be done deterministically: activity on your own issues or PRs (second entry from the top here) is always considered relevant. But what if a teammate opens an issue or a PR that is related to what you're working on? We want to surface those, too.

For events that we can't classify programatically, Mjolnir uses a classifier to determine relevance. (Provided free out of the box via a SlopCop-hosted proxy, with a local-sized "almost as good as GLiDE" model coming soon for those who want to keep their data completely private.) We started with Jev, because it was there first, but when Google released Gemini Decisions I decided to perform a bakeoff to see if we could do better.

We ended up choosing GLiDE from [Fastino Labs](https://www.fastino.ai/?ref=blog.slopcop.com) for our revision. As a bonus, not only does it give the best results for our use case, it's also one of the cheapest options:

It's also [open-ish weights](https://huggingface.co/fastino/GLiDE-27B?ref=blog.slopcop.com). (Commercial use requires a separate agreement if your annual revenue is over $10M.)

The only real ding against GLiDE is that it was the second-slowest service tested; this is where Gemini and OpenAI most excelled. I guess having some of the top inference-speed experts in the world on tap, with massive hardware footprints, is worth something!

## Lessons learned

I asked Fable to go through our eval notebooks for generalizable lessons:

1. **Never use the incumbent's verdicts as truth, or labels built from them.** In our first label set, a strong adjudicator saw a pair only when the first labeller disagreed with Jev; most pairs (where both said no) were never checked. Against those labels Jev's recall looked like 0.91; against an independent panel it is 0.57. Any truth that is filtered through the incumbent's answer flatters the incumbent.
2. **Give the judges more context than the contenders, from more than one vendor.** Our judges read the whole session transcript; the contenders see only the bounded production evidence. Two judges from different labs, plus a third model that breaks ties after seeing both rationales unattributed, keeps one model family's quirks out of the truth.
3. **Sample by stratum, weight back to the population, and stratify on something that isn't competing.** Relevant pairs are 11% of traffic, so a uniform sample wastes most labels on easy negatives. We stratified on the incumbent's verdict and on an independent reranker score (Voyage, not a contender), then weighted each pair by its stratum's size. Stratifying on a contender's own score would bias the comparison toward it.
4. **When building your dataset, record why a pair is relevant, not only whether.** The judges' "own / dependency / adjacent" field showed that every model loses most of its recall on dependency items. That changed the follow-up work from choosing a model to improving evidence.

## My takeaway

If you don't have the time or the data to perform a proper eval, Jev is a good place to start: inexpensive, smart, and reasonably fast. Just know that you can probably do better depending on your domain and your priorities, [just like with coding models](https://slopcop.com/power-ranking?ref=blog.slopcop.com).