# Jev vs. Kev: open-source Jev alternative tested side by side > Source: > Published: 2026-09-25 17:16:20+00:00 # Jev vs. Kev: an open-source Jev alternative, tested side by side By Jose Sabater - TypeSafe launched [Jev](https://opper.ai/typesafe/jev-1-13-0) on 15 September. It is a new kind of model: instead of writing text, it reads a piece of state and answers typed questions (yes/no, pick one, score on a rubric) with a probability for every option. TypeSafe has not published the architecture or the training data. Within a week there were around 30 open reproductions, rebuilt from the API contract alone. We wanted to know how close one of those copies actually gets. So we took [Kev 4B](https://huggingface.co/jaredpalmer/kev-4b), an open model by Jared Palmer and one of the most discussed reproductions, hosted it ourselves in the EU, and put it behind the same endpoint Jev already has on Opper. Then we asked both models the same questions. **What we found, in short:** - **Accuracy is close.** On 362 questions written after both models were released, the two land within 2 points of each other on every task, which is inside the noise for samples this size. - **They count tokens very differently.** Same request, same list price, but Jev counts a fixed ~257 extra input tokens on every request. - **Speed is similar.** Through Opper, Kev answered in about 220 ms and Jev in about 275 ms. ## One endpoint, one string changed Kev speaks the same wire format as Jev, so the comparison needs nothing more than a different model name: ``` curl https://api.opper.ai/v3/compat/v1/systemone \ -H "Authorization: Bearer $OPPER_API_KEY" \ -H 'Content-Type: application/json' \ -d '{ "model": "opper/kev-4b", "state": {"message": "I was charged twice for my order and nobody has replied."}, "questions": { "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "Billing", "support": "Tech support", "sales": "Sales"}} } }' ``` Swap `opper/kev-4b` for `typesafe/jev-latest` and nothing else changes: the request, the response shape, the probabilities and the per-call cost header. ## What Kev is Kev 4B ([model card on Hugging Face](https://huggingface.co/jaredpalmer/kev-4b), [code on GitHub](https://github.com/jaredpalmer/kev)) is a fine-tune by Jared Palmer of Alibaba's [Qwen3.5-4B-Base](https://huggingface.co/Qwen/Qwen3.5-4B-Base) with a small head that reads the answer from a single forward pass. It is Apache-2.0 and returns a probability for every option. We host it in the EU. ## Why we built our own test set The natural choice for a benchmark would have been well-known public datasets. The problem is that Kev was trained on many of them: AG News, Banking77, BoolQ, MNLI, SST-5, Yelp reviews, IMDb, Amazon reviews, DBpedia, TREC, ARC, OpenBookQA and CommonsenseQA. We tried a few of them anyway, and as expected, Kev matched or beat Jev on most. So we built a small new dataset from text published after both models were released: new arXiv papers, Stack Exchange questions and GitHub issues. Neither model could have seen any of it. ## Methodology **Same endpoint, one question per request.** Every item was sent to both models through `POST /v3/compat/v1/systemone` on Opper with an identical body. Only `model` differed: `typesafe/jev-latest` or `opper/kev-4b`. Each request carried a single question, 8 requests were in flight at a time, and we recorded the answer, the latency and the cost header Opper returns on every call. **Data neither model could have seen.** The headline numbers come from 362 items published after 2026-09-20, after Jev 1.13 (released 2026-09-15) and Kev 4B were both out. The ground truth comes from the source, never from a model: | Task | Source | Question | Answer from | n | |---|---|---|---|---| | arXiv category | [arXiv API](https://info.arxiv.org/help/api/index.html) , newest papers in 8 categories | "Which arXiv category is this paper's primary subject?" (8 options) | The paper's primary category | 160 | | Stack Exchange site | [Stack Exchange API](https://api.stackexchange.com/) , newest questions on 6 sites | "Which Q&A community was this question posted on?" (6 options) | The site it was posted on | 120 | | GitHub bug or feature | New issues in VS Code, Godot, Rust, Windows Terminal, Ollama, Flutter | "Is this issue a bug report (as opposed to a feature request)?" (yes/no) | The maintainers' label | 82 | Categories and sites were balanced (20 items each), GitHub bugs were sampled down to match the 41 feature requests, and the set was frozen with a fixed seed so every model saw the same items. Two real examples: ``` {"model": "opper/kev-4b", "state": {"title": "Is there any function you can integrate with respect to sqrt(dx) and get a finite nonzero result?", "body": "Messing around trying to build intuition for manipulating differentials, I had the idea to ..."}, "questions": {"q": {"type": "choice", "instructions": "Which Q&A community was this question posted on?", "criteria": {"math": "Mathematics", "physics": "Physics", "askubuntu": "Ask Ubuntu (Ubuntu Linux)", "superuser": "Super User (general computing)", "electronics": "Electrical engineering and electronics", "diy": "Home improvement (DIY)"}}}} ``` Posted on Mathematics. Jev answered `math` at 1.00, Kev `math` at 0.93. On [a Rust issue](https://github.com/rust-lang/rust/issues/163145) titled "`f16b.rs` fails on LoongArch", labelled a bug by the maintainers, Jev said bug at 0.97 and Kev hedged at 0.48, just on the wrong side of the line. **Scoring.** Accuracy counts the top answer (yes/no at 0.5). Calibration error compares stated confidence with the actual hit rate across 10 bins; lower is better. Cost per 1,000 calls is the average of Opper's per-call cost header times 1,000, at list price when we ran it (September 2026): USD 0.042 per million input tokens on both models, output free. The secondary check uses the five older datasets Kev did not train on ([emotion](https://huggingface.co/datasets/dair-ai/emotion), [offensive tweets](https://huggingface.co/datasets/cardiffnlp/tweet_eval), [QNLI](https://huggingface.co/datasets/nyu-mll/glue), [PAWS](https://huggingface.co/datasets/google-research-datasets/paws), [SciQ](https://huggingface.co/datasets/allenai/sciq)). ## Accuracy and confidence | Task | Jev accuracy | Kev accuracy | Jev calibration error | Kev calibration error | Jev cost per 1k calls | Kev cost per 1k calls | |---|---|---|---|---|---|---| | arXiv category (8 options, n=160) | 96.9% | 95.0% | 0.032 | 0.044 | $0.030 | $0.015 | | Stack Exchange site (6 options, n=120) | 97.5% | 98.3% | 0.027 | 0.125 | $0.026 | $0.012 | | GitHub bug or feature (yes/no, n=82) | 95.1% | 93.9% | 0.049 | 0.137 | $0.025 | $0.013 | Accuracy is within a couple of points either way, which is inside the noise for samples this size. The clearer difference is calibration: Jev's stated probabilities track how often it is actually right, while Kev's drift further, most on the Stack Exchange and GitHub tasks. On the five older public datasets, Jev led clearly on paraphrase detection (PAWS, 87.0% vs 74.5%) and slightly on QNLI and SciQ; Kev was ahead on emotion and offensive tweets. Kev never trained on these, but its authors used them to pick their best training run, so we read them as a secondary signal. ## The token accounting surprise Both models listed at the same price when we ran this (September 2026), USD 0.042 per million input tokens with free output, and receive the identical request. The difference is in what gets counted. We sent the same requests to TypeSafe directly and through Opper and got identical counts, so this is Jev's own accounting, not ours. Growing the text adds the same number of tokens on both models, but Jev adds a fixed charge of about 257 input tokens to every request, most likely its internal prompt. A one-line message with one question counts as 280 tokens on Jev and 23 on Kev. So the gap in cost per call depends entirely on input length: | Request | Jev tokens | Kev tokens | Jev cost vs Kev | |---|---|---|---| | One-line message, 1 question | 280 | 23 | 12x | | arXiv abstract, 1 question | about 720 | about 360 | 2x | | Long message (about 40 sentences), 1 question | 960 | 703 | 1.4x | | Long message, 5 questions in one request | 1,030 | 777 | 1.3x | The overhead is per request, not per question, so asking several questions about the same text in one request spreads it out. Our benchmark sent one question per request, which is the least favourable case for Jev. ## Speed Sent one at a time through Opper from Stockholm, Kev answered in about 180 to 200 ms and Jev in about 265 ms. With 8 requests in flight, Kev's median was about 220 ms and Jev's about 275 ms. For both models most of that time is the network trip rather than the model itself. Kev is scaled down when idle, so the first call after a quiet period is slower. ## What we take from it A week-old open reproduction, built without access to Jev's architecture or training data, gets surprisingly close on straightforward classification. Where it falls short is exactly where you would expect a small model trained on a few public datasets to fall short: calibrated confidence, subtle meaning and long documents. And comparing the two turned up something we would not have found otherwise: the per-request overhead in Jev's token count. Both models are on Opper behind the same endpoint, so you can run your own questions against both and see which one fits. ## Caveats - One run per suite, a few hundred items. Differences under about 5 points are within noise. - The fresh tasks are fairly easy for both models (95% and up), which compresses the differences. Harder fresh tasks would separate them more. - New data is not new tasks: Kev has trained on topic, intent and yes/no classification, even though it never saw these items. Jev's training data is not published. - On the older datasets, Kev's authors used emotion, offensive tweets, QNLI, PAWS and SciQ to choose between training runs, so Kev's results there are not fully independent. - Kev was trained on short inputs (states of up to 384 tokens) and misses details buried deep in long documents. - Latency was measured from one location, through Opper. ## Links - Jev: [docs.typesafe.ai](https://docs.typesafe.ai/models) and[on Opper](https://opper.ai/typesafe/jev-1-13-0) - Kev 4B: [weights on Hugging Face](https://huggingface.co/jaredpalmer/kev-4b) ,[code](https://github.com/jaredpalmer/kev) ,[on Opper](https://opper.ai/community/kev-4b) - Base model: [Qwen3.5-4B-Base](https://huggingface.co/Qwen/Qwen3.5-4B-Base) by Alibaba - The endpoint: `POST https://api.opper.ai/v3/compat/v1/systemone` ## Reproduce it The benchmark code, the list of test items and our results are at [github.com/opper-ai/jev-vs-kev](https://github.com/opper-ai/jev-vs-kev). With an Opper API key, run `python3 fetch_texts.py` and then `SUITE=fresh python3 jev_vs_kev.py`.