JevBench, a reproducible benchmark for typed decision models

TypeSafe AI's Jev 1.13.0 topped the JevBench leaderboard for typed decision models with a score of 74.4, ahead of Theodore Lee's SemIf (73.1) and Maisa's djev (73.0), according to the published benchmark table. Jev 1.13.0 posted 85.7 on one subscore, 82.7 on another, 83.3 on a third, 52.0 on a fourth, a cost of $0.040, and 0.65 s raw latency with 0.72 s raw p95 on a production API. OpenAI's GPT-5.6 Luna, run at low reasoning effort, ranked 14th at 65.9 with a $0.242 cost, the highest listed price in the table.

| 1 | by TypeSafe AI Jev 1.13.0 https://docs.typesafe.ai | 74.4 | 85.7 | 82.7 | 83.3 | 52.0 | $0.040 | 100.0% | 99.0% | 94.5% | 74.1% | 0.65 s rawp95 0.72 s raw | production API | |---|---|---|---|---|---|---|---|---|---|---|---|---|---| | 2 | by Theodore Lee TheoLeeCJ SemIf https://github.com/TheoLeeCJ/openjev formerly OpenJev Qwen3.5-4B, TheoLeeCJ | 73.1 | 79.0 | 72.6 | 83.7 | 59.5 | ~$0.022 est. | 100.0% | 97.9% | 95.2% | 59.5% | 0.20 s raw→ 0.55 s adjustedp95 0.32 s raw → 0.78 s | our RunPod GPU | | 3 | by Maisa David Villalón djev https://github.com/Davipar/djev-dev <sup † jev12-note-djev </sup Maisa, diffusion-gemma | 73.0 | 82.7 | 65.4 | 91.4 | 57.6 | $0.026 announced | 100.0% | 97.9% | 93.2% | 69.5% | 0.24 s rawp95 0.31 s raw | production API | | 4 | by Eldan Ring Winnow-12B Q8 https://huggingface.co/EldanRing/Winnow-12B <sup † jev12-note-winnow-12b </sup | 71.2 | 82.0 | 72.0 | 82.3 | 52.9 | ~$0.037 est. | 100.0% | 96.9% | 91.1% | 70.9% | 0.23 s raw→ 0.60 s adjustedp95 0.41 s raw → 0.98 s | our RunPod GPU | | 5 | by kshetrajna12 reflex 4B https://github.com/kshetrajna12/reflex <sup † jev12-note-reflex-4b </sup | 70.3 | 80.1 | 75.2 | 68.0 | 59.7 | ~$0.022 est. | 100.0% | 94.8% | 97.3% | 63.2% | 1.80 s raw→ 3.75 s adjustedp95 2.05 s raw → 4.26 s | our RunPod GPU | | 6 | by hjmurmur Octalab jqv https://github.com/Octalab-Inc/jqv <sup † jev12-note-jqv </sup Qwen3-32B zero-shot | 68.6 | 79.3 | 79.0 | 74.6 | 47.5 | ~$0.056 est. | 100.0% | 95.8% | 92.5% | 64.5% | 0.75 s raw→ 1.64 s adjustedp95 0.97 s raw → 2.10 s | our RunPod GPU | | 7 | by milliseconds.ai Baptiste Laget decision-machine-1 https://www.milliseconds.ai <sup † jev12-note-decision-machine-1 </sup milliseconds.ai | 68.3 | 62.1 | 70.4 | 92.9 | 53.7 | $0.035 | 100.0% | 76.0% | 89.7% | 46.8% | 0.17 s rawp95 0.30 s raw | production API | | 8 | by Mapika decider-35b-a3b https://huggingface.co/Mapika/decider-35b-a3b <sup † jev12-note-decider-35b-a3b </sup | 67.6 | 79.6 | 71.5 | 80.8 | 45.3 | ~$0.067 est. | 100.0% | 96.9% | 91.1% | 65.5% | 0.29 s raw→ 0.73 s adjustedp95 0.49 s raw → 1.14 s | our RunPod GPU | | 9 | by IkerMoel open-alternative-jev https://github.com/ikermoel/open-alternative-jev <sup † jev12-note-open-alternative-jev </sup Qwen3.5-4B, IkerMoel | 67.0 | 64.0 | 63.2 | 83.5 | 59.6 | ~$0.022 est. | 100.0% | 84.4% | 74.7% | 56.8% | 0.21 s raw→ 0.56 s adjustedp95 0.32 s raw → 0.80 s | our RunPod GPU | | 10 | by mithalouni system-one-open https://github.com/mithalouni/system-one-open Gemma 4 E2B LoRA on an L4 | 66.6 | 69.5 | 56.7 | 77.0 | 64.8 | ~$0.015 est. | 100.0% | 93.8% | 87.7% | 49.1% | 0.65 s raw→ 1.30 s adjustedp95 0.77 s raw → 1.54 s | author's demo server | | 11 | by razorback16 / Codiv OpenJev https://github.com/razorback16/openjev DiffusionGemma 26B-A4B NVFP4, razorback16 | 66.4 | 79.2 | 64.8 | 83.2 | 45.5 | ~$0.066 est. | 100.0% | 95.8% | 91.1% | 65.5% | 0.24 s raw→ 0.63 s adjustedp95 0.31 s raw → 0.76 s | our RunPod GPU | | 12 | by Featherless AI SimpleJev Qwen3.8-27B https://github.com/featherless-ai/simple-jev <sup † jev12-note-simplejev-qwen3.8-27b </sup | 66.3 | 84.7 | 81.1 | 71.2 | 39.5 | ~$0.104 est. | 100.0% | 96.9% | 93.2% | 75.0% | 1.01 s raw→ 2.03 s adjustedp95 1.88 s raw → 3.76 s | author's demo server | | 13 | by ZeroEntropy ZeroEntropy zerank-2 https://huggingface.co/zeroentropy/zerank-2-reranker <sup † jev12-note-zerank-2 </sup | 66.0 | 63.0 | 76.5 | 79.0 | 49.8 | $0.047 | 100.0% | 79.2% | 88.4% | 47.3% | 0.13 s raw→ 0.40 s adjustedp95 1.50 s raw → 3.15 s | our RunPod GPU | | 14 | by OpenAI GPT-5.6 Luna https://platform.openai.com low reasoning effort | 65.9 | 95.3 | 89.8 | 77.5 | 28.5 | $0.242 | 100.0% | 97.9% | 96.6% | 94.5% | 0.97 s rawp95 1.82 s raw | production API | | 15 | by ekzhang openjev-sglang https://github.com/ekzhang/openjev-sglang Qwen3.6-35B-A3B on SGLang | 65.3 | 83.4 | 77.4 | 77.1 | 36.5 | ~$0.131 est. | 100.0% | 95.8% | 95.2% | 71.4% | 0.68 s raw→ 1.36 s adjustedp95 0.73 s raw → 1.45 s | author's demo server | | 16 | by Qwen Qwen3-Reranker-4B https://huggingface.co/Qwen/Qwen3-Reranker-4B <sup † jev12-note-qwen3-reranker-4b </sup | 63.8 | 64.0 | 67.0 | 78.7 | 49.2 | $0.050 | 100.0% | 79.2% | 87.7% | 50.0% | 0.13 s raw→ 0.41 s adjustedp95 1.56 s raw → 3.27 s | our RunPod GPU | | 17 | by kshetrajna12 reflex-27b https://github.com/kshetrajna12/reflex <sup † jev12-note-reflex-27b </sup Qwen3.8-27B | 63.3 | 85.8 | 86.2 | 67.5 | 32.3 | ~$0.181 est. | 100.0% | 95.8% | 95.9% | 75.9% | 1.89 s raw→ 3.93 s adjustedp95 2.21 s raw → 4.57 s | our RunPod GPU | | 18 | by Zhengxu Yu LitJev https://github.com/zhengxuyu/litjev <sup † jev12-note-litjev </sup Qwen3.8-27B | 62.7 | 82.4 | 83.5 | 66.7 | 33.6 | ~$0.163 est. | 100.0% | 97.9% | 88.4% | 73.2% | 2.03 s raw→ 4.20 s adjustedp95 2.46 s raw → 5.06 s | our RunPod GPU | | 19 | by Jared Palmer kev 0.6B https://github.com/jaredpalmer/kev <sup † jev12-note-kev-0.6b </sup research preview | 62.5 | 51.9 | 51.1 | 75.6 | 76.1 | ~$0.0063 est. | 100.0% | 81.3% | 66.4% | 40.0% | 0.59 s raw→ 1.33 s adjustedp95 0.97 s raw → 2.09 s | our RunPod GPU | | 20 | by Featherless AI SimpleJev Qwen3.6-35B-A3B https://github.com/featherless-ai/simple-jev <sup † jev12-note-simplejev-qwen3.6-35b-a3b </sup | 62.5 | 79.5 | 67.1 | 75.0 | 38.1 | ~$0.116 est. | 100.0% | 93.8% | 93.2% | 66.4% | 0.85 s raw→ 1.70 s adjustedp95 0.93 s raw → 1.86 s | author's demo server | | 21 | by David Villalon / Maisa djev https://github.com/Davipar/djev-dev <sup † jev12-note-djev-thinking </sup thinking | 62.4 | 80.8 | 92.7 | 75.2 | 26.9 | ~$0.274 est. | 95.8% | 99.0% | 80.1% | 77.7% | 0.43 s raw→ 1.00 s adjustedp95 1.45 s raw → 3.05 s | our RunPod GPU | | 22 | by us GitHub jev-local https://github.com/us/jev-local <sup † jev12-note-jev-local </sup Qwen3.5-9B | 61.8 | 70.8 | 68.7 | 69.2 | 43.3 | ~$0.077 est. | 100.0% | 84.4% | 89.0% | 59.1% | 1.05 s raw→ 2.24 s adjustedp95 2.62 s raw → 5.38 s | our RunPod GPU | | 23 | by Mapika decider-2b https://huggingface.co/Mapika/decider-2b <sup † jev12-note-decider-2b </sup | 61.7 | 61.2 | 46.6 | 83.2 | 61.0 | ~$0.020 est. | 100.0% | 85.4% | 77.4% | 47.3% | 0.26 s raw→ 0.67 s adjustedp95 0.28 s raw → 0.72 s | our RunPod GPU | | 24 | by Bespoke Labs Bespoke Nimble 9B https://github.com/bespokelabsai/nimble <sup † jev12-note-nimble-9b </sup | 60.5 | 77.9 | 65.3 | 78.7 | 33.4 | ~$0.166 est. | 100.0% | 94.8% | 89.0% | 65.5% | 0.39 s raw→ 0.93 s adjustedp95 0.65 s raw → 1.46 s | our RunPod GPU | | 25 | by Google Gemini 3.1 Flash-Lite https://ai.google.dev | 60.1 | 85.6 | 68.1 | 81.8 | 27.4 | $0.264 | 100.0% | 99.0% | 93.2% | 75.0% | 0.76 s rawp95 0.88 s raw | production API | | 26 | by razorback16 OpenJev https://github.com/razorback16/openjev <sup † jev12-note-openjev-thinking </sup thinking, BF16 | 60.0 | 88.0 | 69.6 | 76.1 | 27.8 | ~$0.255 est. | 100.0% | 100.0% | 94.5% | 78.2% | 0.46 s raw→ 1.08 s adjustedp95 1.08 s raw → 2.31 s | our RunPod GPU | | 27 | by Jared Palmer kev 4B https://github.com/jaredpalmer/kev <sup † jev12-note-kev-4b </sup research preview | 59.7 | 64.8 | 42.0 | 75.7 | 61.8 | ~$0.019 est. | 100.0% | 91.7% | 85.6% | 42.3% | 0.55 s raw→ 1.25 s adjustedp95 0.99 s raw → 2.13 s | our RunPod GPU | | 28 | by DeepSeek DeepSeek V4.1 Flash https://api-docs.deepseek.com thinking default | 57.5 | 94.3 | 96.7 | 71.6 | 16.8 | $0.594 | 98.6% | 99.0% | 93.2% | 95.0% | 1.42 s rawp95 4.89 s raw | production API | | 29 | by Jared Palmer kev 8B https://github.com/jaredpalmer/kev <sup † jev12-note-kev-8b </sup research preview | 56.4 | 69.4 | 44.2 | 74.9 | 44.0 | ~$0.073 est. | 100.0% | 92.7% | 90.4% | 47.3% | 0.59 s raw→ 1.33 s adjustedp95 1.15 s raw → 2.45 s | our RunPod GPU | | 30 | by Zefan Cai @Zefan Cai Open-Jev 9B https://github.com/Zefan-Cai/Open-Jev <sup † jev12-note-open-jev-zefan-9b </sup Zefan Cai | 55.0 | 71.2 | 63.3 | 72.0 | 28.1 | ~$0.249 est. | 100.0% | 90.6% | 81.5% | 60.9% | 0.75 s raw→ 1.66 s adjustedp95 1.81 s raw → 3.77 s | our RunPod GPU | | 31 | by Sean Goedecke system-one https://github.com/sgoedecke/system-one Qwen3-8B, Sean Goedecke | 54.8 | 70.3 | 36.8 | 84.4 | 41.5 | ~$0.089 est. | 100.0% | 90.6% | 91.8% | 50.0% | 0.17 s raw→ 0.48 s adjustedp95 0.30 s raw → 0.76 s | our RunPod GPU | | 32 | by Logan Markewich jeff https://github.com/logan-markewich/jeff <sup † jev12-note-jeff </sup Logan Markewich, GLiFormer 400M | 54.4 | 46.9 | 64.6 | 63.5 | 76.6 | ~$0.0060 est. | 100.0% | 76.0% | 61.6% | 37.7% | 0.94 s raw→ 2.03 s adjustedp95 10.97 s raw → 22.09 s | our CPU | | 33 | by Convai Innovations Laya https://huggingface.co/convaiinnovations/laya <sup † jev12-note-laya </sup Convai Innovations, ModernBERT-large 421M | 54.4 | 45.8 | 62.5 | 71.1 | 86.2 | ~$0.0029 est. | 94.4% | 72.9% | 69.2% | 34.1% | 0.79 s raw→ 1.72 s adjustedp95 2.20 s raw → 4.54 s | our CPU | | 34 | by Zefan Cai @Zefan Cai Open-Jev 2B https://github.com/Zefan-Cai/Open-Jev <sup † jev12-note-open-jev-zefan-2b </sup Zefan Cai | 51.3 | 61.0 | 55.1 | 73.5 | 28.1 | ~$0.249 est. | 100.0% | 79.2% | 88.4% | 42.7% | 0.66 s raw→ 1.48 s adjustedp95 1.45 s raw → 3.05 s | our RunPod GPU | | 35 | by Deepan Wadhwa OpenDecision https://github.com/deepanwadhwa/OpenDecision <sup † jev12-note-opendecision </sup ModernBERT-large zero-shot | 40.6 | 40.8 | 56.1 | 79.9 | 75.3 | ~$0.0066 est. | 87.5% | 62.5% | 71.2% | 33.2% | 0.34 s raw→ 0.83 s adjustedp95 0.54 s raw → 1.24 s | our RunPod GPU | | 36 | by Hemant heman10x openJev Verdict 1.4 https://huggingface.co/heman10x/rlcd-modernbert-151m <sup † jev12-note-openjev-verdict-1.4 </sup | 38.9 | 38.6 | 74.1 | 78.1 | 82.4 | ~$0.0039 est. | 86.1% | 67.7% | 56.2% | 37.7% | 0.31 s raw→ 0.78 s adjustedp95 0.92 s raw → 2.00 s | our CPU | | 37 | by Hemant heman10x openJev Verdict https://github.com/Heman10x-NGU/openJev-verdict-2.0 <sup † jev12-note-openjev-verdict </sup heman10x, ModernBERT-base 151M | 38.1 | 39.8 | 51.3 | 76.7 | 83.1 | ~$0.0037 est. | 86.1% | 65.6% | 61.0% | 38.2% | 0.28 s raw→ 0.71 s adjustedp95 1.45 s raw → 3.04 s | our CPU | | 38 | by Jared Palmer kev 0.5B https://github.com/jaredpalmer/kev <sup † jev12-note-kev-0.5b </sup | 33.2 | 38.2 | 47.4 | 77.0 | 76.1 | ~$0.0063 est. | 95.8% | 52.1% | 71.2% | 30.9% | 0.43 s raw→ 1.01 s adjustedp95 0.92 s raw → 1.99 s | our RunPod GPU | | 39 | by Fastino GLiNER2 large https://huggingface.co/fastino/gliner2-large-v1 <sup † jev12-note-gliner2-large </sup | 29.6 | 40.1 | 24.3 | 61.7 | 73.3 | ~$0.0077 est. | 98.6% | 62.5% | 61.0% | 36.4% | 1.10 s raw→ 2.34 s adjustedp95 14.49 s raw → 29.13 s | our CPU | | 40 | by Aditya isHeSatoshi smalljev semantic-v9 https://github.com/isHeSatoshi/smalljev <sup † jev12-note-smalljev </sup | 27.4 | 35.1 | 58.9 | 79.8 | 57.9 | ~$0.025 est. | 97.2% | 68.8% | 40.4% | 38.2% | 0.41 s raw→ 0.98 s adjustedp95 0.46 s raw → 1.07 s | our RunPod GPU | | 41 | by Fastino GLiNER2 https://github.com/fastino-ai/GLiNER2 <sup † jev12-note-gliner2 </sup Fastino, gliner2.5-base | 24.0 | 35.6 | 23.7 | 71.8 | 83.1 | ~$0.0037 est. | 97.2% | 66.7% | 45.9% | 36.4% | 0.31 s raw→ 0.78 s adjustedp95 4.15 s raw → 8.46 s | our CPU | | 42 | by Kotoba Labs open-jev-deberta-v3-large https://github.com/kotoba-lang/typed-decisions local CPU | 23.1 | 31.9 | 66.4 | 66.0 | 74.0 | ~$0.0073 est. | 100.0% | 49.0% | 53.4% | 36.4% | 1.77 s raw→ 3.69 s adjustedp95 3.35 s raw → 6.85 s | our CPU | | 43 | by Fastino GLiNER2.5 multi https://huggingface.co/fastino/gliner2.5-multi-v1 <sup † jev12-note-gliner2.5-multi </sup Fastino, 287M | 16.6 | 27.7 | 56.1 | 67.8 | 82.4 | ~$0.0039 est. | 90.3% | 51.0% | 43.8% | 37.7% | 0.43 s raw→ 1.01 s adjustedp95 8.18 s raw → 16.50 s | our CPU | | 44 | by Fastino GLiNER2.5 small https://huggingface.co/fastino/gliner2.5-small-v1 <sup † jev12-note-gliner2.5-small </sup Fastino, 74M | 13.8 | 25.6 | 47.2 | 77.8 | 82.4 | ~$0.0039 est. | 83.3% | 47.9% | 50.0% | 33.2% | 0.11 s raw→ 0.38 s adjustedp95 2.10 s raw → 4.35 s | our CPU | | 45 | by Mixedbread Mixedbread mxbai-rerank-base-v2 https://huggingface.co/mixedbread-ai/mxbai-rerank-base-v2 <sup † jev12-note-mxbai-rerank-base-v2 </sup | 0.8 | 6.7 | 83.1 | 87.5 | 67.9 | $0.012 | 44.4% | 33.3% | 26.7% | 40.0% | 0.07 s raw→ 0.29 s adjustedp95 0.23 s raw → 0.62 s | our RunPod GPU | | 46 | by BAAI BAAI bge-reranker-v2-m3 https://huggingface.co/BAAI/bge-reranker-v2-m3 <sup † jev12-note-bge-reranker-v2-m3 </sup | 0.7 | 6.3 | 83.8 | 89.5 | 73.4 | $0.0077 | 43.1% | 36.5% | 8.9% | 36.8% | 0.03 s raw→ 0.22 s adjustedp95 0.18 s raw → 0.51 s | our RunPod GPU | | 47 | by Alibaba-NLP Alibaba GTE Reranker ModernBERT-base https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base <sup † jev12-note-gte-reranker-modernbert-base </sup | 0.3 | 4.6 | 76.8 | 90.6 | 69.6 | $0.010 | 33.3% | 39.6% | 30.1% | 33.6% | 0.05 s raw→ 0.25 s adjustedp95 0.10 s raw → 0.35 s | our RunPod GPU | | 48 | by AltSlate Labs Certo v1 https://huggingface.co/altslate/certo-decision-model <sup † jev12-note-certo </sup | 0.0 | 0.0 | 82.0 | 94.0 | 100.0 | ~$0.0010 est. | 27.8% | 30.2% | 21.9% | 31.8% | 0.02 s raw→ 0.19 s adjustedp95 0.03 s raw → 0.21 s | our RunPod GPU | | Honorable mentions — services built on another entrant's model jev12-honorable — shown, not ranked: A service that runs another entrant's model is listed with all of its scores and axes, but is not ranked against the models. | | | by mrmps @michael chomsky classifier.dev https://classifier.dev <sup † jev12-note-classifier-dev-fast </sup fast tierhonorable mention · not ranked | 83.6 | 85.1 | 77.9 | 87.6 | 84.3 | ~$0.0033 est. | 100.0% | 99.0% | 97.3% | 70.5% | 0.39 s rawp95 0.45 s raw | production API | | Partial runs — shown, not ranked: a tier attempted for fewer than 95 % of its decisions. | | | by Qwen / Chutes Qwen3.8 27B https://chutes.ai Chutes TEEpartial run · not ranked | 24.8 | 67.4 | 92.1 | 61.3 | 0.0 | ~$2.669 est. | 98.6% | 99.0% | 95.3% | 21.4% | 5.75 s rawp95 12.97 s raw | production API | | | by Cactus Compute Needle 3, options as tools https://github.com/cactus-compute/needle post-hoc adapter modepartial run · not ranked | 1.1 | 13.5 | none label only | 52.8 | 65.3 | ~$0.014 est. | 66.7% | 31.3% | 34.2% | — | 3.78 s raw→ 7.71 s adjustedp95 33.64 s raw → 67.42 s | our CPU | | | by Cactus Compute Needle 3 https://github.com/cactus-compute/needle Cactus, 2-bit, local CPUpartial run · not ranked | 0.1 | 4.6 | none label only | 59.9 | 58.7 | ~$0.024 est. | 47.2% | 16.7% | 31.5% | 7.7% | 1.69 s raw→ 3.52 s adjustedp95 14.36 s raw → 28.88 s | our CPU |