{"slug": "jev-deserves-hype-but-not-the-type-its-getting", "title": "Jev deserves hype but not the type its getting", "summary": "TypeSafe AI's rubric-conditioned classification model Jev scored 98.84% overall accuracy on the CT1-8 core classification tasks, beating Claude Haiku 4.5 (96.46%), Claude Sonnet 5 (96.18%) and the open-weights OpenJev (96.28%) across a 124-test, $110.11 benchmark, according to the published jev-benchmark repo. Jev cost $0.031 per 1,000 calls versus $1.590 for Haiku 4.5 and $3.599 for Sonnet 5, making it roughly 50-116x cheaper, and its confidence scores flagged real errors on 3/3 tasks while both Claude models' calibration was useless on 3/3. The benchmark also found Anthropic's stronger Sonnet 5 model was the best of the four at single-document, single-question tasks but the worst at batched versions such as 30-question whole-exam grading and full 10-layer tree traversal in one call.", "body_md": "**Does [Jev](https://typesafe.ai) (TypeSafe AI's rubric-conditioned classification model) genuinely\nread and apply a multi-clause rubric — and how does it stack up against two general-purpose LLMs\n(Claude Haiku 4.5, Claude Sonnet 5) and a free, self-hostable alternative (OpenJev), on both\nquality *and* price?**\n\nThis repo is the full benchmark: every corpus, every prediction, every dollar spent, every test\nthat failed to find a difference as well as every one that did. Ten structurally distinct test\nsuites, four models, **124 passing tests**, **$110.11 total spend**, all of it reproducible from\nthe seeds and scripts in this repo.\n\n|  | Jev | Claude Haiku 4.5 | Claude Sonnet 5 | OpenJev | \n|---|---|---|---|---|\n| **CT1-8 overall accuracy** | **98.84%** | 96.46% | 96.18% | 96.28% | \n| **Price per 1,000 calls (CT1-8)** | $0.031 | $1.590 | $3.599 | **~$0.014 (self-hosted est.)** | \n| **Total spend, whole benchmark** | $0.75 | $29.67 | $68.10 | **$0.00 real / ~$0.43 self-hosted est.** | \n| Calibration (confidence flags real errors) | **Consistently useful, 3/3 tasks** | Consistently useless, 3/3 tasks | Consistently useless, 3/3 tasks | In between | \n\nOpenJev's real spend in this benchmark genuinely was $0.00 (Codiv's free hosted tier) — the\n\"self-hosted est.\" figures are a documented estimate of what running it yourself on rented GPU\ncompute would cost instead, so it isn't compared against metered API arms as if it were free to\ndeploy. See [Price, in full](#price-in-full) and [methodology.md §17](https://github.com/YidiDev/jev-benchmark/blob/main/methodology.md#17-estimating-self-hosted-compute-cost-for-the-free-arms)\nfor the exact assumptions.\n\n**Jev is both more accurate and ~50-116x cheaper than the two Claude models on the core\nclassification task** — the single clearest number in this repo. It also wins on rubric-following\nunder adversarial relabeling, on the specific arithmetic weakness its own vendor documentation\npredicted, and on calibration (its confidence scores actually flag its mistakes; neither Claude\nmodel's do). Its advantages are *not* universal, though — they concentrate specifically where task\nstructure favors Jev's architecture (small-step execution, native multi-question batching) and\ndisappear where it doesn't.\n\n**The most surprising result belongs to Sonnet, not Jev.** Anthropic's *stronger* model was added\nspecifically to check whether Haiku's underperformance was a Haiku-specific weakness or a\ngeneral-purpose-LLM one. It's neither, cleanly — Sonnet is the best of all four models at the\nnarrowest, most focused version of every task (single document, single question, one decision\nstep at a time) and the *worst* of all four at the broadest, most batched version of the same\ntasks (30-question whole-exam grading, full 10-layer tree traversal in one call). More capability\ndid not transfer into more reliable batching; if anything, the opposite. Read on for the receipts.\n\nMost \"zero-shot classification\" benchmarks test whether a model can match a label's *text* to a\ndocument's *content* (NLI entailment, embedding similarity). That's a different, easier task than\nwhat a real rubric-based sorting system needs: **conditioning on an arbitrary, multi-clause set of\nrules** — thresholds, exceptions, lookups, negations — regardless of what the destination folders\nhappen to be named.\n\nThis benchmark was built to answer five concrete questions, in order:\n\n1. **Does Jev genuinely condition on the rubric** , or is it secretly doing label-text matching\nlike a classifier would? (Part 1 — tested via an adversarial shuffle control.)\n2. **Is Jev good enough to replace a cheap general-purpose LLM** for this task, on quality*and* price? (Part 2 — Claude Haiku 4.5 as the reference ceiling.)\n3. **Is there a viable, free, self-hostable fallback** if Jev's hosted API ever became\nunavailable? (Part 3 — OpenJev, an open-weights model that speaks Jev's exact wire protocol.)\n4. **Where does Jev actually break?** A benchmark where the subject scores 100% tells you\nnothing about its limits — so four \"hard mode\" clause types (Part 2) and a completely\ndifferent task, chained multi-step decision execution (Part 4, CT9), were built specifically\nto find failure modes.\n5. **Does any of this generalize to a genuinely different task shape** — partial-credit rubric\ngrading of open-ended text, not classification? (Part 5, CT10.)\n\nEvery design decision below — the seeded-RNG discipline, the shuffle control, the price ledger,\nthe decision to build *harder* tests once the easy ones stopped being informative — is documented\nwith its rationale in [`methodology.md`](https://github.com/YidiDev/jev-benchmark/blob/main/methodology.md), including the places where an initial\napproach turned out to be wrong and was replaced.\n\n| Model | What it is | Pricing (list) | \n|---|---|---|\n| **Jev** (`jev-1.13` ) | TypeSafe AI's purpose-built rubric-classification model. Three primitives: `Noul` (yes/no),`Choice` (pick 1 of ≤255 options),`Score` (rate against an ordered rubric). | $0.042 / Mtok input, **output free** | \n| **Claude Haiku 4.5** | Anthropic's fast general-purpose LLM — the reference-ceiling comparison arm. | $1 / Mtok input, $5 / Mtok output | \n| **Claude Sonnet 5** | Anthropic's stronger general-purpose LLM, added later to check whether Haiku's underperformance was Haiku-specific or general — same rubric, same forced tool-use, same grid as Haiku throughout. | $2 / Mtok input, $10 / Mtok output | \n| **OpenJev** | `razorback16/openjev` , an open-weights model (DiffusionGemma 26B-A4B, Apache-2.0) that speaks Jev's exact wire API. Tested via the free-hosted[Codiv](https://codiv.ai) endpoint — real cost here is $0.00, but self-hosting it (its actual real-world deployment path) is estimated at ~$0.028/Mtok input on a 24GB-class GPU. | **$0.00** (hosted tier used) / ~$0.028/Mtok (self-hosted est.) | \n| NLI (bart-large-mnli) / Embeddings (bge-m3) | Standard zero-shot classification baselines — the \"does it actually read the rubric\" control group for Part 1. Not real contenders (they can't follow a rubric at all), included to prove the point. Ran locally in this benchmark (real cost $0.00); self-hosting either commercially is estimated at ~$0.03/Mtok on a small cloud GPU. | Local, **$0.00** real / ~$0.03/Mtok (self-hosted est.) | \n\n| Suite | Tests | Ground truth | \n|---|---|---|\n| CT1 descriptive | Simple 3-way document type sort | Structured metadata | \n| CT2 conjunctive+threshold | Numeric threshold rule | Structured metadata | \n| CT3 relational | Lookup + override exception | Structured metadata | \n| CT4 negative/exclusionary | Negation-based rule | Structured metadata | \n| CT5 computed_threshold | **Arithmetic** : sum stated line items, no total given | Structured metadata | \n| CT6 temporal_reasoning | Date comparison, zero narrative cues | Structured metadata | \n| CT7 multi_hop_relational | Two chained lookups (team → division → program) | Structured metadata | \n| CT8 long_context_distractor | Same logic as CT1, buried in ~600-word padding | Structured metadata | \n| CT9 chained decision execution | 10-layer decision tree, walked in 1/2/5/10-step chunks, 4 outcome folders | Code-computed tree walk | \n| CT10 exam grading | Partial-credit rubric grading of 30 AP World History paragraph answers, 100 students | Seeded student-ability model | \n\nCT1-4 is the original design. CT5-8 (\"hard mode\") and CT9/CT10 were built afterward, specifically because CT1-4 turned out to be a ceiling task (Jev: 100%) — a benchmark that never finds a failure can't characterize one.\n\nThe critical test: **Condition C** relabels folders with *misleading* but semantically real names\n(a text-matcher's worst case), and the **SHUFFLE control** adversarially permutes the\nrubric-clause-to-folder-ID mapping under opaque IDs. If Jev were secretly doing label-text\nmatching, its accuracy under these conditions would collapse toward the NLI/embedding baselines.\n**It doesn't — Jev stays at 100.00% in every single condition**, while the baselines collapse to\n26-48%. This is the foundational result everything else builds on.\n\n**Price**: this part's baselines ran locally in this benchmark, real cost $0.00 — CT1-4 corpus\ngeneration cost **$1.06** total (one-time Sonnet 5 cost to build the 240-document corpus), Jev's\nown inference cost **$0.18** for all 2,880 classifications, and NLI/embeddings' estimated\nself-hosted cost (if you deployed them instead of running locally like this benchmark did) is\n**~$0.03 and ~$0.01** respectively for the same 1,440 real predictions each — see\n[methodology.md §17](https://github.com/YidiDev/jev-benchmark/blob/main/methodology.md#17-estimating-self-hosted-compute-cost-for-the-free-arms).\n\n|  | Jev | Haiku | Sonnet | \n|---|---|---|---|\n| CT1-4 (original corpus) | **100.00%** | 97.74% | 99.86% | \n| CT5-8 (hard mode, built to find Jev's limits) | **97.67%** | 95.17% | 92.50% | \n| **Price, CT1-4 + CT5-8 combined (5,760 calls)** | **$0.18** | $9.18 | $20.73 | \n\n**Jev wins on both accuracy and price against both Claude models**, by a wide margin on price\n(**~50-116x cheaper**). Sonnet was added later specifically because Haiku underperformed Jev by a\nwider margin on CT9/CT10 than on CT1-8 — the question was whether a stronger model would close\nthe gap. On CT1-4 it nearly does (99.86%, within noise of Jev's 100%). **On CT5-8 it doesn't — it's\nthe worst of the three**, and for a different reason than Haiku's. Digging into *why* CT5-8 exists:\n\nJev's one real weakness anywhere in CT1-8 — arithmetic near a stated threshold — was predicted in\nadvance by the vendor's own model documentation. It's real (9.3% error rate), and **both Claude\nmodels are worse at it** (Haiku 18.2%, Sonnet 13.7%). But CT5 arithmetic isn't what drags Sonnet's\nCT5-8 average down the most — **Sonnet has an isolated, severe failure on CT7's two-hop lookup\nunder Condition C** (misleading-but-plausible folder names) that neither Jev nor Haiku share at\nall: accuracy collapses from 100% (every other condition) to **34.4%**, at *high* confidence\n(0.95 mean, indistinguishable from its confidence when correct) — see [Part 3](#part-3--openjev-fallback-viability)'s\nchart, which now shows this side-by-side with OpenJev's own, independently-diagnosed collapse on\nthe exact same clause type and condition. Two of the three a priori predicted weaknesses\n(multi-hop lookup, long-context distraction) materialize for *exactly one* model each (Sonnet on\nCT7, nobody on CT8) — reported as found, not smoothed into an average.\n\n**Result, per the pre-registered comparison rule in [`test-plan.md`](https://github.com/YidiDev/jev-benchmark/blob/main/test-plan.md): Jev matches\nor exceeds both Claude models on accuracy** in every scope tested, at 50-116x lower price — a wide\nmargin on both axes of the comparison. Being the *stronger* general-purpose model did not help\nSonnet here; it has a sharper, more isolated failure mode than Haiku, not a smaller one.\n\nOpenJev cost $0.00 in this benchmark (free Codiv hosted tier) and scores 96.28% overall on CT1-8 —\na genuinely viable, low-cost fallback for simple classification (self-hosting it for real is\nestimated at ~$0.014 per 1,000 calls on CT1-8, still 2-100x cheaper than every other arm — see\n[Price, in full](#price-in-full)). It has exactly one sharp, fully-diagnosed weakness: CT7's multi-hop lookup\ncollapses to 40% under Condition C specifically (red bar above) — but the *identical* tree scores\n100% under the SHUFFLE control (opaque random IDs). That rules out \"OpenJev can't follow the\nrubric\" — it's a label-*collision* bug (Condition C's misleading names are other real, plausible\nlabels; SHUFFLE's aren't), isolated to one clause type under one condition. **Sonnet (violet bar)\nindependently collapses on the exact same clause type and condition**, to a similar 34.4% — two\nstructurally different models, hitting the same specific trap, for what looks like the same\nunderlying reason (both fail confidently, and both recover completely under SHUFFLE).\n\n**Price**: $0.00 real spend, always — the free Codiv tier never touched the $110 Anthropic budget\nfor any of CT1-8, CT9, or CT10. Self-hosting it for real is estimated at **$0.43 total** across\nall three task families (methodology.md §17) — still far below every metered API arm, but not\nactually free the way \"$0.00\" implies.\n\n**Verdict**: viable low-cost fallback for CT1-8-style classification; not yet viable for CT9-style\nchained execution (see below) — meaningfully behind all three other models at every step size.\n\nA completely different task from CT1-8: walking a 10-layer decision tree in chunks of size\nk ∈ {1, 2, 5, 10}, with **real compounding** between chunks (no ground-truth rescue). The\nheadline finding inverts the a priori hypothesis: going in, more handoffs (lower k) was expected\nto *hurt* accuracy via compounding error. **It's the opposite for every model** — frequent small\nhandoffs beat one unassisted full-chain call, decisively.\n\n**Jev's advantage over Haiku is real but narrow**: +17-22pp at k=1/2, but statistically tied at\nk=5/10 — Jev's edge is about small-step execution discipline, not raw multi-hop reasoning\ncapacity. **Sonnet crosses both other lines, in opposite directions, as k grows.** At k=1 it's the\nbest of all three (86.7%, +11.7pp over Jev, +28.9pp over Haiku) — the strongest model wins\ndecisively at the narrowest possible decision. By k=10 it's the *worst* of all three (22.8% vs.\nJev's 29.4% and Haiku's 31.7%) — more capability did not help it hold up under one large,\nunassisted, multi-step call; if anything it degraded faster than either weaker model. **The one\nplace in this whole benchmark where Jev is *less* stable than Haiku**: at k=10, Jev disagrees with\nitself across repeats 8.3-10% of the time vs. Haiku's 0% — but Sonnet is less stable still at k=10\nunder opaque labeling (20%), the highest disagreement rate anywhere in this project. Reported as\nfound, not smoothed over.\n\n**Price**: 900 traces × 4 arms = 3,420 API calls per model. Jev **$0.32**, Haiku **$10.27**,\nSonnet **$24.55**, OpenJev $0.00 real / ~$0.20 self-hosted est. — both Jev and a self-hosted\nOpenJev undercut both Claude models by more than two orders of magnitude on this much more\ndemanding multi-step task (see [Price, in full](#price-in-full)).\n\nA third structurally distinct task: partial-credit rubric grading of open-ended paragraph\nanswers (30 questions, 100 simulated students, non-uniform rubric summing to 100 points), with an\northogonal axis — does the grading model need the answer key, or does it already know the\nmaterial? Two grading modes: **chained** (1 call/question) and **whole-exam** (1 call, all 30\nquestions at once).\n\n**A genuine architecture-driven finding**: Jev is essentially flat between chained and whole-exam\ngrading (its `system_one` call natively evaluates multiple `Score` questions in parallel, so its\n\"whole exam\" call is structurally ~30 independent judgments). **Haiku, Sonnet, and OpenJev all\nlose ground in whole-exam mode** — their single-shared-JSON-object approach to \"many answers in\none call\" is a genuinely harder task shape. This is avoidable by architecture, not an inherent\nmodel-quality gap. **Sonnet loses by far the most: 90% → 54% exact-match, a 36-point drop** —\nmore than double Haiku's 15pp drop and OpenJev's 19pp drop.\n\n**This is the sharpest, most decision-relevant Sonnet result in the whole benchmark.** In\n*chained-with-key* mode, Sonnet is the best grader of all four models by a wide margin — MAE\n0.039 (96.1% exact-match) vs. Jev's 0.129 and Haiku's 0.139, and total-exam-score error of just\n0.93 points vs. Jev's 3.82 and Haiku's 3.86. Ask it one focused question at a time, with the\nanswer key, and it's clearly the strongest grader here. But in *whole-exam* mode — the exact same\nunderlying knowledge, batched into one 30-question call — it becomes the **worst** grader of all\nfour: MAE 0.44-0.57 (vs. Haiku's 0.30-0.33 and Jev's 0.12-0.17), total-exam-score error up to\n16.44 points (vs. Haiku's 8.84, Jev's 4.63). Same model, same material, same rubric — the only\nvariable that moved is how many judgments it had to hold in one call, and on that axis alone it\nwent from best to worst. Confidence doesn't flag this either: Sonnet's confidence-at-errors\n(0.839) is barely below its confidence-at-correct (0.909), the same \"fails confidently\" pattern\nas Haiku, just less extreme.\n\nOn the knowledge question: **Haiku needs the answer key the least** in chained mode (its own\nhistorical knowledge does almost as much work as being handed the key, +2.5-3pp gap vs. Jev's\n+5-7pp) — Sonnet needs it *the most* (+13pp chained exact-match gap between with/without key,\nthe largest of the three LLM arms) — a genuinely different profile from the accuracy/price story\nabove.\n\n**Price**: 6,200 grading actions × 4 arms = 24,800 calls. Jev **$0.25**, Haiku **$10.22**,\nSonnet **$22.82**, OpenJev $0.00 real / ~$0.15 self-hosted est.\n\nThe single most decision-relevant number in this whole repo, replicated on **three structurally\nunrelated tasks**: does a model's confidence actually predict whether it's wrong? **Jev's does,\nconsistently.** Both Claude models' confidence is nearly flat between correct and incorrect\nanswers — they fail *confidently*, giving a downstream review queue nothing to act on. Sonnet's\ngap (0.01-0.07 across the three tasks) is marginally larger than Haiku's (0.00-0.05) but nowhere\nclose to Jev's (0.16-0.49) — being a stronger model didn't make Sonnet's confidence more useful,\neither. A rule like \"route anything under 0.6 confidence to human review\" would catch the large\nmajority of Jev's mistakes; the equivalent rule for either Claude model would catch almost none of\nthem.\n\nPrice is a first-class result in this benchmark, not an afterthought — every table above includes it, and here's the full picture:\n\n| Model | CT1-8 (5,760 calls) | CT9 (3,420 calls) | CT10 (~6,200 calls) | **Total, all arms** | \n|---|---|---|---|---|\n| Jev | $0.18 | $0.32 | $0.25 | **$0.75** | \n| Claude Haiku 4.5 | $9.18 | $10.27 | $10.22 | **$29.67** | \n| Claude Sonnet 5 | $20.73 | $24.55 | $22.82 | **$68.10** | \n| OpenJev, real spend | $0.00 | $0.00 | $0.00 | **$0.00** | \n| OpenJev, self-hosted estimate | ~$0.08 | ~$0.20 | ~$0.15 | **~$0.43** | \n| NLI (bart-large-mnli), self-hosted estimate | ~$0.03 | — | — | **~$0.03** | \n| Embeddings (bge-m3), self-hosted estimate | ~$0.01 | — | — | **~$0.01** | \n\n**Two different numbers, deliberately kept apart.** OpenJev's row above genuinely cost $0.00 in\nthis benchmark (Codiv's free hosted tier), and NLI/embeddings genuinely cost $0.00 too (ran\nlocally on this project's own hardware) — that's the real, metered spend, unchanged in\n`results/spend_ledger.jsonl`. The \"self-hosted estimate\" rows are a separate, documented\nprojection of what actually deploying each of these three yourself would cost on rented GPU\ncompute (methodology.md §17) — not real spend, and never counted against the Anthropic budget\nbelow. Even under that estimate, Jev and a self-hosted OpenJev both still cost under 1% of the\ntwo Claude models' combined spend across the entire benchmark — Jev is not the *cheapest* option\nby this estimate on any single task family (OpenJev's self-hosted estimate undercuts it\neverywhere, a genuinely counterintuitive result explained in methodology.md §17), but both remain\nin a completely different cost class from either Claude model.\n\nCorpus generation (one-time, via Claude Sonnet 5 in its upstream content-authoring role — not the\nsame as its downstream grading-arm role above — to build the 480 CT1-8 documents + 60 CT9 forms +\n100 CT10 exams) cost an additional $11.59 — not a recurring cost, since the corpus itself is\ncommitted to this repo and never needs regenerating. **Grand total (real, metered spend):\n$110.11**, all logged to [`results/spend_ledger.jsonl`](https://github.com/YidiDev/jev-benchmark/blob/main/results/spend_ledger.jsonl) call-by-call\nas it was spent, not estimated after the fact — cumulative Anthropic spend (corpus generation +\nHaiku + Sonnet) landed at $109.36 of the $110.00 approved budget, $0.64 under the hard cap.\n\nFull detail in [`methodology.md`](https://github.com/YidiDev/jev-benchmark/blob/main/methodology.md) (17 sections, one per phase); the highlights\nthat matter most for trusting these results:\n\n- **Seeded RNG discipline.** Every choice that should be uninfluenced by semantics — opaque folder\nIDs, the shuffle-control permutation, which criteria a partial-credit answer satisfies, student\nability — is drawn from`harness.constants.sub_rng(purpose)` , an independently-reproducible\nstream keyed off one`MASTER_SEED` . Nothing here was hand-picked to look good.\n- **The shuffle control** (Part 1) is the load-bearing methodological device of this whole\nproject: it's the one test that can actually distinguish \"the model reads the rubric\" from \"the\nmodel matches folder-name vibes,\" and it was applied to every classification-shaped task.\n- **Price tracked as it was spent** , not estimated afterward — every API call, successful or not,\nis logged to`results/spend_ledger.jsonl` with a hard budget ceiling enforced*before* the call\nthat would exceed it, not after. Real spend for nli-bart/emb-bge/OpenJev genuinely is $0.00 in\nthis ledger — the separate self-hosted cost estimate (methodology.md §17) is exactly that,\nclearly labeled and never mixed into the real ledger, precisely because this principle matters.\n- **Negative results are reported, not hidden.** CT7 and CT8 (hard mode) found no weakness for\nJev or Haiku (Sonnet is the exception on CT7 — also reported, not smoothed over). CT9 found\nJev*less* stable than Haiku at k=10, and Sonnet less stable still. All of it is in the numbers\nabove, not filtered out.\n- **Decomposition frozen up front** (test-plan.md §8): rubric-clause granularity is a large free\nparameter that can make a task look artificially easy or hard; the exact decomposition is fixed\nbefore any arm sees the test set and applied identically across every arm.\n- **A comparison arm added after the fact, reasoned about openly.** Sonnet was proposed, declined,\nand later revisited once CT9/CT10 existed and showed Haiku underperforming Jev by a wider margin\nthan CT1-8 ever did — the reversal and its cost accounting are logged in methodology.md, not\npresented as though Sonnet had been in scope from the start.\n\n```\nuv sync\ncp .env.example .env   # fill in TYPESAFE_API_KEY, ANTHROPIC_API_KEY, CODIV_API_KEY\n\n# Everything below is already committed (corpus, predictions, spend ledger) --\n# these commands regenerate results from what's already here, or extend it.\npytest tests/ -q                        # 124 tests, exercises every scoring function\npython -m harness.spend_ledger          # print the full price ledger (real, metered spend)\npython -m scripts.estimate_self_hosted_cost  # rebuild results/self_hosted_cost_estimate.json\npython -m scripts.generate_summary      # rebuild results/summary.{json,csv}\npython -m scripts.generate_charts       # rebuild every chart in charts/\n\n# Re-running an arm against already-generated corpora (resumable, will skip\n# anything already in results/predictions/):\npython -m scripts.run_arm --arm nli-bart\npython -m scripts.run_api_arm --arm jev        # or haiku / sonnet / openjev\npython -m qtree.runner --arm jev               # CT9\npython -m examgrade.runner --arm jev           # CT10\n```\n\nRegenerating the corpus from scratch (not needed — it's committed — but fully reproducible):\n`corpus/generate_metadata.py` → `corpus/generate_prose.py` (and the `qtree`/` examgrade`\nequivalents), all seeded from `MASTER_SEED` in `harness/constants.py`.\n\n```\ncorpus/       CT1-8 document generator + frozen manifest + ground-truth engine\nrubrics/      Rubric clause text, folder-name conditions, shuffle-control permutation\narms/         One module per CT1-8 arm: jev, haiku, sonnet, openjev, nli-bart, emb-bge\nqtree/        CT9: decision tree, chunked execution arms, scoring\nexamgrade/    CT10: exam questions/rubrics, student corpus, grading arms, scoring\nharness/      Shared scoring (bootstrap CI, ECE, disagreement), spend ledger, constants\nscripts/      run_arm / run_api_arm (CT1-8), generate_summary, generate_charts,\n              estimate_self_hosted_cost\nresults/      Every raw prediction, the spend ledger, the self-hosted cost\n              estimate, and the consolidated summary\ncharts/       Every chart in this README, regenerable via scripts/generate_charts.py\ntests/        124 tests covering ground truth, RNG determinism, and every scoring function\n```\n\n- [`test-plan.md`](https://github.com/YidiDev/jev-benchmark/blob/main/test-plan.md) — the original design document (written before any code existed)\n- [`methodology.md`](https://github.com/YidiDev/jev-benchmark/blob/main/methodology.md) — every decision made, in order, including what didn't work\n- [`results.md`](https://github.com/YidiDev/jev-benchmark/blob/main/results.md) — the full numeric results this README summarizes\n- [`results/summary.json`](https://github.com/YidiDev/jev-benchmark/blob/main/results/summary.json) /[` results/summary.csv`](https://github.com/YidiDev/jev-benchmark/blob/main/results/summary.csv) — every computed metric, machine-readable\n\n[MIT](https://github.com/YidiDev/jev-benchmark/blob/main/LICENSE).", "url": "https://wpnews.pro/news/jev-deserves-hype-but-not-the-type-its-getting", "canonical_source": "https://github.com/YidiDev/jev-benchmark", "published_at": "2026-09-23 23:52:11+00:00", "updated_at": "2026-09-24 00:01:12.216448+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-products"], "entities": ["Jev", "TypeSafe AI", "Claude Haiku 4.5", "Claude Sonnet 5", "OpenJev", "Anthropic", "Codiv"], "alternates": {"html": "https://wpnews.pro/news/jev-deserves-hype-but-not-the-type-its-getting", "markdown": "https://wpnews.pro/news/jev-deserves-hype-but-not-the-type-its-getting.md", "text": "https://wpnews.pro/news/jev-deserves-hype-but-not-the-type-its-getting.txt", "jsonld": "https://wpnews.pro/news/jev-deserves-hype-but-not-the-type-its-getting.jsonld"}}