{"slug": "choice-score-and-noul-four-mistakes-with-jev-s-primitives", "title": "Choice, Score and Noul: four mistakes with Jev's primitives", "summary": "A developer spent a week building a Magic 8 Ball on TypeSafe's Jev, a System One model that returns probability distributions over supplied options instead of prose, and documented four mistakes made while using its Choice, Score, and Noul primitives. The writeup shows that rounding a Score's probability-weighted average can land on a bucket holding only 16% of the mass while 56% sits on the adjacent level, and that the built-in confidence field measures distribution peakedness rather than genuine uncertainty, reporting 0.98 for questions like \"Should I quit my job?", "body_md": "Jev is a System One model from TypeSafe. It reads natural language like any LLM,\n\nand instead of writing a reply it returns a probability distribution over\n\noptions you supply. No prose, no reasoning trace, no JSON to repair — a number\n\nper option, summing to 1.\n\nI spent a week building a Magic 8 Ball on it, which\n\nsounds like a toy and turned out to be a decent test rig: every answer is a\n\njudgment with no ground truth, which is where calibrated probabilities either\n\nearn their keep or embarrass you. It's live at\n\n[/jevball](https://dave8172-website.vercel.app/jevball).\n\nI got the primitives wrong four times. Each mistake produced working code that\n\nlooked right, so they're worth writing down.\n\n**Choice** takes a map of named options and returns a probability for each, plus\n\nthe highest-scoring one. Routing a ticket to a department, classifying a\n\ndocument, picking a handler.\n\n**Score** takes an ordered array of level descriptions — two to ten — and returns\n\na position along them that can land between levels. Severity, frustration, skill\n\nlevel. Anything on a spectrum you can describe in words.\n\n**Noul** takes a yes/no statement and returns one number: the probability it's\n\ntrue. No confidence field, because the probability *is* the answer.\n\nAll three take the same two inputs: `state`, the data being judged, and\n\n`instructions`, the question asked about it. That's the whole surface.\n\nA Score returns `score`, a position on your levels. My ball rounded it to pick a\n\nbucket. Someone asked it *\"is black a color?\"* and got this:\n\n```\nbars   : 0=2%  1=6%  2=19%  3=16%  4=56%\nscore  : 3.18\n```\n\n`Math.round(3.18)` is 3 — a bucket holding **16%** — while 56% of the mass sat\n\non level 4 beside it.\n\nThe score is a **probability-weighted average**. That 19% on level 2 dragged the\n\naverage down across a bucket boundary. An average of a skewed distribution\n\npoints at a place the distribution isn't.\n\nTypeSafe defines a Choice's answer as the option with the highest probability.\n\nScore has no equivalent field, and I quietly substituted rounding for one. The\n\nfix is to take the argmax of `probabilities` yourself:\n\n``` js\nconst probs = Array.from({ length: 5 }, (_, i) => v.probabilities[String(i)] ?? 0);\nconst level = probs.indexOf(Math.max(...probs));\n```\n\nIt agrees with rounding on every peaked distribution, which is why it survived\n\nmy whole test suite. It differs exactly when the distribution is skewed, which\n\nis when it matters.\n\nEvery Choice and Score comes back with `confidence`, 0 to 1. I built a \"reply\n\nhazy\" branch on the assumption that a vague question would produce a low one.\n\nThen I measured it:\n\n| question | score | confidence | \n|---|---|---|\n| Should I quit my job? | 1.97 | **0.98** | \n| Will it rain next Tuesday? | 1.99 | **0.98** | \n| Is black a color? | 3.18 | **0.33** | \n\nThe first two are as uncertain as a question gets, and confidence is 0.98.\n\nNearly all the mass sat on the level that *means* \"could go either way\" — the\n\ndistribution was a sharp spike on a level labelled uncertainty.\n\nConfidence measures how **peaked** the distribution is. It answers \"did the\n\noptions separate cleanly\", and it will happily report 0.98 for a confident\n\ncoin-flip. My hazy branch would have fired on precisely the wrong questions.\n\nLow confidence has three different causes, and they need different fixes: the\n\nquestion is genuinely contested, the options overlap each other, or `state`\n\ndoesn't contain enough to decide. Only the first one is the model being honest.\n\nI tried to work out the formula by fitting twelve real responses. `max(p)` gets\n\nclosest at 0.04 mean error and matches several exactly, but not all —\n\n*\"will humans land on Mars before 2050?\"* reported 0.58 against a peak of 0.50,\n\nhigher than the peak. The docs are upfront that it's a convenience statistic and\n\nhand you the full `probabilities` so you can compute your own. If a decision\n\nrides on it, do that.\n\nThe classic 8 Ball has twenty answers. My first design asked Jev to pick one, as\n\na Choice.\n\nTen of those twenty mean \"yes\". \"It is certain\", \"Without a doubt\", \"Yes\n\ndefinitely\" — the same claim in different words. Here's what happens, on one\n\nquestion with four different option sets:\n\n| options offered | answer | winner's share | \n|---|---|---|\n| `yes` /`no` | yes | 69% | \n| `yes` /`definitely` /`certainly` /`no` | yes | **48%** | \n\nThe total yes-mass is identical: 70% in the second row, spread across three\n\nnear-synonyms. Nothing changed about the question. The options ate each other,\n\nand confidence fell from 0.38 to 0.31 reporting a disagreement that existed only\n\nin my option list.\n\nSo the ball asks Jev for one of **five ordered buckets**, and the code picks\n\nwhich of that bucket's phrasings to show. One judgment with a right answer goes\n\nto the model; the theatre stays in software.\n\nYour question IDs never reach the model. The `instructions` and every entry in\n\n`criteria` do — names and descriptions both. The option list isn't a filter you\n\napply to a result. It's part of what you're asking.\n\nSame question, same model, options changed:\n\n| options offered | answer | \n|---|---|\n| `yes` /`no` | **yes** (69%) | \n| `yes_everyday` /`no_physics` /`depends` | **depends** (51%) | \n\n\"Depends\" won — an answer that simply didn't exist in the first row. Black is a\n\ncolour in everyday use and the absence of light in physics, so \"depends\" is\n\narguably the best answer available. It was unreachable until I offered it.\n\nThe docs put it plainly: the model cannot choose an omitted value. Leave an\n\noption out and you haven't biased the result, you've made it impossible.\n\nEvery judgment above cost about **$0.000014**. Jev charges $0.042 per million\n\ninput tokens and nothing for output, against $1.00/$5.00 for the cheapest\n\nfrontier model I'd otherwise reach for — roughly 27× on a like-for-like\n\nclassification, before an LLM's format instructions and reasoning tokens widen\n\nit further.\n\nThat price only matters because of what you give up. Jev writes no prose,\n\nexplains nothing, and can't answer anything whose answer space you can't\n\nenumerate. For summarising, drafting, coding or open-ended reasoning it's the\n\nwrong tool entirely.\n\nWhat it's good at is the narrow decision a pipeline makes ten thousand times a\n\nday, where you need a number you can threshold on. The documented pattern is to\n\nput it in front of the expensive work: score everything, route the confident\n\ncases to deterministic code, escalate the rest to a model or a person. You pay\n\nfourteen dollars a million to decide, and frontier prices only on the slice that\n\nearned it.\n\nOne caveat worth keeping. Calibration is a property of *groups* of predictions —\n\nacross many answers, the ones marked 0.8 should be right about 80% of the time.\n\nIt guarantees nothing about any single answer, and it's measured on TypeSafe's\n\ndata, not yours. Validate it in your own domain before trusting a threshold.\n\nThe ball is at [/jevball](https://dave8172-website.vercel.app/jevball). Ask it something you actually want to know\n\nand open the panel underneath — it shows the full distribution, the confidence,\n\nand which of the four questions produced the answer.", "url": "https://wpnews.pro/news/choice-score-and-noul-four-mistakes-with-jev-s-primitives", "canonical_source": "https://dev.to/dave8172/choice-score-and-noul-four-mistakes-with-jevs-primitives-1eng", "published_at": "2026-09-19 06:25:37+00:00", "updated_at": "2026-09-19 07:24:26.635503+00:00", "lang": "en", "topics": ["ai-products", "ai-tools", "artificial-intelligence", "large-language-models"], "entities": ["TypeSafe", "Jev", "Magic 8 Ball", "Jevball"], "alternates": {"html": "https://wpnews.pro/news/choice-score-and-noul-four-mistakes-with-jev-s-primitives", "markdown": "https://wpnews.pro/news/choice-score-and-noul-four-mistakes-with-jev-s-primitives.md", "text": "https://wpnews.pro/news/choice-score-and-noul-four-mistakes-with-jev-s-primitives.txt", "jsonld": "https://wpnews.pro/news/choice-score-and-noul-four-mistakes-with-jev-s-primitives.jsonld"}}