{"slug": "ask-twice-jev-s-choice-splits-its-own-vote", "title": "Ask twice: Jev's Choice splits its own vote", "summary": "A developer built a two-stage emotion-naming system on TypeSafe's System One model, in which a first Choice selects an emotion family and a second selects the specific shade within that family, scoring 22 of 24 on a hand-labeled test set. The same developer found that a single 60-word Choice collapsed confidence because near-synonyms split probability, and that lexicon-based approaches using Warriner, Kuperman & Brysbaert valence/arousal/dominance norms scored only 3 of 24. The work concludes that deciding which parts of a task belong to code versus the model is a design decision with a measurable cost, and that a Choice's confidence reflects distribution concentration rather than correctness.", "body_md": "Jev is a System One model from TypeSafe. It reads natural language like any LLM,\n\nand instead of writing a reply it returns a probability distribution over options\n\nyou define. No prose, no reasoning trace, no JSON to repair.\n\nI spent a day building waif, which reads a piece of\n\ntext and names the feeling in it. That job has no right answer, which makes it an\n\nunusually honest test rig: nothing can be graded against a key, so every design\n\ndecision has to be either argued or measured. I measured five.\n\nThe obvious build is one Choice over sixty emotion words. It does not work, and\n\nthe reason is structural rather than a prompting problem you can fix with better\n\nwording.\n\n*Annoyed*, *irritated* and *frustrated* are one feeling in three wordings. A\n\nChoice divides probability between them, so a text the model read perfectly\n\nclearly comes back at 0.19 / 0.17 / 0.15, and confidence collapses for a reason\n\nthat has nothing to do with the input. The distribution is telling you about your\n\noption list, not about the text.\n\nThe rule that falls out of it: **a Choice's options have to be alternatives to each other, not the same thing described from different angles.** Anything a\n\nFamilies do not have that problem. Anger, fear, sadness and shame are genuinely\n\ndifferent answers. And once the family is fixed, so are its shades — *which shade of anger* is a fair question, because the context has already ruled out the\n\nSo the naming became two Choices in sequence: the first picks the family, the\n\nsecond picks the shade from that family alone. TypeSafe's docs say a second\n\nrequest is warranted when an earlier answer determines the next question's\n\noptions. This is exactly that, and here is what it bought — scored on 24 texts\n\nagainst a list of acceptable answers for each:\n\n| Design | Score | \n|---|---|\n| Nearest word in the whole vocabulary, by published valence / arousal / dominance | 3 / 24 | \n| Nearest word within a family, by rank on the axis that separates that family | 9 / 24 | \n| Nearest word within a family, by distance in those published ratings | 12 / 24 | \n| **Family chosen by the model, then the shade chosen by the model** | **22 / 24** | \n\nBoth of the two failures were the wrong *family*. Given the family, the shade was\n\nright every single time.\n\nThe alternative was speculative fan-out: ask all eleven within-family Choices in\n\nthe first request and consume only the relevant answer. That keeps a reading to\n\none request, at roughly 3–4k input tokens against 1,250 + 444 for two. Two\n\nrequests won on cost, and on a latency story I could explain.\n\nThe first three rows of that table are me trying to do the naming in code.\n\nI had a good reason. Published human ratings exist for exactly the dimensions I\n\nwas measuring — Warriner, Kuperman & Brysbaert's norms give valence, arousal and\n\ndominance for 13,915 English words, rated by people. Placing the text on those\n\naxes and taking the nearest word looks like the principled version:\n\nresearch-backed coordinates instead of somebody's guesses.\n\nIt scored **3/24**, and the three reasons are worth knowing before anyone else\n\nreaches for an emotion lexicon:\n\nI also tried fixing the scale mismatch by z-scoring both sides against a probe\n\ncorpus. Worse, and instructively so: the corpus was negative-skewed, so the\n\nmapping inherited the skew, and a plainly warm text landed below the mean and got\n\nnamed from the sad half of the space.\n\nThe lesson is not *don't use lexicons*. It is that **the boundary between what code owns and what the model owns is a design decision with a number attached**,\n\nA Score's confidence summarises how concentrated the distribution is. It is not\n\nhow sure the model is of being right, which is the trap everyone hits first.\n\nHere is the second-order version, which cost me a bug. One reading came back at\n\n**confidence 0.72** on an axis — comfortably above any threshold I would set —\n\nwith the mass sitting **50% on one level and 49% on the one next to it**. The\n\ndistribution genuinely is peaked: nearly all the weight is in two adjacent\n\nbuckets. It is also a coin toss, and my page reported the winner in exactly the\n\nvoice it uses at 0.98.\n\nConfidence cannot catch that, because concentration across two neighbours is\n\nstill concentration. **The margin between first and second is a separate test** —\n\nand it is the one a reader actually cares about.\n\nA question set grows by accretion. Each addition looks free, and none of them\n\nannounce that they have stopped saying anything.\n\nSo run them over a corpus and look at the spread. Mine had a Noul asking *is more than one feeling present*. It returned ≥0.6 on \n\nTwo more went for a different reason. They were informative, but nothing\n\ndownstream consumed them beyond appending a line to the output. A question whose\n\nentire effect is an occasional footnote costs a reader more attention than it\n\nreturns.\n\nThe check is cheap: for every question, the min, max and spread of its answer\n\nacross a representative corpus. Anything that barely moves is either a gate or a\n\nmistake.\n\n**A rubric level must not contain a word from another axis.** My control rubric\n\nhad a level reading *\"Overwhelmed: struggling to keep any grip on it\"*. That\n\nprimes the model with a feeling while asking it about agency, and it labels the\n\noutput with a word that is not a position on a control scale at all. Every level\n\nof an axis has to be a point on that axis.\n\n**Input is dominated by your rubrics, not by your input.** Criteria are sent on\n\nevery call, so a one-line text costs almost exactly what a paragraph does. At\n\nthis size requests are the scarce resource and tokens are not — which is the\n\nwhole argument for batching every independent question into one call, and the\n\nreason a *dependent* second call is a real cost rather than a rounding error.\n\nwaif is live at [dave8172-website.vercel.app/waif](https://dave8172-website.vercel.app/waif). Every number in this post is on the page, under\n\n*How it works*, next to the vocabulary it was scored against.", "url": "https://wpnews.pro/news/ask-twice-jev-s-choice-splits-its-own-vote", "canonical_source": "https://dev.to/dave8172/ask-twice-jevs-choice-splits-its-own-vote-58kk", "published_at": "2026-09-20 11:23:30+00:00", "updated_at": "2026-09-20 11:54:19.225094+00:00", "lang": "en", "topics": ["ai-tools", "natural-language-processing", "large-language-models", "ai-products"], "entities": ["TypeSafe", "System One", "Jev", "waif", "Warriner", "Kuperman", "Brysbaert"], "alternates": {"html": "https://wpnews.pro/news/ask-twice-jev-s-choice-splits-its-own-vote", "markdown": "https://wpnews.pro/news/ask-twice-jev-s-choice-splits-its-own-vote.md", "text": "https://wpnews.pro/news/ask-twice-jev-s-choice-splits-its-own-vote.txt", "jsonld": "https://wpnews.pro/news/ask-twice-jev-s-choice-splits-its-own-vote.jsonld"}}