{"slug": "jev-in-the-wild-a-decision-model-on-my-own-work", "title": "JEV in the wild - a decision model on my own work", "summary": "A developer tested TypeSafe's Jev, a decision model built for fixed-answer questions, against frontier models on personal tasks including note comparison, correction detection, and event selection for a daily brief. On a 26-pair memory test, Jev matched Qwen 3.6 Plus at 25 correct across three runs while answering in a median 0.3 seconds versus 14.2 and at roughly 105 times lower recorded cost per answer. On a 63-candidate event-selection set, the chosen Jev setup said yes to 62 candidates, and the developer found that phrasing the question as include/drop versus a yes/no probability changed the outcome for the same event.", "body_md": "The daily brief a concept most of us is trying to reinvent, mine is supposed to help me find things worth leaving the house for. On 2 October, I tried asking Jev to make one of its decisions: should this event/hackathon go in on my list? Sourced by Luma, Devpost and across the internets.\n\nThe candidates included rock-paper-scissors by the Eiffel Tower, AI Builders and Databases Paris, and Dog fest. Different dates, all real listings from the history of my briefs.\n\nThere were 63 candidates in the test set. The Jev setup we picked in an earlier round said yes to 62 of them.\n\nHelpful If I would have a longer week.\n\nThat was not what I had seen in the memory test. There, Jev had matched the model I already used. So this became a bigger experiment. Could it check my notes? Tell whether a source supported a claim? Notice when I was correcting something? Could I trust it when an agent says a piece of work is done?\n\nI also wanted to know what happened against the frontier models I actually use. We have now run that comparison, including checks against source code from three of my projects. There is a difference between finding something worth reviewing and letting it decide that the review is finished.\n\n## first, what is Jev?\n\nJev is TypeSafe’s model for decisions with a fixed set of possible answers. Give it some information and ask for a yes/no probability, a choice or a score. It gives you that answer rather than writing a paragraph around it.\n\nIf I need to know whether two notes disagree, that sounds quite appealing. I have a lot of notes. And quite a lot of corrections to things the agents have confidently told me.\n\nThe memory test had 26 pairs: 14 based on facts I had previously settled, and 12 harder comparisons. Jev got 25 right. So did Qwen 3.6 Plus. The result repeated across three runs.\n\nBoth missed the same pair, comparing a number of devices with a number of users. I am not convinced the test’s answer was right. Those can be different populations. Even the neat tie comes with a question about the test.\n\nFor those questions, Jev’s median response took about 0.3 seconds against Qwen’s 14.2. Its recorded cost per answer was about 105 times lower. That gives me a reason to try it on lots of small comparisons. It does not yet tell me which decisions I can leave to it.\n\n## choosing is harder than saying yes\n\nThe brief has limited space. Choosing one thing depends on what else is available and what I tend to find interesting. Our setup asked about each candidate separately.\n\nWe chose how to ask on 75 candidates, then tested on the 63 held back. The earlier briefs had included 16. Jev included 62.\n\nRock-paper-scissors shows how much the question mattered. Asked to choose “include” or “drop”, Jev said drop. Asked for a yes/no probability, it returned 0.14. The cutoff selected in the earlier round was 0.10, so that version included it.\n\nSame event, different ways of asking, different answers to the thing I needed to decide.\n\nMatching the earlier brief would not prove good taste either. Those reference choices came from another model. I had not personally rated all 63 candidates, and the brief’s prompt had changed during that period. The useful conclusion is narrower: this setup gave me almost the whole list back. I still had to choose.\n\n## who even uses Haiku?\n\nThat was my reaction when I read the first correction-detection comparison. We had tested whether Jev could notice messages where I tell an agent it got something wrong. The generative baseline was Haiku 4.5.\n\nThere is a sensible cheap-model comparison there. But I wanted to know whether Jev could take over something I currently ask a frontier model to do. A win on cost against Haiku would not answer that.\n\nSo we ran Jev 1.13, GPT-6 Astra and Claude Opus 5.5 on the same messages, with the same written definition of a correction. We kept the old labels, rather than changing them to agree with the new models.\n\nThere were 33 messages labelled as corrections and 59 labelled otherwise. Against those old labels:\n\n- Jev found 21 of the 33 corrections, with six false alarms among the other 59 messages.\n- Astra found 30 of 33, with 12 false alarms among the other 59.\n- Opus found 31 of 33, with 14 false alarms among the other 59.\n\nJev asked me to look at fewer messages, but missed 12 of the labelled corrections. Astra missed three; Opus missed two. If the job is remembering when I say something is wrong, those misses matter more to me than a tidy overall score.\n\nFor the 92 answers, OpenRouter reported inference charges of about $0.0021 for Jev, $0.43 for Astra and $0.28 for Opus. Median response times were about 0.29, 2.00 and 3.42 seconds. Each model saw each message once. The generative models had medium reasoning and a 2,048-token output limit. This was the same yes/no question, not the four-way Jev setup selected in the earlier Haiku comparison.\n\nThese are still model-made labels. When the models disagree, somebody has to read the message. Three models agreeing does not turn their answer into mine.\n\n## does VERIFIED mean verified?\n\nThen we gave them code from real projects: ZUP, which shows the state of my work; Transcripto, which reads agent sessions; and Helicon, which checks my stored notes. The job was specific: does this code actually support this claim about what the tool does?\n\nOne ZUP check was supposed to keep an agent’s self-graded pass from lighting a VERIFIED indicator. It did reject a record explicitly marked “same-session”. But if the check’s author was missing, that protection did not apply. The indicator could still light.\n\nAcross ten claims, each asked three times, Astra matched the source-derived answers in all 30 calls. Jev matched 26 of 30. In a fresh pass after fixing our result reader, Opus matched 30 of 30. The first Opus pass had six responses our strict reader could not use; follow-up checks found explanations followed by JSON. Those six are recorded as unusable, not wrong judgments, and the rerun is kept separate.\n\nJev approved the missing-author claim once and rejected it twice, with probabilities of 0.54, 0.48 and 0.46. Our cutoff was 0.50. It also approved another claim three times: that the writer correctly rejected `--ok false`. The code actually treats the string “false” as a true value and writes a pass.\n\nThe complete new comparison, including that parser investigation and rerun, produced 400 model responses for about $1.30 in provider-reported inference charges. That is the cost of these calls, not my subscription bill.\n\nThat is a much more useful mistake for me to see than an average score. If a model helps me find things to look at, a false alarm costs a read. If it tells me a check is safe when the code has a way around it, I may stop looking exactly where I need to look.\n\nThese were bounded source excerpts from real projects, with expected answers written from the code before the calls. They were not an autonomous repository review, an installed product trial or proof that somebody else would use the tools. The models did not run the code. We separately executed the real ZUP functions against temporary records: the missing-author record lit VERIFIED, and `--ok false` wrote a passing result. That limit matters when the word on the screen is VERIFIED.\n\n## very confident, still wrong\n\nThe earlier source-checking test had another version of this problem. Jev matched 28 of 29 model-written labels, but its one mistake accepted evidence from the wrong annex with a probability of 0.99.\n\nThen we tested arithmetic. It missed every wrong sum: five examples, each asked twice. The wrong sums did not make it noticeably unsure.\n\nDoing arithmetic and date checks in code improved the written tests. But the saved scan of my real notes reported that those checks settled none of 67,747 candidate pairs. Better written tests had not yet made that particular part useful in my actual notes.\n\n## and does it agree with me?\n\nWe also tried predicting decisions I had already made. On 17 cases with an agent recommendation, Jev picked the recommended option seven times when it could not see the recommendation. Show it the recommendation, and that became 15.\n\nI had rejected the agent’s advice in 12 cases. Jev predicted eight of those rejections with the advice hidden, and two with it visible.\n\nSo, more agreement with the AI. Less agreement with me.\n\nThose were small retrospective tests, prepared with the outcomes already known. I would not call that learning my taste. But it gives me a concrete thing to change: get the second opinion before showing it the first answer.\n\n## next: one evening, three options\n\nFor the next personal test, I want this to end with me actually going somewhere.\n\nThree events I could realistically attend on the same evening. I pick one and write down why before seeing Jev’s answer. Every model gets the same options, date, travel, cost and description of what I enjoy. Staying home is allowed too.\n\nThen we compare. Did it pick what I picked? Did reading its choice change my mind? Where did I actually go, and was I glad I went?\n\nA model could disagree with my first choice and suggest something I end up enjoying. Or predict my choice perfectly and we both pick a disappointing evening. If I never go because it rains, that belongs in the result too.\n\nFor now, I would try Jev where it can bring a suspicious pair of notes to my attention. I would keep the actual evidence beside any decision about whether work is done. And I would give the event picker a fixed number of places on the list.\n\nSixty-two plans is still quite a week.\n\n*How this was made: AI helped run the experiments and prepare this draft. The memory, brief, source-support and arithmetic results above come from the earlier saved runs. The new comparison uses the stated model routes and bounded settings, not every model’s maximum possible effort. Correction labels came from one earlier model rater; source-check expectations were written by Codex. This is a personal investigation, not an independent benchmark. The evening test has not happened. Private transcript text is not published with the results.*", "url": "https://wpnews.pro/news/jev-in-the-wild-a-decision-model-on-my-own-work", "canonical_source": "https://morkeeth.substack.com/p/jev-in-the-wild-a-decision-model", "published_at": "2026-10-04 19:12:15+00:00", "updated_at": "2026-10-23 23:43:36.068318+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-tools"], "entities": ["Jev", "TypeSafe", "Qwen 3.6 Plus", "Haiku 4.5", "GPT-6 Astra", "Claude Opus 5.5", "Luma", "Devpost"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/jev-in-the-wild-a-decision-model-on-my-own-work", "markdown": "https://wpnews.pro/news/jev-in-the-wild-a-decision-model-on-my-own-work.md", "text": "https://wpnews.pro/news/jev-in-the-wild-a-decision-model-on-my-own-work.txt", "jsonld": "https://wpnews.pro/news/jev-in-the-wild-a-decision-model-on-my-own-work.jsonld"}}