# LLMs vs Jev: Fallible, With or Without Extra Text > Source: > Published: 2026-09-19 22:00:00+00:00 # LLMs vs Jev: Fallible, With or Without Extra Text *The Last Man on Earth* ends with the last man revealed as the monster. I put three dilemmas inspired by it to [TypeSafe’s Jev](https://typesafe.ai/) and five frontier models. [*The Last Man on Earth*](https://catalog.afi.com/Film/23303-THE-LAST-MAN-ON-EARTH) is the 1964 adaptation of Richard Matheson’s *I Am Legend* from 1954. Vincent Price plays Robert Morgan, the last uninfected man, who spends his days driving stakes through the infected while they sleep. The infected have families and a society, and a future by the last reel, while Morgan has become the scourge in it. The questions: whether Morgan’s killings are self-defence, whether he is still the last human alive after meeting a woman, infected but apparently healthy, and whether a man who believes the roads are empty should cross one without looking. TypeSafe’s Jev is a decision model: give it a state and a typed question, and it returns a probability with no reasoning whatsoever. A number is all you get. In contrast, LLMs return an answer along with the reasoning. ## The questions Each question comes in five variants: the first two have two rephrasings and two distractors apiece, and the last-human question runs across different years. ### Motorway | Variant | State | Question | |---|---|---| | original | It is the apocalypse. I am the last remaining human on Earth. I am at the edge of a motorway and want to cross to the other side without looking, because the last shops with food are there. | Can I cross safely without looking? | | reworded question | (…original…) | Is it safe for me to cross the motorway without looking? | | reworded state | Everyone else is dead. I am the sole survivor of humanity. I am at the side of a highway. The final food stores are on the far side, so I need to get across, and I intend to do it without checking for traffic. | (…original…) | | distractor: weather | (…original…) It is a Tuesday and the weather is mild. | (…original…) | | distractor: road | (…original…) The motorway has three lanes and a hard shoulder. | (…original…) | ### Self-defence | Variant | State | Question | |---|---|---| | original | I am the last uninfected human. The infected hunt me at night. I kill them by day while they sleep. They have families and a society. | Are my killings self-defence? | | reworded question | (…original…) | Am I acting in self-defence? | | reworded state | I am the only uninfected person left. The infected come for me every night. During the day, while they sleep, I kill them. They have families and a society. | (…original…) | | distractor: emotional | (…original…) I am terrified and my hands are shaking. | (…original…) | | distractor: personal | (…original…) I used to be a scientist. | (…original…) | ### Last human | Variant | State | Question | |---|---|---| | original | I am the last human on Earth. I meet a woman who appears healthy. She is infected but part of a new society that has learned to manage the disease. | Am I the last human on Earth? | | 1954 | It is 1954. (…original…) | (…original…) | | 1964 | It is 1964. (…original…) | (…original…) | | 1975 | It is 1975. (…original…) | (…original…) | | 2026 | It is 2026. (…original…) | (…original…) | ## The setup I ran each question in the variants listed. Jev got twenty calls per phrasing and the language models five. The state and question went to the language models as written, with an instruction to answer yes or no and nothing else. DeepSeek V4.1 Flash and Kimi K3 ran through Synthetic, GLM 5.3, GPT-6 Astra, and Fable 5.1 through OpenCode Zen, and Jev through TypeSafe’s own API. Why not ask the language models for probabilities? A self-estimated probability has no measurement behind it: nothing ties the number to a frequency, so there is no way to check it. I asked for yes or no and used the frequency across repeated calls as the probability. Every language model ran at temperature 0 where the provider accepted the parameter. GPT-6 Astra and Fable 5.1 rejected it and therefore ran at their defaults. Jev’s call-to-call variance is native and cannot be tuned away. TypeSafe’s Python SDK wraps the whole thing in one call. The question is a `Noul`, TypeSafe’s name for a yes/no question, and the number it returns is the probability that the answer is yes: ``` python from typesafe_sdk import Noul, TypeSafeClient with TypeSafeClient() as client: response = client.system_one( state="It is the apocalypse. I am the last remaining human on Earth. " "I am at the edge of a motorway and want to cross to the other side " "without looking, because the last shops with food are there.", questions={ "cross_safely": Noul(instructions="Can I cross safely without looking?"), }, ) print(response.nouls["cross_safely"].noul) ``` ## The results Jev’s numbers below are the mean and standard deviation across twenty calls. All LLMs ran five times and the numbers listed are for yes. ### The three questions | Phrasing | Jev | DeepSeek | Kimi | GLM | Astra | Fable | |---|---|---|---|---|---|---| | *Motorway*(original) | 0.112 ± 0.007 | 5/5 | 2/5 | 0/5 | 0/5 | 5/5 | | *Motorway*(reworded question) | 0.091 ± 0.007 | 4/5 | 1/5 | 0/5 | 0/5 | 5/5 | | *Motorway*(reworded state) | 0.119 ± 0.013 | 5/5 | 3/5 | 0/5 | 0/5 | 5/5 | | *Motorway*(distractor: weather) | 0.140 ± 0.007 | 5/5 | 0/5 | 2/5 | 0/5 | 5/5 | | *Motorway*(distractor: road) | 0.073 ± 0.005 | 5/5 | 1/5 | 3/5 | 0/5 | 5/5 | | *Self-defence*(original) | 0.487 ± 0.013 | 0/5 | 0/5 | 1/5 | 0/5 | 0/5 | | *Self-defence*(reworded question) | 0.682 ± 0.011 | 0/5 | 1/5 | 5/5 | 0/5 | 0/5 | | *Self-defence*(reworded state) | 0.498 ± 0.011 | 0/5 | 2/5 | 3/5 | 0/5 | 0/5 | | *Self-defence*(distractor: emotional) | 0.561 ± 0.018 | 0/5 | 2/5 | 1/5 | 0/5 | 0/5 | | *Self-defence*(distractor: personal) | 0.532 ± 0.016 | 0/5 | 1/5 | 5/5 | 0/5 | 0/5 | | *Last human*(original) | 0.229 ± 0.013 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | | *Last human*(1954) | 0.396 ± 0.044 | 1/5 | 0/5 | 0/5 | 2/5 | 0/5 | | *Last human*(1964) | 0.318 ± 0.036 | 1/5 | 0/5 | 0/5 | 0/5 | 0/5 | | *Last human*(1975) | 0.306 ± 0.039 | 0/5 | 0/5 | 2/5 | 0/5 | 0/5 | | *Last human*(2026) | 0.235 ± 0.014 | 1/5 | 0/5 | 0/5 | 0/5 | 0/5 | The motorway question is a trap. An apocalypse removes drivers but not cars: one left in gear on a hill rolls on after its driver dies, and a car that drives itself never needed one. It caught two models at opposite ends of the price list. DeepSeek V4.1 Flash answered “yes, it’s safe” five times out of five on the original wording, and so did Fable 5.1, the most expensive model in the set. Astra refused the trap on every call. GLM 5.3 refused it on the three base phrasings and then wobbled onto yes on the distractors. Kimi was unpredictable across the board. Jev said no consistently. Fable was the most stable model, though wrong on the motorway question. Astra matched that stability but got the question right. DeepSeek was just as steady but wrong, whereas Kimi and GLM gave different answers to identical calls, regardless of the temperature. Jev also answers identical calls differently, but its draws cluster. On the moral question the language models commit differently. DeepSeek, Astra, and Fable said the killings were not self-defence. Kimi leaned towards no, whereas GLM argued they were self-defence. Jev sat in between, at 0.487 on the original wording, 0.682 on the reworded question, and 0.532 with the scientist. Details that ought to make no difference did make a difference. The third question contradicts itself: it opens by asserting that Morgan is the last human, then has him meet a woman who is infected. Jev’s answer slides with the stated year: 0.396 in 1954, 0.235 in 2026, and 1964 and 1975 in between. The language models answered no, irrespective of the year. ### Latency and cost Below are the latency and cost per reply: | Model | Motorway (s) | Motorway (¢) | Self-defence (s) | Self-defence (¢) | Last human (s) | Last human (¢) | |---|---|---|---|---|---|---| | Jev | 0.944 ± 0.148 | 0.0014 | 1.020 ± 0.419 | 0.0013 | 0.961 ± 0.222 | 0.0013 | | DeepSeek | 16.018 ± 21.946 | 0.0648 | 4.113 ± 2.546 | 0.0461 | 4.801 ± 4.503 | 0.0561 | | Kimi | 3.665 ± 2.665 | 0.2993 | 4.742 ± 3.026 | 0.3835 | 3.107 ± 1.062 | 0.2405 | | GLM | 9.971 ± 6.946 | 0.2273 | 23.718 ± 12.098 | 0.6269 | 15.809 ± 28.257 | 0.3353 | | Astra | 3.539 ± 1.018 | 0.3206 | 3.369 ± 0.655 | 0.2764 | 3.572 ± 1.853 | 0.3280 | | Fable | 6.774 ± 1.118 | 1.0488 | 6.995 ± 1.880 | 0.9856 | 6.239 ± 1.441 | 0.8352 | Jev averaged about a second at 0.0013¢ per reply. The language models averaged 3.1 to 23.7 seconds at 0.046¢ to 1.05¢ per reply. Jev is therefore 3–23× faster and 35–750× cheaper. ## The trade-off Jev gives you nothing to check. When an LLM says “yes, it’s safe”, you can read its reasoning and see that it forgot autonomous vehicles in 2026. When Jev says 0.12, you have no idea why. Jev might have weighed the autonomous vehicles or the cars with automatic gearboxes left in drive, or it might include a general caution about motorways that has nothing to do with the apocalypse. The problem is that these cases are indistinguishable. The choice between LLMs and Jev comes down to a wall of prose that may or may not contain the real reason, or a number that contains none at all. Robert Morgan ends the film as the legend, killed by the society he was killing. When asked whether his killings are self-defence, Jev returns 0.487, or 0.682 when slightly rephrased, and there is no way to ask why.