{"slug": "we-measured-the-200x-claim-and-got-it-wrong-twice-first", "title": "We Measured the 200x Claim, and Got It Wrong Twice First", "summary": "A developer benchmarked TypeSafe AI's Jev model against a mid-size open model on a real transactional email classification workload and initially produced two wrong results, including a 100% detection figure that also recommended suspending an account it had classified as a legitimate developer running test sends. The error stemmed from asking Jev two parallel questions that cannot see each other's answers, so the action question guessed blind; the fix, per TypeSafe's own guidance, was to keep policy in code and derive the action from the model's verdict.", "body_md": "Last week we wrote about [the classification problem hiding in your LLM bill](https://dev.to/devopsdaily/jev-and-the-classification-problem-hiding-in-your-llm-bill-191j), and about how to read a \"193.6x faster\" claim before repeating it. The post ended with a line admitting we had no access to the model in question, so every figure in it was the vendor's.\n\nWe have access now. So we ran it against a real production workload, next to that workload, with the same inputs going to both models.\n\nThe headline result is fine and slightly boring. The interesting part is that we produced two confident, wrong numbers before we got there, and one of them was the exact mistake we had criticised the vendor for a week earlier.\n\nThe workload is real and small enough to describe completely. A transactional email service has an admin page with a button on it: review what this account has been sending. Pressing it gathers volume counters for the account, the domains it mails, the link hostnames in its messages and a dozen recent subject lines, hands them to a model, and gets back a classification from a fixed list, a confidence, and a few sentences of reasoning. Message bodies are not part of what the model sees.\n\nTwo things make it a good test subject. It is a bounded decision, which is exactly the shape the new model class claims to be built for. And the button `await` s the model inside the request handler, so the admin sits there until it answers. Latency is not an abstraction, it is someone tapping a desk.\n\nThe comparison is between the model we already run, a mid-size open model on a serverless inference endpoint, and Jev, from TypeSafe AI. Not a frontier model, which matters when reading their published multiple.\n\nThe API takes a state object and a set of typed questions. We asked it two: which category is this account, and what should the operator do about it. Each question gets a list of options and returns one of them with a probability for each.\n\nThe first run looked spectacular. It caught every bad account in the sample. A hundred percent.\n\nIt also recommended suspending an account it had, in the same response, classified as a developer running test sends. That is not a borderline call. It is incoherent, and incoherent results are a gift, because they are impossible to talk yourself into.\n\nThe documentation says it plainly: questions in one request run in parallel and cannot see one another's answers. It is in their [guidance on composing questions](https://docs.typesafe.ai/concepts/how-to-build-with-system-one.md), alongside the advice to keep policy in code and raw judgments reusable, which is the fix we ended up applying. Our second question was choosing an action without knowing the verdict, so it was guessing from the raw state every time, and its guess skewed hard toward the severe option.\n\nThe model we already run does not hit this particular failure, because it produces its verdict and its action in one pass of generated text, so the action is written after the verdict. That is not a guarantee of consistency, it just removes the way we broke it here.\n\nThe fix is the one their own guidance recommends: keep the policy in code.\n\n```\n// The rule the text model gets in its prompt, written out instead of asked for.\nfunction deriveAction(verdict: Verdict, confidence: number): Action {\n  if (verdict === \"spam\" || verdict === \"phishing\") {\n    return confidence >= 80 ? \"suspend\" : \"hold_sends\";\n  }\n  if (verdict === \"suspicious\" || verdict === \"unclear\") return \"watch\";\n  return \"none\";\n}\n```\n\nAsk the model for the judgment. Derive the decision yourself. It is faster, one question instead of two, it is auditable, and it cannot contradict itself.\n\nThe general lesson is not about this API. It is that a benchmark comparing two tools has to give both of them their best shape. We had accidentally handed one of them a question it could only answer blind, and the result was a number flattering enough that we nearly wrote it down.\n\nWith the harness fixed, we needed to know whether the answers were any good, not just fast.\n\nGround truth was the part we were pleased with. Not another model's opinion, which is the thing we criticised in the last post, but real outcomes: accounts a human administrator had actually suspended, and accounts still running normally. Independent of both models, because this feature is advisory and has never suspended anyone.\n\nThe result came back stark. Across the eight human-suspended accounts that both models scored, neither ever reached the two verdicts that trigger action, spam or phishing, except twice from Jev. The model we run reached them zero times.\n\nThat reads as a strong finding. It says the feature does not work, and that swapping the model would only make a broken thing faster. We said so, twice, with some confidence.\n\nIt was wrong, for two reasons that compound.\n\nThe feature had its first run on 7 September. Of the fourteen suspended accounts on record, twelve were suspended between May and August, before it existed. Ten of those fourteen were suspended by a human; the rest were automatic reputation suspensions, which are a different question. The models scored eight of the human ones.\n\nAnd the sample they receive counts volume over a trailing thirty day window. For eight of the ten human-suspended accounts, every counter read zero:\n\n```\nvolume: { last24h: 0, last7d: 0, last30d: 0, sampled: 50 }\n```\n\nBoth models were being asked to judge accounts that had not sent anything in a month, and both answered that nothing much was happening. Which is correct. A dormant account is not a threat.\n\nThat input does not occur in production, because reviews only fire on accounts that are actively sending. We had built a test population that cannot exist, then drawn a conclusion about a feature from how two models behaved on it.\n\nThere is a second problem with how we counted. We had defined a catch as reaching spam or phishing. But the model we run called four of those eight accounts **suspicious** , and a suspicious verdict already raises an alert for an administrator. Under the definition that matches what the system actually does, it was not silent at all. We had picked a threshold that made it look silent.\n\nOnly two of the accounts were suspended after the feature existed and still had real recent volume, so only two were a fair test. On the first, Jev said phishing and recommended suspension, and our model returned one of its three unreadable replies, so it gave no verdict at all. On the second, both said suspicious, which alerts.\n\nTwo cases do not prove a feature works. What they do is remove the evidence that it was broken, which is the claim we had been about to publish.\n\nThis is the same mistake as benchmarking from a laptop on the other side of the country. Not identical in mechanism, identical in kind: a measurement taken in conditions that do not match the thing you are claiming to describe.\n\nFifty accounts, one call to each model per account, identical state, run on the same host as the workload rather than from a laptop.\n\nThree of the fifty are missing from every figure below, because our existing model answered with text the parser could not read and the harness recorded no timing for those attempts. Two were accounts a human had suspended and one was an automatic reputation suspension, so all three come from the half of the sample we most wanted to see it handle. The latency and cost numbers therefore describe our model only on the calls where it succeeded. We cannot say which way that biases them, because we have no measurements for the calls it failed. Forty-seven complete pairs remain.\n\n**Time to answer, same workload, same inputs**\n\n|  | Value | Series | \n|---|---|---|\n| median | 7326ms | existing model | \n| p95 | 11517ms | existing model | \n| worst | 12713ms | existing model | \n| median | 605ms | Jev | \n| p95 | 671ms | Jev | \n| worst | 687ms | Jev | \n\n*47 complete pairs out of 50 accounts, one call each, run on the application host. The 3 excluded are calls our existing model answered unparseably, with no timing recorded. Lower is better.*\n\nTwelve times faster at the median. Before repeating that, do to it what we told you to do to the vendor's number.\n\nOur existing model writes prose as well as a verdict: a median of 459 output tokens per call, against 138. So it is doing more work, and some of the 12x is that rather than speed.\n\nDivide latency by output tokens and the gap narrows to roughly **3x**. Roughly, because how you average changes the answer:\n\nWe are quoting the first, which is the more conservative of the two. Neither is wrong. The point is that two reasonable methods land 14% apart on the same data, which is worth knowing before anyone quotes a decimal place back at you.\n\nBe careful what this adjusted figure means. It is not a measure of raw model speed: it still contains the network, the queueing and the time to read the input, none of which scale with output length. It says the workload as we run it is 12x, and that a meaningful part of that is our own choice to ask for paragraphs. It does not prove we would get most of that back by shortening the prompt. We have not run that test, so we are not claiming it.\n\nThe spread. Across 47 calls, Jev's slowest was 687ms and its fastest 526ms, a standard deviation of 38ms. Our existing model ranged from 3.9 to 12.7 seconds, a standard deviation of 2353ms.\n\nFor a background job, nobody cares. For a button a person is waiting on, the p95 is the experience, and a p95 of 11.5 seconds is a button people learn not to press. Predictability turned out to matter more than the average, which is not what we went looking for.\n\nAt published prices, $0.055 per million input tokens and $0.85 per million output for our current model, and $42 per billion input tokens for Jev:\n\nThat is 7x, and it is real, but not for the reason it looks like.\n\nSplit our own bill: **$0.089 of input and $0.435 of output** per 1000 calls. Output is 83% of it, because output costs 15.5x input on that provider.\n\nNow compare the two tariffs. Per input token they are $0.055 and $0.042, which is **1.31x**. Nearly the same. On this run the input spend landed 1.18x apart, because Jev's structured state used about 10% more input tokens than our rendered prompt did.\n\nThe whole of the rest is that **Jev does not charge for output at all**. Their usage dashboard says so in a footnote: \"Estimated at $0.042/MTok input, free output\".\n\nThat is worth separating from \"it is a cheaper model\", because they are different claims with different lifespans. Per token of input, the two are within a third of each other. The 7x comes from a pricing decision, that output is free, and pricing decisions are the easiest thing for a company to change. The engineering difference is real and measurable. The billing difference is a choice someone made and can unmake.\n\nIf you are budgeting on this, budget on the input price and treat free output as a discount that may not last.\n\nSo it is cheaper because it says less. Any vendor comparison where one side is answering a different question is really a comparison of the questions.\n\nThree of fifty calls to our existing model returned text our JSON parser could not read. On this route a parse failure becomes an error page, so roughly one press in sixteen ended in a failure rather than an answer.\n\nWe cannot tell you how long those three waited. The harness recorded no timing for a call it could not parse, which is a hole in our instrumentation rather than a finding. A successful call takes 7.3 seconds at the median, so the wait was probably in that region, but probably is not measured and we are not going to print a number we do not have.\n\nA model whose API contract is \"return one of these options\" cannot fail that way. Not \"fails less often\". Per their [API reference](https://docs.typesafe.ai/api.md) the answer is one of the values you supplied, or the request errors and you handle it. We validate the returned value against our own list anyway, because a contract is a promise about an interface and not a reason to stop checking. That was the most concrete improvement of the day and it had nothing to do with being fast.\n\nTyped output still guarantees only the interface. A valid category that is the wrong category is still wrong, and no schema will tell you. But the class of failure where the model writes a perfectly good paragraph into a field expecting an enum goes away entirely.\n\nThe admin button now calls the fast model and comes back in about 600ms with a verdict and a probability for every category. The written reasoning became a second button, because it costs several seconds and the reader usually does not want it.\n\nShowing the distribution instead of prose turned out to be an improvement on its own. Four well-formed sentences read as confidence whatever the model actually thought. This does not:\n\n```\nphishing 83%\ntesting 8%\nsuspicious 7%\nspam 2%\n```\n\nBackground reviews, where nobody is waiting, still use the existing model. And the choice lives in a settings row rather than an environment variable, so going back is one request with no deploy. If you are trialling a vendor in a path that matters, build the way back before you need it.\n\nThe numbers are ours and they will not be yours. The method is transferable.\n\n**Measure next to the workload.** We said this last week about someone else's laptop benchmark. It is easy to agree with and easy to skip.\n\n**Give both tools their best shape.** We asked one side a question it could only answer blind, then scored it on the answer. That measures our misunderstanding of the interface, not the tool.\n\n**Check that your test population can actually occur.** This is the one we would have caught if we had asked a single question earlier: does this input ever reach the thing in production? Eight of our ten cases could not have.\n\n**Divide the headline by what is actually different.** 12x became about 3x once we accounted for output volume. The 7x on cost survived, but turned out to be a billing decision rather than a cheaper model: per input token the two are 1.31x apart, and the rest is that output is free. Neither of those makes the tool worse. They move the credit to the right place, which matters when you are guessing what will still be true next year.\n\n**Be suspicious of a result that flatters you.** Both of our wrong numbers were interesting. A 100% catch rate was interesting. \"The feature is broken\" was interesting. What survived checking is duller: about 3x once you adjust, a cost comparison we cannot complete, and no evidence either way about detection. Dull is not proof of correctness, but interesting is a reason to look twice.\n\nWe published a checklist last week for reading other people's numbers. Most of it applies to your own, and your own are the ones you are most likely to believe.\n\n*Performance figures here are our measurements on one workload on 19 September 2026, run on the application host. Prices are as published on that date. TypeSafe AI's published claims are their own, measured differently, against different models. Our raw harness is in the repository that produced these numbers.*\n\n*Originally published at [devops-daily.com](https://devops-daily.com/posts/we-measured-the-200x-claim).*", "url": "https://wpnews.pro/news/we-measured-the-200x-claim-and-got-it-wrong-twice-first", "canonical_source": "https://dev.to/devopsdaily/we-measured-the-200x-claim-and-got-it-wrong-twice-first-5ch5", "published_at": "2026-09-21 15:00:00+00:00", "updated_at": "2026-09-21 15:25:49.043724+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-agents", "ai-products"], "entities": ["TypeSafe AI", "Jev"], "alternates": {"html": "https://wpnews.pro/news/we-measured-the-200x-claim-and-got-it-wrong-twice-first", "markdown": "https://wpnews.pro/news/we-measured-the-200x-claim-and-got-it-wrong-twice-first.md", "text": "https://wpnews.pro/news/we-measured-the-200x-claim-and-got-it-wrong-twice-first.txt", "jsonld": "https://wpnews.pro/news/we-measured-the-200x-claim-and-got-it-wrong-twice-first.jsonld"}}