{"slug": "typesafe-s-jev-can-t-see-i-made-it-guess-what-i-drew-anyway", "title": "TypeSafe's Jev Can't See. I Made It Guess What I Drew Anyway", "summary": "TypeSafe's Jev model, which reads text only and cannot process images, audio, or video, was tested on 400 Quick, Draw! sketches across ten categories sampled with a fixed random seed, with the median drawing taking 336 characters as SVG versus 3,018 as base64. TypeSafe documents Jev as a System One model that returns a probability for every answer and lists known weak spots including handling words better than numbers and unreliable comparison of two hex colors. The test compared Jev reading SVG, Jev reading base64-encoded PNG, and Claude Sonnet 5 viewing the actual image.", "body_md": "I tried to draw a bicycle using a mouse, with photos of bicycles open on a second screen. After the third attempt, I deleted *bicycle* from the game. I changed the game’s list of objects three times, always for the same reason. I can’t draw.\n\nTable of Contents\n\nThe other player had a bigger problem. It can’t see.\n\nThe other player was Jev, a model made by TypeSafe. Jev reads text and nothing else, at least for now. [TypeSafe’s documentation](https://docs.typesafe.ai/concepts/system-one) says images, audio, and video are not supported yet. So I turned every drawing into text and made Jev guess what I drew anyway.\n\n## \n\nI keep seeing posts about Jev on X. Tech bros describe it as a cheaper, faster classifier and show off ideas they came up with. My reaction: “Dude, I could have done that five years ago with XGBoost and BERT.”\n\nTheir ideas undersell the model. With XGBoost or a fine-tuned BERT, every new question meant a new labeled dataset and a new training run. With Jev, I describe the classifier I want in plain English and get an answer from a single API call. I don’t collect labels, and I don’t train anything. Neither XGBoost nor BERT gave me that.\n\nTypeSafe calls Jev a System One model, after the fast, intuitive System 1 that Daniel Kahneman popularized. Jev doesn’t write text. You send it some text and a question, and it answers in a fixed format. For a multiple-choice question like mine, it returns a probability for every answer.\n\nGuessing a doodle is the most System 1 game I know. Nobody at a party reasons their way to *cat*. You squint at the doodle through the beer and shout. Preferably without spilling it all over yourself.\n\nTypeSafe also publishes a list of [Jev’s known weak spots](https://docs.typesafe.ai/model-jaggedness/jev-1.13). One of them says Jev handles words better than numbers and can’t reliably tell whether two hex colors are close to each other. A drawing turned into text is a pile of numbers. I was about to test Jev, on purpose, on the kind of input its makers had warned me about.\n\n## \n\nThe game is a small web app. I draw on a canvas with the mouse, and the app squeezes my strokes into a 100 by 100 box. It drops the points that barely change the shape (the Ramer-Douglas-Peucker algorithm does that part). Then it writes the result as an SVG, one `<path>` per stroke. That string of coordinates is everything Jev gets from the drawing.\n\nNext to the drawing, Jev gets the names of ten objects and one line describing each. For *airplane*, the line says: a long thin body with two swept-back wings and a small upright tail fin. Those ten sentences and a one-line question are the whole classifier.\n\nIn the game, Jev ranked my flower, my cat, and my tree first.\n\nNone of it proved anything.\n\nI made somewhere between 20 and 30 drawings. The flower and the cat come from a different object list than the tree, and neither list is the one I tested later. A game where I delete every object I fail to draw makes a nice demo. For numbers, I needed drawings I didn’t make and rules I couldn’t change halfway through.\n\n## \n\nFor the test, I used the [Quick, Draw!](https://github.com/googlecreativelab/quickdraw-dataset) dataset from Google: doodles made by players of the Quick, Draw! game, each drawn in under 20 seconds. I picked ten categories of objects because I can draw them (more or less).\n\nFrom each category, I sampled 40 drawings with a fixed random seed, 400 in total. I only sampled drawings that Google’s own recognizer had classified correctly. Every number below describes sketches that were already recognizable.\n\nThe drawings went through the same code as the game, so the test measures exactly what the game sends to Jev.\n\nFour setups saw all 400 drawings:\n\n- Jev reading the SVG, exactly as in the game.\n- Jev reading the PNG as base64 text, which is the picture itself spelled out in characters. The median drawing takes 336 characters as SVG and 3,018 as base64.\n- Claude Sonnet 5 looking at the actual image. A model that can see sets the ceiling.\n- Claude Sonnet 5 reading the same SVG as Jev. The fair fight.\n\n## \n\nI wrote the analysis plan before the full run. I didn’t want to see the results first and then pick the test that flattered them.\n\nEvery accuracy comes with a range, the 95% [Wilson](https://mikulskibartosz.name/wilson-score-in-python-example) confidence interval. The range shows which true accuracies are consistent with what I measured on 400 drawings.\n\nFirst, each setup had to beat guessing. With ten objects, a random guess is right 10% of the time, so I checked how often pure guessing would score at least as well as each setup. That’s an exact binomial test against 10%.\n\nThe main question was whether Jev is as good as Sonnet when both read the same SVG, and I defined *as good* in advance. Jev was allowed to trail Sonnet by up to 5 percentage points. If even the worst end of the range for the difference stayed within those 5 points, I would call Jev non-inferior, meaning no more than 5 points worse.\n\nI also compared all six pairs of setups. They all saw the same 400 drawings, so the comparison goes drawing by drawing. Drawings both setups got right, or both got wrong, say nothing about which one is better. Only the disagreements count, and McNemar’s exact test looks only at those. Six comparisons give luck enough chances to make one of them look impressive, so I adjusted the p-values with the Holm method.\n\nI only scored right or wrong. Both models report a confidence number, but Sonnet’s is self-reported, and an LLM can write down any number. Confidence stayed out of every test.\n\n## \n\n- Sonnet looking at the image: about 91% (88% to 93%).\n- Sonnet reading the SVG: about 57% (52% to 62%).\n- Jev reading the SVG: about 35% (31% to 40%).\n- Jev reading base64: about 9% (6% to 12%).\n\nThree setups beat guessing by a wide margin (p < 0.001 for each). Jev reading base64 did not.\n\nReading the same SVG, Jev is 21.8 points behind Sonnet, and the plausible range runs from 16 to 28 points behind. My margin was 5, so the answer to the main question is a clear no. Jev doesn’t compete with Sonnet on accuracy here.\n\nAll six pairwise differences held up after the correction.\n\n## \n\nI opened the [confusion matrix](https://mikulskibartosz.name/how-to-read-a-confusion-matrix), and the 35% stopped meaning what I thought it meant.\n\nJev answered *airplane* for 209 of the 400 drawings. More than half.\n\nIts accuracy per object:\n\n| Object | Correct | \n|---|---|\n| airplane | 34/40 | \n| skull | 28/40 | \n| sailboat | 27/40 | \n| tree | 25/40 | \n| fish | 13/40 | \n| sun | 8/40 | \n| house | 3/40 | \n| flower | 3/40 | \n| clock | 1/40 | \n| cat | 0/40 | \n\nJev ranked my hand-drawn cat first in the game. In the test, it got none of the 40 cats. It called 23 of them airplanes and 15 of them skulls. My cat never had to compete with an airplane or a skull. The object list in that version of the game had neither.\n\nThe signal is still real. Answering *airplane* every time would score exactly 10%, because airplanes are 40 of the 400 drawings. Jev got about 35% right, so it does read something from the coordinates. But the 35% is an average of 85% on airplanes, zero on cats, and everything in between.\n\nWhy airplane? I don’t know. I can’t ask Jev either. System One models don’t explain their reasoning.\n\n## \n\nJev reading the base64 PNG scored about 9% (6% to 12%). The range includes 10%, so the result is indistinguishable from chance. It’s not evidence that Jev did worse than guessing, or that it did exactly as well.\n\nJev wasn’t picking at random, though. It answered *airplane* 199 times and *house* 197 times. Whatever the object, 396 of its 400 answers were one of those two. With ten balanced categories, any strategy that doesn’t use the shape scores around 10%.\n\nI think the file format explains it. PNG compression loses nothing, so the whole shape is still somewhere in the text. But pixels that sit next to each other in the picture end up nowhere near each other in the base64 string, and no readable spatial structure survives. So Jev’s 35% on SVG comes from the coordinates, which still describe lines.\n\n## \n\n- Only drawings that Google’s recognizer had already classified correctly.\n- Ten objects I chose because I can draw them. A different ten would give different numbers.\n- One version of each model and one attempt per drawing. I measured the difference between models, not the difference between two runs of the same model.\n- 400 drawings. Enough for the large differences above, too few for claims about any single object.\n\n## \n\nSonnet does far better on the same SVG, and better still when it can see the picture. For practical sketch recognition, Jev isn’t useful.\n\nTypeSafe sells Jev as a text model and lists numbers among its weak spots. I fed it pictures written as numbers. It still found enough shape in the coordinates to beat chance, even if much of what it found was airplanes.\n\nI pushed Jev somewhere it shouldn’t go, and it wasn’t that bad.\n\n*Quick, Draw! Dataset by Google, licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). [https://github.com/googlecreativelab/quickdraw-dataset](https://github.com/googlecreativelab/quickdraw-dataset)*", "url": "https://wpnews.pro/news/typesafe-s-jev-can-t-see-i-made-it-guess-what-i-drew-anyway", "canonical_source": "https://mikulskibartosz.name/typesafe-jev-guess-what-i-drew", "published_at": "2026-09-19 17:46:05+00:00", "updated_at": "2026-09-19 18:24:26.360928+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-products", "ai-tools"], "entities": ["TypeSafe", "Jev", "Claude Sonnet 5", "Google", "Quick, Draw!", "Daniel Kahneman", "XGBoost", "BERT"], "alternates": {"html": "https://wpnews.pro/news/typesafe-s-jev-can-t-see-i-made-it-guess-what-i-drew-anyway", "markdown": "https://wpnews.pro/news/typesafe-s-jev-can-t-see-i-made-it-guess-what-i-drew-anyway.md", "text": "https://wpnews.pro/news/typesafe-s-jev-can-t-see-i-made-it-guess-what-i-drew-anyway.txt", "jsonld": "https://wpnews.pro/news/typesafe-s-jev-can-t-see-i-made-it-guess-what-i-drew-anyway.jsonld"}}