TypeSafe's Jev Can't See. I Made It Guess What I Drew Anyway

TypeSafe's Jev model, which reads text only and cannot process images, audio, or video, was tested on 400 Quick, Draw! sketches across ten categories sampled with a fixed random seed, with the median drawing taking 336 characters as SVG versus 3,018 as base64. TypeSafe documents Jev as a System One model that returns a probability for every answer and lists known weak spots including handling words better than numbers and unreliable comparison of two hex colors. The test compared Jev reading SVG, Jev reading base64-encoded PNG, and Claude Sonnet 5 viewing the actual image.

I tried to draw a bicycle using a mouse, with photos of bicycles open on a second screen. After the third attempt, I deleted bicycle from the game. I changed the game’s list of objects three times, always for the same reason. I can’t draw. Table of Contents The other player had a bigger problem. It can’t see. The other player was Jev, a model made by TypeSafe. Jev reads text and nothing else, at least for now. TypeSafe’s documentation https://docs.typesafe.ai/concepts/system-one says images, audio, and video are not supported yet. So I turned every drawing into text and made Jev guess what I drew anyway. I keep seeing posts about Jev on X. Tech bros describe it as a cheaper, faster classifier and show off ideas they came up with. My reaction: “Dude, I could have done that five years ago with XGBoost and BERT.” Their ideas undersell the model. With XGBoost or a fine-tuned BERT, every new question meant a new labeled dataset and a new training run. With Jev, I describe the classifier I want in plain English and get an answer from a single API call. I don’t collect labels, and I don’t train anything. Neither XGBoost nor BERT gave me that. TypeSafe calls Jev a System One model, after the fast, intuitive System 1 that Daniel Kahneman popularized. Jev doesn’t write text. You send it some text and a question, and it answers in a fixed format. For a multiple-choice question like mine, it returns a probability for every answer. Guessing a doodle is the most System 1 game I know. Nobody at a party reasons their way to cat . You squint at the doodle through the beer and shout. Preferably without spilling it all over yourself. TypeSafe also publishes a list of Jev’s known weak spots https://docs.typesafe.ai/model-jaggedness/jev-1.13 . One of them says Jev handles words better than numbers and can’t reliably tell whether two hex colors are close to each other. A drawing turned into text is a pile of numbers. I was about to test Jev, on purpose, on the kind of input its makers had warned me about. The game is a small web app. I draw on a canvas with the mouse, and the app squeezes my strokes into a 100 by 100 box. It drops the points that barely change the shape the Ramer-Douglas-Peucker algorithm does that part . Then it writes the result as an SVG, one <path per stroke. That string of coordinates is everything Jev gets from the drawing. Next to the drawing, Jev gets the names of ten objects and one line describing each. For airplane , the line says: a long thin body with two swept-back wings and a small upright tail fin. Those ten sentences and a one-line question are the whole classifier. In the game, Jev ranked my flower, my cat, and my tree first. None of it proved anything. I made somewhere between 20 and 30 drawings. The flower and the cat come from a different object list than the tree, and neither list is the one I tested later. A game where I delete every object I fail to draw makes a nice demo. For numbers, I needed drawings I didn’t make and rules I couldn’t change halfway through. For the test, I used the Quick, Draw https://github.com/googlecreativelab/quickdraw-dataset dataset from Google: doodles made by players of the Quick, Draw game, each drawn in under 20 seconds. I picked ten categories of objects because I can draw them more or less . From each category, I sampled 40 drawings with a fixed random seed, 400 in total. I only sampled drawings that Google’s own recognizer had classified correctly. Every number below describes sketches that were already recognizable. The drawings went through the same code as the game, so the test measures exactly what the game sends to Jev. Four setups saw all 400 drawings: - Jev reading the SVG, exactly as in the game. - Jev reading the PNG as base64 text, which is the picture itself spelled out in characters. The median drawing takes 336 characters as SVG and 3,018 as base64. - Claude Sonnet 5 looking at the actual image. A model that can see sets the ceiling. - Claude Sonnet 5 reading the same SVG as Jev. The fair fight. I wrote the analysis plan before the full run. I didn’t want to see the results first and then pick the test that flattered them. Every accuracy comes with a range, the 95% Wilson https://mikulskibartosz.name/wilson-score-in-python-example confidence interval. The range shows which true accuracies are consistent with what I measured on 400 drawings. First, each setup had to beat guessing. With ten objects, a random guess is right 10% of the time, so I checked how often pure guessing would score at least as well as each setup. That’s an exact binomial test against 10%. The main question was whether Jev is as good as Sonnet when both read the same SVG, and I defined as good in advance. Jev was allowed to trail Sonnet by up to 5 percentage points. If even the worst end of the range for the difference stayed within those 5 points, I would call Jev non-inferior, meaning no more than 5 points worse. I also compared all six pairs of setups. They all saw the same 400 drawings, so the comparison goes drawing by drawing. Drawings both setups got right, or both got wrong, say nothing about which one is better. Only the disagreements count, and McNemar’s exact test looks only at those. Six comparisons give luck enough chances to make one of them look impressive, so I adjusted the p-values with the Holm method. I only scored right or wrong. Both models report a confidence number, but Sonnet’s is self-reported, and an LLM can write down any number. Confidence stayed out of every test. - Sonnet looking at the image: about 91% 88% to 93% . - Sonnet reading the SVG: about 57% 52% to 62% . - Jev reading the SVG: about 35% 31% to 40% . - Jev reading base64: about 9% 6% to 12% . Three setups beat guessing by a wide margin p < 0.001 for each . Jev reading base64 did not. Reading the same SVG, Jev is 21.8 points behind Sonnet, and the plausible range runs from 16 to 28 points behind. My margin was 5, so the answer to the main question is a clear no. Jev doesn’t compete with Sonnet on accuracy here. All six pairwise differences held up after the correction. I opened the confusion matrix https://mikulskibartosz.name/how-to-read-a-confusion-matrix , and the 35% stopped meaning what I thought it meant. Jev answered airplane for 209 of the 400 drawings. More than half. Its accuracy per object: | Object | Correct | |---|---| | airplane | 34/40 | | skull | 28/40 | | sailboat | 27/40 | | tree | 25/40 | | fish | 13/40 | | sun | 8/40 | | house | 3/40 | | flower | 3/40 | | clock | 1/40 | | cat | 0/40 | Jev ranked my hand-drawn cat first in the game. In the test, it got none of the 40 cats. It called 23 of them airplanes and 15 of them skulls. My cat never had to compete with an airplane or a skull. The object list in that version of the game had neither. The signal is still real. Answering airplane every time would score exactly 10%, because airplanes are 40 of the 400 drawings. Jev got about 35% right, so it does read something from the coordinates. But the 35% is an average of 85% on airplanes, zero on cats, and everything in between. Why airplane? I don’t know. I can’t ask Jev either. System One models don’t explain their reasoning. Jev reading the base64 PNG scored about 9% 6% to 12% . The range includes 10%, so the result is indistinguishable from chance. It’s not evidence that Jev did worse than guessing, or that it did exactly as well. Jev wasn’t picking at random, though. It answered airplane 199 times and house 197 times. Whatever the object, 396 of its 400 answers were one of those two. With ten balanced categories, any strategy that doesn’t use the shape scores around 10%. I think the file format explains it. PNG compression loses nothing, so the whole shape is still somewhere in the text. But pixels that sit next to each other in the picture end up nowhere near each other in the base64 string, and no readable spatial structure survives. So Jev’s 35% on SVG comes from the coordinates, which still describe lines. - Only drawings that Google’s recognizer had already classified correctly. - Ten objects I chose because I can draw them. A different ten would give different numbers. - One version of each model and one attempt per drawing. I measured the difference between models, not the difference between two runs of the same model. - 400 drawings. Enough for the large differences above, too few for claims about any single object. Sonnet does far better on the same SVG, and better still when it can see the picture. For practical sketch recognition, Jev isn’t useful. TypeSafe sells Jev as a text model and lists numbers among its weak spots. I fed it pictures written as numbers. It still found enough shape in the coordinates to beat chance, even if much of what it found was airplanes. I pushed Jev somewhere it shouldn’t go, and it wasn’t that bad. Quick, Draw Dataset by Google, licensed under CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ . https://github.com/googlecreativelab/quickdraw-dataset https://github.com/googlecreativelab/quickdraw-dataset