{"slug": "my-name-is-jev", "title": "My name is Jev", "summary": "TypeSafe's Jev 1.13.0 model outperformed GPT-4o mini (2024-07-18) on a small text-classification experiment run by developer Emil Silvis, averaging 91.7% accuracy versus 81.3% across three public datasets. Jev scored 86.8% on AG News topics, 91.6% on question types, and 96.9% on SNIPS user intentions, while GPT-4o mini scored 77.0%, 77.0%, and 90.0% respectively; Jev was also faster and cheaper, according to the experiment published on GitHub.", "body_md": "# My name is Jev\n\n**tl;dr: Jev *was* more accurate, faster and cheaper than GPT-4o mini in this small classification experiment.**\n\n## What is Jev even?\n\nYou've probably seen Jev pop up in all of your social media feeds. But what is it even?\n\nJev is a model from [TypeSafe](https://docs.typesafe.ai/concepts/system-one) built to make decisions, such as choosing which category a piece of text belongs to.\n\nThis weekend I played around with a small toy problem to understand it better.\n\nLet's say you want to classify some text. For example, you have news articles and you want to classify them into different themes.\n\nHere are two ways you can do this:\n1. **LLM (large language model)**: Give the text + a list of categories to an LLM and ask the LLM to categorise it.\n\n1. **Jev** : You give Jev the same*input* , but instead of predicting the next token (a small piece of text), it predicts the*likelihood* of the text belonging to each category and picks one.\n\n## So is Jev better?\n\nTo find out, I did a [small experiment](https://github.com/emilesilvis/jev-experiment).\n\nI looked for public datasets that *were already labeled*. I found these three:\n\n1. The [AG News dataset](https://github.com/zhangxiangxiao/Crepe/tree/449852dc565d658e930f571a3ad2fb9107aeefb2) contains articles that are categorised according to theme:`world news` ,`sports` ,`business` , or`science and technology` .\n2. A [question classification dataset](https://cogcomp.seas.upenn.edu/Data/QA/QC/) where the task is to identify the kind of question, with these categories:`people or groups` ,`places` ,`numbers` ,`descriptions` or`explanations` ,`abbreviations` , and`entities` .\n3. The [SNIPS user intention dataset](https://github.com/sonos/nlu-benchmark/tree/b86ac7f1577868c42158d0dec77db50956046696/2017-06-custom-intent-engines) that classifies what a user wants to do:`play music` ,`add music to a playlist` ,`book a restaurant` ,`check the weather` ,`rate a book` ,`find a creative work such as a film or song` , or`find movie showtimes` .\n\nThen I hid the labels and gave Jev (1.13.0) and GPT-4o mini (2024-07-18) the same tasks: classify the piece of text using the supplied list of labels and return probabilities for each label. I could then use the hidden labels to score the accuracy of each model.\n\n### Results\n\n| Task | Jev accuracy | GPT-4o mini accuracy | \n|---|---|---|\n| News topics | 86.8% | 77.0% | \n| Question types | 91.6% | 77.0% | \n| User intentions | 96.9% | 90.0% | \n\nSo we see **91.7% accuracy for Jev and 81.3% for GPT-4o mini**. I counted answers with probabilities that didn't add up to one as wrong. Jev was also faster and cheaper.\n\nYou can reproduce this experiment by using https://github.com/emilesilvis/jev-experiment.", "url": "https://wpnews.pro/news/my-name-is-jev", "canonical_source": "https://emilesilvis.com/jev.html", "published_at": "2026-09-19 00:00:00+00:00", "updated_at": "2026-09-19 10:53:09.836812+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-products"], "entities": ["TypeSafe", "Jev", "GPT-4o mini", "Emil Silvis", "AG News", "SNIPS", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/my-name-is-jev", "markdown": "https://wpnews.pro/news/my-name-is-jev.md", "text": "https://wpnews.pro/news/my-name-is-jev.txt", "jsonld": "https://wpnews.pro/news/my-name-is-jev.jsonld"}}