My name is Jev

tl;dr: Jev was more accurate, faster and cheaper than GPT-4o mini in this small classification experiment.

What is Jev even? #

You've probably seen Jev pop up in all of your social media feeds. But what is it even?

Jev is a model from TypeSafe built to make decisions, such as choosing which category a piece of text belongs to.

This weekend I played around with a small toy problem to understand it better.

Let's say you want to classify some text. For example, you have news articles and you want to classify them into different themes.

Here are two ways you can do this:

1. LLM (large language model): Give the text + a list of categories to an LLM and ask the LLM to categorise it.

2. Jev : You give Jev the sameinput , but instead of predicting the next token (a small piece of text), it predicts thelikelihood of the text belonging to each category and picks one.

So is Jev better? #

To find out, I did a small experiment. I looked for public datasets that were already labeled. I found these three:

1. The AG News dataset contains articles that are categorised according to theme:world news ,sports ,business , orscience and technology .
2. A question classification dataset where the task is to identify the kind of question, with these categories:people or groups ,places ,numbers ,descriptions orexplanations ,abbreviations , andentities .
3. The SNIPS user intention dataset that classifies what a user wants to do:play music ,add music to a playlist ,book a restaurant ,check the weather ,rate a book ,find a creative work such as a film or song , orfind movie showtimes .

Then I hid the labels and gave Jev (1.13.0) and GPT-4o mini (2024-07-18) the same tasks: classify the piece of text using the supplied list of labels and return probabilities for each label. I could then use the hidden labels to score the accuracy of each model.

Results

So we see 91.7% accuracy for Jev and 81.3% for GPT-4o mini. I counted answers with probabilities that didn't add up to one as wrong. Jev was also faster and cheaper.

You can reproduce this experiment by using https://github.com/emilesilvis/jev-experiment.