# Decision Models: Jev vs. a Fine-Tuned Classifier

> Source: <https://www.anaconda.com/blog/decision-time>
> Published: 2026-10-20 15:49:46+00:00

A customer is billed twice for a $340 hotel booking. Their card issuer refuses to reverse the second charge, it goes to a collections agency, and it now shows on their credit report. They file a complaint describing all of this in a few hundred words. A complaint like this needs three things:

1. A resolution path: close it with an explanation, fix something such as the credit report, or pay the customer back.
2. A yes or no on whether a person should look at it quickly.
3. A severity grade.

Most organizations handle that with hand-written rules, a conventional machine learning model, and, as a fallback, a person reading whatever the first two missed. Increasingly there is a fourth option: give the whole complaint to a large language model and ask for JSON back so it can be plugged back into a standard programmatic workflow.

That works, but when you unpack it, it’s an odd way to get a one-word answer. The model writes its answer one token at a time, may reason through hundreds of words first, bills you for all of them, and needs a schema, a validator, and retry code to land in a fixed format (though often this last part is now handled by the model provider). Its confidence is weaker still: if it writes `"confidence": 0.91`, that is text it chose to write, not a measured probability. Yet plenty of organizations have started building agentic systems for fraud screening, moderation, and routing on top of these numbers.

Last month [TypeSafe released Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev), a model that works differently. This post covers what it does, what is known about how it works, and how it compares on real complaints with a small open version we trained using [AI orchestration](https://www.anaconda.com/platform/ai-orchestration) in Anaconda Platform.

## What a Decision Model Is

[TypeSafe calls Jev a “System One model](https://typesafe.ai/blog/introducing-system-one-models-and-jev),” after Daniel Kahneman’s *Thinking, Fast and Slow*, where System 1 is fast, automatic judgment and System 2 is slow, deliberate thinking. The argument is that the language models people build software on are System 2 tools, while most decisions inside software are System 1: approve or reject, route to team A or team B, pick one of five labels.

The generic name that has stuck is “decision model.” It is a classifier: you give it some input (the “state”) and a set of questions, each with a fixed list of answers, and for each question it returns a probability for every answer. A decision model never writes text, so it cannot summarize a document, write code, or explain its choice. OpenAI has since announced a similar [Decisions API](https://openai.com/index/devday-2026-recap/), in limited preview, at DevDay on 29 September.

Kahneman’s System 1 is also the part of us that is fast and often wrong. TypeSafe sells Jev on the quality of its probabilities, and the second half of this post tests that on one task.

## Why Developers Care

Sebastian Raschka, a longtime builder of text classifiers, went from assuming he could easily build Jev himself to finding it worked better than he expected. He calls it “the ChatGPT moment for classification”: one model that handles many kinds of classification cheaply, without training a classifier per task ([Ahead of AI](https://magazine.sebastianraschka.com/p/classifier-history-and-jev)).

Three things make it useful as a building block.

**The answer is always one of your options.** A language model can return a wrong answer or an invalid one. A decision model only returns options you supplied, so the only thing left to check is whether the answer is right.

**The probability is read off the model, not written by it.** It comes from the model’s internal state, so it can be checked against outcomes, adjusted, and thresholded, though any model can still be over- or underconfident. One trap: the API’s `confidence` field is not a second accuracy estimate, but [a measure of how far the top answer stands out](https://github.com/typesafe-ai/system-one-adapter-python/blob/fb52b1030b7fc1f4f1cf39910afa5da54f9835e3/src/system_one_adapter/_utils/confidence_metrics.py) (three options with a top probability of 0.8 report 0.7). Set thresholds on the probability, and check them on your own data.

**It is cheap (and effective).** Jev is named after the economist William Stanley Jevons, who observed that making a resource cheaper tends to increase how much of it gets used. It costs [$0.042 per million input tokens](https://typesafe.ai/blog/introducing-system-one-models-and-jev), and nothing for output. Our complaints averaged about 630 tokens, so each cost around $0.00003, with a median of 88 milliseconds per answer. [Raschka ran it over the 25,000 IMDb test reviews](https://magazine.sebastianraschka.com/p/classifier-history-and-jev) for about 65 cents and scored 96.5%, on par with a [ModernBERT](https://arxiv.org/abs/2412.13663) model he fine-tuned for the task. At that price you can decide on every row, event, or log line.

## What We Know About Jev

TypeSafe has not published how Jev is built, and the model cannot be downloaded for separate inspection. The most thorough outside analysis is Archer Hume’s [Jev’s Architecture Unmasked](https://archerhume.com/posts/jevs-architecture-unmasked) (17 September 2026), built on about ten thousand probing API calls. We follow its separation of what TypeSafe says, what can be observed, and what can only be inferred.

**What TypeSafe says.** [All answers are produced at once](https://typesafe.ai/blog/introducing-system-one-models-and-jev). [Questions can be multiple choice, yes/no, or a score on a scale](https://docs.typesafe.ai/). The model is an existing language model given extra training with an unpublished method TypeSafe calls [Reinforcement Learning for Calibrated Decisions (RLCD)](https://typesafe.ai/blog/introducing-system-one-models-and-jev), aimed at making the probabilities match reality; its closest public relative is [Reinforcement Learning with Calibration Rewards (RLCR)](https://arxiv.org/abs/2507.16806), which rewards accurate stated confidence. [TypeSafe’s CEO has said](https://techcrunch.com/2026/09/18/a-new-kind-of-ai-model-from-a-chatgpt-inventor-is-thrilling-developers/) all of the training data is synthetic.

**What the API shows.** [Questions in the same request cannot see each other](https://archerhume.com/posts/jevs-architecture-unmasked/#2-share-the-state-isolate-the-questions): a secret code hidden in one question is invisible to another (probability 0.00), but found when placed in the shared input (about 0.90). Extra questions cost very little time until there are around a hundred.

The list of options affects the answers. [Adding an irrelevant fifth option to a four-option question](https://archerhume.com/posts/jevs-architecture-unmasked/#4-let-the-options-interact-before-choosing) shifted the balance between two of the original four, consistently across ten trials. That rules out scoring each option independently, where an extra option could not change the balance between two others. Order matters too: reversing it in an ordinary support-ticket question raised the top answer’s probability from around 0.84–0.89 to 0.93–0.96.

**What is guessed.** Since the model itself is proprietary, there are plenty of details that we can only infer. [Hume’s investigation](https://archerhume.com/posts/jevs-architecture-unmasked) leads him to infer a large, general language model underneath that reuses the shared input across questions; [Raschka guesses](https://magazine.sebastianraschka.com/p/classifier-history-and-jev) something small, similar to ModernBERT, which would explain the low latency. The rest of our analysis doesn’t depend on which is right.

## Why It Can Answer All the Questions at Once

When a language model reads a prompt, it [processes all the words together in one pass](https://arxiv.org/abs/1706.03762). The slow part is writing the reply, which it can only do one word after another. A decision model stops after reading and takes each question’s answer straight from that pass.

The limit is that a question cannot depend on another’s answer. Dealing with that would need a second round of questions, or one question covering both.

## Where the Promise Falls Short

**Good probabilities on a benchmark do not mean good probabilities on your data.** On a [public set of general-knowledge questions](https://archerhume.com/posts/jevs-architecture-unmasked/#5-train-the-distribution-then-calculate-confidence) Jev’s probabilities look excellent, but most of those questions are easy and answered confidently. On a family of [freshly written math problems](https://archerhume.com/posts/jevs-architecture-unmasked/#5-train-the-distribution-then-calculate-confidence) it was right 56% of the time while giving itself about 35%, which is far too cautious. You have to measure them on examples from your own workflow where you know the answer, as we do below.

**Reordering the options moves the probabilities.** A probability that moves from 0.84 to 0.95 because the options were listed differently will cross a threshold like 0.9 and change what your system does. On 40 of our complaints, Jev picked the same answer after reordering in 79 of 80 cases, but its probabilities moved by up to 0.20 and crossed 0.9 in 7 of the 80. Some of that is noise from batched GPU computation on TypeSafe’s side: a question whose options we did not touch moved by up to 0.04 between identical requests. Our own model (discussed later) did much worse. Test option order before deploying any of these models.

**It cannot reason in steps or look things up.** Those still need a language model. The sensible arrangement is TypeSafe’s own suggestion: the decision model decides, and the language model writes.

## The Task

To test the capabilities of these models, we decided we’d use what is known about the architecture, combine it with a fine-tune, and apply it to a common business problem. The data we built our model on is the [CFPB Consumer Complaint Database](https://www.consumerfinance.gov/data-research/consumer-complaints/) of complaints about US financial companies, each with the consumer’s account and the company’s reported response. The CFPB [stopped publishing the accounts in August 2026](https://www.consumerfinance.gov/about-us/newsroom/the-cfpb-to-cease-discretionary-publication-of-complaint-narratives-and-visualizations/), so we used an earlier copy.

We asked three questions of each complaint:

- How was it resolved? Closed with an explanation only, with non-monetary relief (a fix such as correcting a credit report), or with monetary relief. The answer is what the company says it did.
- Does it need quick human review? Yes or no.
- How severe is it? Low, medium, or high.

Only the resolution has a real recorded answer. Payouts are rare (about 3%), so we oversampled them to about a quarter.

We made up the other two answers with a small set of rules. A complaint needs quick review if its issue category is on a list of serious ones (fraud, legal threats, misuse of credit reports and so on), if the consumer is a service member or older American with a moderately serious issue, or if it mentions at least two words from a list such as “attorney,” “lawsuit,” or “fraud.” Severity adds up points for the same signals.

The complaints are split by date: the oldest only for historical figures such as each company’s payout rate, then training, then a set for adjusting the probabilities, and the most recent 2,400 for testing. Every model reads the first 1,500 characters, which shortens about a quarter of complaints; the rules read the full text.

This is quite unlike IMDb sentiment. “Positive” means what everyone thinks it means; “needs quick review” means whatever our rules say, and “closed with an explanation” depends on how these companies behave. That difference matters more than anything else in the results.

## Building Our Own Version

Our model follows the open [jaredpalmer/kev](https://github.com/jaredpalmer/kev) implementation closely.

Each complaint and its three questions go into the model as one piece of text: the complaint first, then each question with its options. The trick is controlling what each part may look at. Every question sees the whole complaint, no question sees another, and each question is positioned as if it came straight after the complaint, so all three are treated as though they were asked on their own.

This reproduces both behaviors observed in Jev. Questions cannot read each other. Options within one question are read together, in order: each option is read in the light of the ones listed before it but not after, so adding an option changes how the others are read, and an option listed early is read without seeing its rivals.

At the end, the model compares its internal summary of each question with its summary of each option and turns each comparison into a probability. Because it compares rather than picking from a fixed list of slots, the options can change from one request to the next without retraining, and an option can be a whole phrase.

[Raschka sketches the main alternative](https://magazine.sebastianraschka.com/p/classifier-history-and-jev): feed the model the input, the instructions, and one option at a time, score each, and turn the scores into probabilities. Scored alone, options cannot affect each other, so the design is immune to the order problem, at the cost of reading the complaint once per option. As shown above, this approach is not what Jev does.

We started from Qwen3 0.6B, a small open language model, and trained a small add-on rather than the whole model, which keeps training cheap enough to repeat often.

## Running It

The pipeline runs on [AI orchestration in Anaconda Platform](https://www.anaconda.com/platform/ai-orchestration), which runs on Metaflow. It is three Metaflow flows, for data, training, and evaluation, each starting when the previous one finishes and reading its results directly, so every number traces back to the data and model that produced it. Training runs on a cloud GPU even when started from a laptop. A [Panel](https://panel.holoviz.org) dashboard serves the model: try complaints, flip one fact to see which answers change, and inspect the probabilities.

**Scaling up and down is a decorator.** Training needs a GPU and nothing else does, so only the training step asks for one:

``` python
@kubernetes(gpu=1)
@step
def train(self):
    ...
```

Starting the flow from a laptop still sends that step to a cloud GPU, while data preparation and evaluation run on ordinary CPU machines. Going the other way is just as easy. For quick checks, we ran the whole pipeline locally on a small sample, with each step’s software environment supplied by the project’s lock file instead of being built in the cloud. Moving from Qwen3 0.6B to a larger base model changes the resource request on one step and nothing else.

**Comparing many models is running them side by side.** A flow can fan out into parallel branches, one per configuration, and join the results afterwards. That is how a comparison like ours grows cheaply: each [XGBoost](https://github.com/dmlc/xgboost) variant or Jev prompt is another branch, and adding one does not slow down the others.

**Every result can be pulled up later.** Anything a step stores, such as a trained adapter, a data split, a metric, or a reliability table, is saved and versioned as an artifact on S3 with the run that produced it. From a notebook, getting it back takes a few lines:

``` python
from metaflow import Flow
run = Flow("ComplaintTriageCalibrationFlow").latest_successful_run
run.data.metrics
```

The charts in this post were drawn from those stored results, and each flow also publishes a small report page that shows its numbers next to the run. When a result looks wrong, it can be traced back to the exact data and model version that produced it.

**A language model can run the experiments.** Because every experiment is code and every result can be queried, a coding agent can do much of the loop itself: change a flow, launch a run, read the stored results, compare them with the last run, and propose the next change. The next post covers this in detail.

## Results

We compared our model with XGBoost, a standard and fast machine learning method, given either the structured fields of each complaint or those fields plus word counts from its text, and with Jev, both zero-shot and with prompts that describe our rules. Besides the complaints themselves, our model trained on 2,666 “minimal pairs”: copies of complaints with one decisive fact changed, such as adding “I have already contacted an attorney about this matter.” Because “attorney” is on the rule’s word list, these pairs effectively taught our model the rules, and no comparison model saw them. All results are on 2,400 held-out complaints. The full method, every score, and the calibration analysis are in the [technical report](https://github.com/philippjfr/decision-time).

**Asked cold, Jev does worse than always guessing the most common answer on all three questions.** Nothing in the prompt says that half these complaints close with an explanation only, or where our rules draw the line. Jev says 77% deserve non-monetary relief, 74% need quick review, and 45% are high severity. This is the same model that matched a fine-tuned classifier on IMDb. Jev does well when the labels mean what most people would assume, and poorly when they depend on your own rules or on how your own organization has behaved in the past.

Shifting Jev’s answers toward this data’s base rates lifts resolution accuracy from 45% to 55%, so its reading does carry information about outcomes. On the made-up questions it carries almost none: after the same correction, Jev says no complaint needs quick review.

Writing out the rules gets Jev to 82% on quick review and 54% on severity, both below the XGBoost model that never reads the complaint. The rules are word matches and list lookups, which XGBoost learns almost perfectly and Jev has to follow from a paragraph of instructions. On resolution, Jev’s best is 55%, below every model that reads the text.

Jev’s probabilities on resolution are honest: after adjustment, they match how often it is right. Ours are not at the top end: of 237 answers given at about 95% confidence, only 47% were right. [Raschka found](https://magazine.sebastianraschka.com/p/classifier-history-and-jev) that adding a Brier penalty to training barely improved calibration for his ModernBERT classifier, so that is probably not the fix here.

Our model’s lead on the made-up questions is smaller than it looks. Applying the rules to the issue category and tags alone, without reading the complaint, already gets 94% of quick-review and 89% of severity answers right, and on severity our model scores below that. Since the minimal pairs taught our model the rules, and rules rather than people produced the answers, these results show how well each model recovers our rules. On resolution, the one real outcome, XGBoost with word counts beats our model on both accuracy and probability quality.

Our model on a laptop GPU is about as fast as Jev’s servers, and most of Jev’s time is network; under load, one Jev request in twenty took about 7 seconds. The word-count model is one to two orders of magnitude faster than either and needs no GPU.

## Making It Yours

The three questions, the labeling rules, and the minimal pairs were written for this demo. To use it on your own data, replace them with your own questions and your own definitions, ideally with labels written by people. A larger starting model (Qwen3 also comes in 1.7, 4, and 8 billion parameters) trades speed for accuracy, and the data can be your own tickets instead of Consumer Financial Protection Bureau (CFPB) complaints.

## What We Think This Changes

Until now, many teams have [reached for a general-purpose large language model even for classification](https://magazine.sebastianraschka.com/p/classifier-history-and-jev), with prompts and extra code to cover the gaps. A decision model handles classification as a separate step: it takes structured input, returns a fixed set of answers with probabilities, and is fast and cheap enough to run on every item.

Jev appears to be built from known techniques. Deploying it still takes more care than the launch suggested, because the main risks, such as sensitivity to how the options are worded and ordered, do not show up in an accuracy score.

Raschka concludes that Jev raises the bar for justifying a fine-tuned model, and for labels that mean what everyone assumes, we agree. When the labels depend on your own rules and history, it does not hold: here a small model trained on the questions beat the general one on all three, and a word-count model beat both on the real outcome while running many times faster. Jev did several things well: its probabilities could be trusted after one adjustment, its answers barely moved when options were reordered, and each decision cost a few thousandths of a cent. But a general decision model has to be told your base rates and definitions before it can match your labels, and a prompt can only supply part of that. Whether a larger model trained on your own data, or labels written by people, closes the gap is what the remaining experiments are for.

Two results in this post were easy to misread: our model’s best scores came from labels it had been taught, and a word-count model won on the only real outcome. We caught both because every number here is stored with the run, data split, and model version that produced it. For a decision model whose probabilities shift with option order, that record is how a team finds the problem before the model reaches production.

The next post will cover running machine learning experiments with a language model on AI orchestration in Anaconda Platform: how an agent launches runs, reads the stored results, compares them with earlier runs, and decides what to try next, using the experiments from this post as the worked example.

## References

- Archer Hume, [Jev’s Architecture Unmasked](https://archerhume.com/posts/jevs-architecture-unmasked) , September 17, 2026.
- Sebastian Raschka, [Language Models for Text Classification: From Bag-of-Words to Jev](https://magazine.sebastianraschka.com/p/classifier-history-and-jev) , September 29, 2026.
- TypeSafe, [Introducing System One Models and Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev) , September 15, 2026, and the[API documentation](https://docs.typesafe.ai/) .
- TypeSafe, [system-one-adapter-python](https://github.com/typesafe-ai/system-one-adapter-python) .
- Tim Fernholz, [A new kind of AI model from a ChatGPT inventor is thrilling developers](https://techcrunch.com/2026/09/18/a-new-kind-of-ai-model-from-a-chatgpt-inventor-is-thrilling-developers/) , TechCrunch, September 18, 2026.
- OpenAI, [DevDay 2026 recap](https://openai.com/index/devday-2026-recap/) , September 29, 2026.
- CFPB, [The CFPB to Cease Discretionary Publication of Complaint Narratives and Visualizations](https://www.consumerfinance.gov/about-us/newsroom/the-cfpb-to-cease-discretionary-publication-of-complaint-narratives-and-visualizations/) , 14 August 2026.
- Damani et al. (2025), *Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty* ,[arXiv:2507.16806](https://arxiv.org/abs/2507.16806) .
- Vaswani et al. (2017), [Attention Is All You Need](https://arxiv.org/abs/1706.03762) .
- Warner et al. (2024), [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://arxiv.org/abs/2412.13663) .
- [jaredpalmer/kev](https://github.com/jaredpalmer/kev) .
- Further prior-art papers are listed in the [technical report](https://github.com/philippjfr/decision-time) .