Local JEV, fine tuned in < 30 mins on Mac Air

An independent experiment fine-tuned a 135M-parameter model on the 10,003-message Banking77 dataset in under 30 minutes on a 16 GB Apple MacBook Air, lifting exact intent accuracy from 1.62% to 83.25% and operational decision accuracy from 47.99% to 92.01% on the 3,080-message test split. At a 70% confidence threshold the model auto-routes 67.05% of messages with 98.45% decision accuracy, while the uncertain 32.95% go to human review, and warm single-message inference runs at a 35.16 ms p50 and 26.26 requests per second. The author credits Duarte O.Carmo's "Jev in 25 lines of Python" post as the inspiration and states the model is not TypeSafe's Jev and does not reproduce its architecture.

Models that do not write. They decide. This tiny local model turns a banking message into a typed decision in about 35 milliseconds. No generated paragraph. No data-center GPU. I trained the whole thing in under 30 minutes on an Apple MacBook Air with 16 GB of RAM. Everyone is talking about models that take text in and return typed choices with probabilities. It sounds like a new category of AI. So I built one. Credit where it is due. This experiment was directly inspired by Duarte O.Carmo’s NobodyWho article, “Jev in 25 lines of Python.” https://www.nobodywho.ai/posts/jev-in-25-lines/ That post showed how a local open model’s next-token logits can become a closed set of probabilities. I took that idea further by fine-tuning a small model on public banking data, measuring it on a held-out test set, and adding an explicit human-review threshold. Jev itself is TypeSafe AI’s product; this is an independent experiment, not a reproduction of its architecture. To be precise, I built my own tiny, local, Jev-shaped decision model. It is not TypeSafe’s Jev and does not claim to reproduce its architecture. It takes the useful product idea seriously: stop asking a language model to compose an answer when the product needs a bounded decision. I fine-tuned a 135M-parameter model on Banking77 https://github.com/PolyAI-LDN/task-specific-datasets/tree/master/banking data : 10,003 customer-support messages spanning 77 intents. The model sees a message such as: Why was my card payment declined? It does not generate a reply. It returns a probability distribution: { "declined card payment": 0.9625, "reverted card payment": 0.0204, "declined cash withdrawal": 0.0092 } Then deliberately boring policy code turns intent into action: if confidence < 0.70: action = "human review" elif intent in security intents: action = "secure account now" elif intent in investigation intents: action = "investigate transaction" elif intent in assisted support intents: action = "assisted support" else: action = "self service" That separation matters. The model estimates what the customer needs. Ordinary, inspectable code decides what the product should do about it. Fine. But does my fake Jev actually work? We evaluated on Banking77’s 3,080-message test split. The threshold curve below is descriptive test-set analysis; a production system must choose and lock its threshold on separate validation data. | Result | Before training | After training | |---|---|---| | Exact intent accuracy | 1.62% | 83.25% | | Operational decision accuracy | 47.99% | 92.01% | | Calibration error | 27.39% | 6.78% | The pre-training intent score is close to the 1-in-77 random baseline. The high pre-training action score is less impressive than it looks: many intents share the same broad action. Probability lets the model decline to decide With a 70% acceptance threshold, the model automatically routes 67.05% of messages. On that accepted subset, exact intents are 95.88% accurate and operational decisions are 98.45% accurate. The uncertain 32.95% go to a person. | Threshold | Automated | Decision accuracy | |---|---|---| | 50% | 83.38% | 96.77% | | 70% | 67.05% | 98.45% | | 90% | 41.56% | 99.69% | More automation is available if we accept more errors. Less automation buys greater precision. The threshold makes that tradeoff explicit instead of hiding it behind a generated paragraph. Is it fast? Very. On the same 16 GB Apple MacBook Air used for training, across 1,000 warm single-message requests: p50 35.16 ms p95 47.43 ms p99 57.88 ms throughput 26.26 requests/second Cold model loading took 1.73 seconds. The warm figures include tokenization, inference, probability normalization, thresholding, and action routing. The full one-epoch fine-tune on all 10,003 training examples took less than 30 minutes on that laptop. That combination—a sub-30-minute training run and answers in tens of milliseconds—is what made the experiment feel less like a demo and more like a practical local component. Run it yourself This repository contains the exact implementation banking77.py , the trained LoRA adapter adapter-banking77/ , and the evaluation evidence evidence/ behind every number above. Clone it, install the pinned dependencies, and make a prediction: git clone https://github.com/shmcsensei/easy-jev-fine-tune.git cd easy-jev-fine-tune python3.12 -m venv .venv source .venv/bin/activate python -m pip install -r requirements.txt python banking77.py predict \ "Why was my card payment declined?" To recreate the adapter from the official training split and evaluate it once on the test split: python banking77.py train \ --examples-per-class 999999 --epochs 1 \ --output adapter-banking77-reproduced python banking77.py evaluate \ --adapter adapter-banking77-reproduced \ --threshold 0.70 The code is part of the post. The README explains the fast smoke-test path, full reproduction commands, evaluation, benchmarking, expected downloads, and sources of result variation. There. I built my own “Jev.” Well, Jev-shaped. It is not the actual Jev, and it is not a production banking system. The probabilities are better calibrated after training, but they are not guarantees. Banking77 labels support intents, not fraud outcomes. The action mapping is prototype business policy and would need operational, risk, and compliance review. The controversial bit is not that this replaces Jev. It does not. It is that a surprisingly large slice of the product experience is reproducible with a tiny open model, one classification head, some LoRA weights, and an honest threshold. The useful part is real: a small local model can turn unstructured customer language into fast probabilistic decisions, expose uncertainty, and hand the hard cases to humans.