{"slug": "jev-attack-of-the-classifiers", "title": "Jev: Attack of the Classifiers", "summary": "A developer compared five approaches to a banking-support classification task using the BANKING77 dataset: a TF-IDF lexical model, a small sentence transformer (MiniLM), a general-purpose language model run through DSPy, an optimized version of that DSPy program, and Jev, a typed decision function the author previously placed at the branches of a state machine. The comparison maps classification shapes — binary, multiclass, multilabel, and ordinal — onto Jev's three typed question primitives: Noul for binary propositions, Choice for single-label multiclass, and Score for ordered levels, with one Noul per label for multilabel work. The author notes that Choice probabilities compete and sum to one, making it usually wrong for multilabel classification, where separate Nouls let each condition be true or false independently.", "body_md": "# Jev: Attack of the Classifiers\n\nTF-IDF, MiniLM, and DSPy draw their swords. Jev raises a shield. BANKING77 keeps score.\n\n1.   1.\n  [Jev: AI Decisions as a Typed Function Call](https://stacktoheap.com/blog/2026/09/18/jev-doesnt-write-review-comments) \n2.   2.\n  [Jev at the Branches: The State Machine Is the Agent](https://stacktoheap.com/blog/2026/09/21/the-state-machine-is-the-agent) \n3. 3. Jev: Attack of the Classifiers\n\nIn the [first Jev article](https://stacktoheap.com/blog/2026/09/18/jev-doesnt-write-review-comments), I used Jev as a typed decision function. In the [second](https://stacktoheap.com/blog/2026/09/21/the-state-machine-is-the-agent), I put that function at the branches of a state machine.\n\nBoth experiments carefully avoided a more basic question:\n\nWhat happens if I make Jev do ordinary classification and compare it with ordinary classifiers?\n\nSo I built the comparison I had been skirting around. I gave the same banking-support problem to a lexical model from the old school, a small sentence transformer, a general-purpose language model through DSPy, an optimized version of that DSPy program, and Jev.\n\nThe title says “attack,” but no classifiers were harmed. Several API credits were mildly inconvenienced.\n\n# The problem: choosing labels\n\nClassification maps an input to one or more categories from a known set. The shape of that output matters because “classification” covers several different problems.\n\n| Type | Output | Example | \n|---|---|---|\n| **Binary** | One of two outcomes | Is this email spam? | \n| **Multiclass** | Exactly one of several labels | Which team should receive this ticket? | \n| **Multilabel** | Any number of independently applicable labels | Does this message contain abuse, spam, or personal data? | \n| **Ordinal** | One ordered level | Is this incident low, medium, or high severity? | \n\nBinary classification is the simplest case. A spam filter chooses `spam` or `not_spam`. Multiclass classification expands the menu but still picks exactly one answer. A moderation system might choose `safe`, `review`, or `block`; a support router might choose billing, shipping, or technical support.\n\nMultilabel classification is different. A message can be both abusive and contain personal data. The labels do not compete for a single winning slot, so forcing them into one multiclass answer would throw information away.\n\nOrdinal classification adds order. `high` is more severe than `medium`, which is more severe than `low`. The distance between levels may matter even though the labels remain discrete.\n\n## How those shapes map to Jev\n\nJev exposes three typed question primitives. They line up with these common classification shapes, but the primitive should follow the meaning of the answer rather than the name of an ML technique.\n\n| Classification need | Jev primitive | Returned answer | \n|---|---|---|\n| Binary proposition | **Noul** | Probability of yes from 0 to 1 | \n| Single-label multiclass | **Choice** | Selected option, full probability distribution, and confidence | \n| Multilabel | **One Noul per label** | Independent yes probability for every label | \n| Ordered levels | **Score** | Probability-weighted score, level distribution, and confidence | \n\nA **Noul** asks whether one proposition holds: “Does this message contain payment-card data?” It returns `P(yes)`. There is no separate Noul confidence value; a result near 0.5 already expresses that yes and no are similarly plausible.\n\nA **Choice** asks which option best fits. Its probabilities compete and sum to one. That makes it appropriate for single-label routing, but usually wrong for multilabel classification: raising the probability of one Choice option necessarily takes probability away from the others. For multilabel work, separate Nouls let every condition be true or false independently.\n\nA **Score** uses an ordered rubric. It returns a probability distribution over the levels and a weighted position across them. It is useful for severity, quality, urgency, or any classification where the order carries meaning. It is not an invitation to ask the model for an unexplained number; each level still needs a concrete description.\n\nChoice and Score also return confidence derived from how concentrated their distributions are. That describes how clearly one answer separates from its alternatives. It does not prove the answer is correct.\n\n# The dataset: 77 ways banking can go wrong\n\n[BANKING77](https://huggingface.co/datasets/PolyAI/banking77) contains 13,083 online-banking queries across those 77 intents. It is a public dataset licensed under CC BY 4.0.\n\nEvery query belongs to exactly one intent, making this a **single-label multiclass** problem. For Jev, that maps naturally to one Choice containing all 77 options.\n\nThe function is conceptually simple:\n\n``` php\nclassify(\"Why has my cash withdrawal not arrived?\")\n  -> pending_cash_withdrawal\n```\n\nThe output space is bounded. The hard part is deciding which evidence in the text separates labels that may be very close to each other.\n\nConsider these three intents:\n\n- `card_payment_fee_charged`\n- `cash_withdrawal_charge`\n- `transfer_fee_charged`\n\nThe word “charged” is not enough. The classifier has to identify which operation the customer is talking about. Other boundaries are subtler: a transfer can be pending, declined, cancelled, or not received by its recipient.\n\nThis is exactly the kind of bounded decision I have been using Jev for. It is also a problem with decades of non-LLM solutions. That makes it a useful place to compare abstractions rather than demos.\n\nThe official training split has 10,003 examples. I reserved 20 deterministic examples from every class for development, producing:\n\n| Split | Examples | Purpose | \n|---|---|---|\n| Train | 8,463 | Fit the local classifiers | \n| Development | 1,540 | Select the local linear head and optimize DSPy | \n| Official test | 3,080 | Final evaluation only | \n\nThe local models were cheap enough to run over all 3,080 test examples. Live services were not, so I selected a deterministic, stratified sample of ten test examples from every class: 770 examples total.\n\nFor the main comparison, I also evaluated the local models on those exact 770 examples. That matters. Comparing one system on an easy random slice and another on the full test set would produce a neat table and a bad experiment.\n\nThis still is not a perfectly controlled model comparison:\n\n- TF-IDF and MiniLM see all 8,463 training examples.\n- Base DSPy and Jev are zero-shot; they see label definitions but no training examples.\n- Optimized DSPy uses one training and one development example per class.\n- Local latency excludes training and model loading; service latency includes the network.\n\nThe benchmark compares useful ways to build the feature, not models given identical supervision and infrastructure.\n\n# Contestant one: TF-IDF, the bag of words with receipts\n\nTF-IDF is the baseline people are often too eager to skip.\n\nIt represents a document using the words and phrases it contains. **Term frequency** rewards terms that appear in this document. **Inverse document frequency** reduces the weight of terms that appear in almost every document. A rare phrase such as “cash withdrawal” carries more information than “please.”\n\nMy implementation uses word unigrams and bigrams, keeps the 4,096 most frequent features, applies sublinear term frequency and IDF weighting, and normalizes the resulting sparse vector. A linear softmax classifier then learns one set of weights for each of the 77 intents.\n\n``` php\nmessage\n  -> word and bigram counts\n  -> TF-IDF sparse vector\n  -> linear softmax head\n  -> one of 77 intents\n```\n\nThis model has no semantic understanding in the transformer sense. It does not know that “cash machine” and “ATM” are related unless the training data gives the linear head enough lexical evidence. But support datasets contain repeated domain language, and linear classifiers are very good at exploiting it.\n\nOn the 770-example comparison set, TF-IDF reached **82.73% accuracy** and **81.68% macro-F1**. Mean inference time was **0.044 ms per example** on my machine.\n\nThat is the first useful result: the supposedly boring baseline is not a ceremonial participant. It is fast, local, inspectable, and difficult to beat casually.\n\n# Contestant two: MiniLM learns what the sentence means\n\nTF-IDF starts from lexical overlap. MiniLM starts from a learned sentence representation.\n\nI used `all-MiniLM-L6-v2`, a compact six-layer sentence transformer that maps each query to a 384-dimensional embedding. Queries with similar meanings can land near each other even when they do not share the same words.\n\nWhy MiniLM rather than a larger embedding model?\n\nI wanted a realistic local classifier, not a second hosted-model benchmark. MiniLM is small enough to run quantized on CPU, widely used, and strong enough to test whether semantic features improve the same simple classifier. I froze the encoder and trained the **same kind of linear softmax head** used by TF-IDF. The comparison therefore changes the representation while keeping the final classifier deliberately plain.\n\n``` php\nmessage\n  -> frozen MiniLM encoder\n  -> 384-dimensional dense vector\n  -> linear softmax head\n  -> one of 77 intents\n```\n\nMiniLM won the benchmark: **89.87% accuracy** and **89.84% macro-F1** on the shared 770 examples, at **1.28 ms per example**.\n\nThat seven-point gain over TF-IDF is the value of semantic representation here. “Where is the nearest cash machine?” can resemble other ATM questions even if the exact phrasing was absent from training.\n\nIt is slower than TF-IDF by a large ratio and still extremely fast in absolute terms. It also remains fully local after downloading the model.\n\n# Contestant three: DSPy turns an LLM call into a program\n\nThe third approach uses [DSPy](https://dspy.ai/) with GLM-5.3 Flash through OpenCode Go.\n\nDSPy is not itself a classifier model. It provides a way to define structured language-model programs and optimize them. I defined a signature with three inputs:\n\n- the customer message;\n- the classification instruction;\n- a typed dictionary containing all 77 intent names and definitions.\n\nThe output is a `Literal` over the exact 77 labels, so the allowed values are part of the DSPy signature rather than a convention hidden in prompt text.\n\nIn abbreviated form, the signature looks like this:\n\n```\nclass IntentClassificationSignature(dspy.Signature):\n    message: str = dspy.InputField()\n    task_instructions: str = dspy.InputField()\n    intent_options: dict[str, str] = dspy.InputField()\n    intent: Literal[\"card_arrival\", ..., \"country_support\"] = dspy.OutputField()\n```\n\nThe base DSPy program was zero-shot. For every test query, GLM received the instructions and all 77 definitions and returned one typed label.\n\nIt scored **82.86% accuracy** and **82.01% macro-F1**. The 770 sequential requests took **23 minutes 44 seconds**, or **1.85 seconds each** on average.\n\nI later ran the same uncompiled typed program with MiMo-V2.6-Flash over the complete test set. On the shared 770-example sample it scored **80.39% accuracy** and **79.32% macro-F1**. Across all 3,080 examples it reached **80.91%** and **80.11%**, averaging **4.08 seconds per request**. It produced no contract failures. A framework name is not a result: the model behind the DSPy program still matters.\n\n# Optimizing the DSPy program\n\nI also ran MIPROv2 with exact intent match. Its light optimization loop received:\n\n- one deterministic training example per class: 77 total;\n- one separate development example per class: 77 total;\n- no test examples;\n- GLM-5.3 Flash as both the prompt and task model.\n\nMIPROv2 proposed instructions and demonstrations, evaluated candidates on the development set, and saved the best program.\n\nOn the test set, that program scored **82.73% accuracy** and **81.87% macro-F1**—slightly below the zero-shot signature. Optimization is an experiment, not an automatic upgrade. This small development set did not produce a prompt that generalized better.\n\n# Contestant four: Jev makes the label space the interface\n\nJev received the same customer message and the same 77 intent definitions as a typed `Choice` question:\n\n```\nstate:     \"customer message\"\nquestion:  Which BANKING77 intent best describes the request?\ncriteria:  77 allowed intent keys and their definitions\nresult:    selected key + probability distribution\n```\n\nThere was no model training, prompt optimization, or demonstration selection. A 77-way Choice is within the documented 255-option limit, and the TypeSafe guidance recommends supplying the full option set rather than creating an arbitrary shortlist. Because BANKING77 guarantees that every test query has one of these labels, I did not add a `none_of_the_above` option.\n\nJev returned an allowed choice, its probability distribution, and confidence. Confidence describes how concentrated that distribution is; it is not a correctness guarantee. I pinned `jev-1.13.0` rather than using the moving `jev-latest` alias.\n\nOn the 770 examples, Jev reached **81.17% accuracy** and **80.37% macro-F1**. The sequential run took **3 minutes 50 seconds**, or **298 ms per example**.\n\nThat result raised an obvious question: was Jev missing language understanding, or was it missing the dataset’s particular boundaries? TypeSafe’s guidance recommends structured Choice criteria for easily confused options, with fields such as `what`, `not_for`, and `examples`. I ran a second arm that changed each option from a short definition to this shape:\n\n```\n{\n  \"what\": \"The customer is asking about cash withdrawal charge.\",\n  \"examples\": [\n    \"I saw I was charged extra for money I withdrew?\",\n    \"I hate this ATM, it charged me an extra fee, Why did it charge?\"\n  ]\n}\n```\n\nThe examples were selected deterministically from the training split: two per intent, 154 in total. No development or test examples were used, and I did not hand-tune exclusions from test-set errors.\n\nWith those examples, Jev reached **85.32% accuracy** and **84.99% macro-F1**. Mean latency rose to **520 ms**, and input usage rose from 1.64 million to 4.36 million tokens. The examples supplied useful domain supervision, but they also made every request larger.\n\nThis four-point gain suggests that much of the zero-shot gap came from BANKING77’s annotation boundaries. General language understanding was not enough to recover all of the dataset’s conventions from terse label definitions.\n\nTypeSafe can evaluate many questions over one shared state in parallel. That does not provide a bulk endpoint for 770 unrelated customer messages: combining them would change the state visible to each question. I therefore kept one message per request.\n\nThe definition-only Jev arm finished behind TF-IDF and base DSPy but was about six times faster than DSPy. The example-guided arm moved ahead of both while remaining below MiniLM. These are end-to-end measurements of different hosted services, not intrinsic hardware benchmarks.\n\n# The scoreboard\n\nHere is the direct comparison on the same ten official test examples from each of the 77 classes:\n\n| Approach | Supervision used | Accuracy | Macro-F1 | Mean inference | \n|---|---|---|---|---|\n| MiniLM + linear | 8,463 train + dev selection | **89.87%** | **89.84%** | 1.28 ms | \n| Jev 1.13 Choice + examples | 154 train | 85.32% | 84.99% | 520.03 ms | \n| DSPy + GLM, base | Zero-shot | 82.86% | 82.01% | 1,849.49 ms | \n| TF-IDF + linear | 8,463 train + dev selection | 82.73% | 81.68% | **0.044 ms** | \n| DSPy + GLM, MIPROv2 | 77 train + 77 dev | 82.73% | 81.87% | 1,706.81 ms | \n| Jev 1.13 Choice | Zero-shot | 81.17% | 80.37% | 298.39 ms | \n| DSPy + MiMo-V2.6-Flash | Zero-shot | 80.39% | 79.32% | 3,873.78 ms | \n\nFor a larger check, the local models, both Jev arms, and DSPy with MiMo also ran over all 3,080 official test examples:\n\n| Approach | Accuracy | Macro-F1 | Mean inference | \n|---|---|---|---|\n| MiniLM + linear | **89.12%** | **89.09%** | 1.14 ms | \n| Jev 1.13 Choice + examples | 85.29% | 85.20% | 376.90 ms | \n| TF-IDF + linear | 82.21% | 81.17% | **0.024 ms** | \n| DSPy + MiMo-V2.6-Flash | 80.91% | 80.11% | 4,076.37 ms | \n| Jev 1.13 Choice | 80.10% | 79.37% | 322.86 ms | \n\nThe full results preserve the same ordering and nearly the same gaps. Most importantly, Jev’s gain from two examples per label holds across the complete test set: **5.19 percentage points of accuracy** and **5.83 points of macro-F1** over definition-only Jev.\n\n## Are Jev’s probabilities calibrated?\n\nAccuracy asks how often a classifier is right. **Calibration** asks whether its probabilities mean what they say. Among predictions made at 80%, roughly 80% should be correct. A model can rank labels well and still be badly calibrated—for example, by assigning 99% to many wrong answers.\n\nThis matters for Jev because probabilities are part of the product, not an implementation detail. TypeSafe says [System One models are trained for calibrated decisions](https://docs.typesafe.ai/concepts/system-one) and that their probabilities are optimized to reflect uncertainty. It also makes the right qualification: calibration is a property of groups of predictions, not a promise that any individual answer is correct. The docs recommend validating thresholds on your own data.\n\nJev’s Choice response provides two related values. `probabilities` is the complete distribution over options. `confidence` is derived from how concentrated that distribution is: one clear winner produces higher confidence than several plausible options. Concentration is not calibration. To test calibration, I used the probability assigned to the selected label and compared it with how often that label was actually correct.\n\nOn the complete 3,080-example test set, I grouped predictions into ten equal-width probability bins and calculated expected calibration error, or ECE. ECE is the weighted average gap between predicted probability and observed accuracy in those bins; lower is better, and zero would be perfect.\n\n| Jev arm | Accuracy | Mean top probability | ECE | \n|---|---|---|---|\n| Definitions only | 80.10% | 90.05% | 9.95% | \n| Two examples per label | 85.29% | 91.72% | 6.44% | \n\nThe definition-only arm was about ten percentage points too sure overall: its selected labels averaged 90.05% probability but were correct 80.10% of the time. Examples improved both classification and calibration, reducing ECE from 9.95% to 6.44%, but the model remained overconfident on this task.\n\nI did not fit temperature scaling or tune thresholds on the test set; that would leak the answers into the reported result. In production, I would fit any calibration layer and action thresholds on separate development data, then verify them on held-out traffic. Jev exposes the probabilities needed to do that, but the probabilities still have to earn trust in the domain where they will be used.\n\n## What this benchmark cannot test\n\nBANKING77 has a fixed taxonomy and a fixed annotation policy. That makes it useful for comparing classifiers, but it cannot measure one of Jev’s more interesting properties: changing instructions or decision criteria at runtime.\n\nA trained local classifier learns the old boundary and normally needs new labels and retraining when the policy changes. Jev can receive a revised criterion in the next request. Whether it follows that revision correctly is an empirical question, not a free point on this scoreboard. Testing it would require a separate benchmark with explicit policy changes and held-out examples for both the old and new rules.\n\n# A good demo is not a deployment strategy\n\nClassification has decades of research, engineering, and operational experience behind it. Linear models, embeddings, fine-tuned encoders, calibration methods, abstention policies, and monitoring practices have all been optimized for this job. A conventional classification problem is not an empty field waiting for a general-purpose model to discover it.\n\nThat context matters because AI demos can become a little **exa-Jev-rated**. Turning a few descriptions into a working classifier with one API call is genuinely useful—and makes for a compelling demonstration. But the short setup does not erase the accuracy, latency, cost, privacy, calibration, and maintenance trade-offs that appear in production. A striking demo is evidence that something is possible, not that it is the best default.\n\nThe established methods earned their place here. With thousands of representative labels, **MiniLM was the obvious winner in this benchmark**: it had the best accuracy, remained fast, and needed no per-request service call. TF-IDF was almost free to run and stayed competitive with the hosted zero-shot systems. Any new approach should have to beat those baselines on the properties that matter, not merely look more modern.\n\nDSPy’s typed zero-shot program showed the other side of the trade-off: useful performance without training examples, but much higher latency. MIPROv2 did not improve GLM on this small optimization split, and swapping GLM for MiMo made the same program less accurate and slower here. Neither an optimizer nor a model change is an automatic upgrade.\n\nJev belongs in the same sober comparison. It did not beat the trained semantic classifier, and I should not imply otherwise. Its advantage is not that decades of classifier work suddenly became obsolete. Its advantage is that it can turn a runtime-defined decision into a bounded, typed probability distribution without first building a task-specific training pipeline.\n\nThe example-guided result sharpens that point. Jev benefited greatly from just two examples per option, but those examples are still labeled supervision. The API made that supervision easy to express; it did not make the need for domain knowledge disappear.\n\nThat makes Jev a good fit when:\n\n- the choices or decision criteria change at runtime;\n- representative labeled data is scarce or does not exist yet;\n- the application needs probabilities over an explicit set of allowed decisions;\n- the judgment is one component inside a larger policy or state machine;\n- hosted-service latency and cost are acceptable.\n\nJev is probably the wrong default when:\n\n- the taxonomy is stable and there are enough representative labels;\n- a small local model already meets the accuracy requirement;\n- requests are high-volume, latency-sensitive, offline, private, or cost-sensitive;\n- the task needs independently validated calibration or domain-specific guarantees;\n- “we can do it in one impressive API call” is the main argument for using it.\n\nThe supervision difference is the lesson hiding behind the leaderboard. MiniLM won a conventional classification task after receiving thousands of conventional labels. Jev becomes interesting at a different boundary: before that dataset exists, while the decision space is changing, or when its typed probabilities are more useful than a trained model artifact.\n\n# Conclusion\n\nThe benchmark code, deterministic importer, split metadata, classifiers, DSPy optimization loop, and reports are in [`classifier-bench`](https://github.com/manojlds/classifier-bench). BANKING77 attribution and source hashes are included in the repository.\n\nMy conclusion is less cinematic than the title:\n\nDo not let an impressive demo overrule decades of classifier practice. Start with the simplest credible baseline, use trained classifiers when the data and task justify them, and use Jev when you specifically need a runtime-defined, bounded probabilistic decision—not because the hype says every decision is now an AI call.", "url": "https://wpnews.pro/news/jev-attack-of-the-classifiers", "canonical_source": "https://stacktoheap.com/blog/2026/09/22/jev-attack-of-the-classifiers/", "published_at": "2026-09-22 00:00:00+00:00", "updated_at": "2026-09-23 10:30:00.831227+00:00", "lang": "en", "topics": ["natural-language-processing", "machine-learning", "large-language-models", "ai-tools"], "entities": ["Jev", "BANKING77", "TF-IDF", "MiniLM", "DSPy", "Noul", "Choice", "Score"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/jev-attack-of-the-classifiers", "markdown": "https://wpnews.pro/news/jev-attack-of-the-classifiers.md", "text": "https://wpnews.pro/news/jev-attack-of-the-classifiers.txt", "jsonld": "https://wpnews.pro/news/jev-attack-of-the-classifiers.jsonld"}}