{"slug": "we-tested-a-35b-llm-against-typed-decision-models-on-12000-real-rfqs-confidence", "title": "We Tested a 35B LLM Against Typed-Decision Models on 12,000 Real RFQs—Confidence Changed the Winner", "summary": "A developer benchmarked a typed-decision API (Jev), a 35B mixture-of-experts LLM (Qwen3.5-35B-A3B), and a 421M open-weight model (Laya) on 12,000 real U.S. federal IT solicitations, finding Jev led primary-class accuracy at 91.9% versus Qwen's 89.6% and Laya's 78.0%. The decisive factor for automation was calibration: Jev's expected calibration error of 0.049 allowed 86.5% of gold rows to be auto-accepted at a 95% precision target, while neither Qwen's three confidence levels nor Laya's scores reached that target at any useful cutoff.", "body_md": "**91.9%. 89.6%. 78.0%.**\n\nThose were the primary-class accuracies of a typed-decision API, a 35B mixture-of-experts LLM, and a 421M open-weight decision model on the same real classification job.\n\nBut accuracy was not the result that changed the deployment decision.\n\nThe decisive question was:\n\nCan the model tell us when its answer is safe enough to automate—and when a human needs to look?\n\nTo find out, I compared **Jev**, **Qwen3.5-35B-A3B**, and **Laya 421M** on 12,000 U.S. federal IT solicitations. The models were graded against behavioral evidence: what a reseller's sales team actually quoted.\n\nThe complete pipeline, aggregate metrics, figures, experiment design, and runbook are open in [jev-laya-classification-bench](https://github.com/bhushankinge/jev-laya-classification-bench).\n\n**Privacy note:** The public repository contains aggregate results only. Solicitation samples, quote identifiers, gold-label files, and per-row predictions remain private.\n\nA federal IT reseller receives solicitations from SEWP, GSA MAS, and GSA 2GIT. Every opportunity must be classified before it can move through the CRM:\n\nThese decisions are operational. Hardware may trigger distributor price lookups. A configured build may need an engineer and an OEM portal. Software can require publisher authorization. Services go to a statement-of-work workflow.\n\nThe goal was not to win a generic benchmark. It was to determine how much of this queue could be automated safely.\n\nAll three paths produced one shared label schema:\n\n| Path | Model | Execution | \n|---|---|---|\n| Jev | TypeSafe System One, jev-1.13.0 | Hosted API with calibrated probabilities | \n| Qwen | Qwen3.5-35B-A3B-FP8 | On-prem vLLM with structured JSON | \n| Laya | Convai Laya, 421M parameters | Open weights on an RTX 2000 Ada laptop GPU | \n\nJev and Laya received byte-identical typed-question bundles. Qwen received a strict JSON schema, and its answers were mapped onto the same taxonomy.\n\nAll three labeled the 12,000-opportunity sample. Qwen produced 69 permanently malformed responses, leaving 11,931 rows in the all-source paired set.\n\nWhen a sales rep quoted an opportunity, the product types on the latest quote lines provided the composition label. That yielded 741 paired opportunities with a single unambiguous primary class.\n\nFor fulfillment mode, configurator fingerprints and distributor information on quote lines produced 634 paired labels.\n\nThis gold is useful because it reflects real downstream behavior, but it is not perfect truth. Quoted opportunities are a selected subset, Hardware represents 77% of the single-class rows, and the fulfillment labels include an unvalidated heuristic. Those limitations matter throughout the interpretation.\n\n| Source | Primary accuracy | Calibration error | Auto-accepted at a bounded 95% precision target | Fulfillment accuracy | \n|---|---|---|---|---|\n| Jev | **91.9%** | **0.049** | **86.5%** | 65.0% | \n| Qwen3.5-35B-A3B | 89.6% | n/a: three confidence levels | not reached | **71.0%** | \n| Laya 421M | 78.0% | 0.322 | not reached | 45.7% | \n\nJev beat the much larger LLM by 2.3 percentage points on the primary class. Laya's shipped defaults trailed by 13.9 points.\n\nYet the biggest difference was not those two points between Jev and Qwen. It was calibration.\n\nJev's expected calibration error was 0.049. At a confidence cutoff of 0.94, it accepted 641 of 741 gold rows—**86.5% coverage with 96.7% observed precision**. The Wilson 95% lower bound remained at or above the 95% target along the cutoff envelope.\n\nQwen reported three prompt-defined confidence levels. Its “high” bucket covered 97.8% of rows, but precision was 90.1%, so no bucket reached the 95% target.\n\nLaya produced 705 distinct confidence values on 741 rows, but those scores were not ranked well enough to reach the target at any useful cutoff.\n\nThat distinction is operationally enormous. A classifier that is slightly more accurate but badly calibrated still leaves you without a defensible automation boundary.\n\nI tested nine Jev variants across:\n\nAll nine landed between **91.2% and 91.9%** on the same 741 rows—676 to 681 correct.\n\nThe winner, A-S2, was chosen using a rule written before the experiments: accuracy first, then coverage at the bounded precision cutoff, then the cheaper input.\n\nThe practical lesson was unexpected: elaborate question structure and a small attachment excerpt did not measurably improve this top-level task. The question bundle itself consumed most of the tokens, and the tested attachment excerpt existed on only 5.6% of rows.\n\nThat is not proof that attachments never help. It is evidence that this particular excerpt strategy did not.\n\nThe original design sent every row with any raised flag to human review. One flag—brand-name-only—fired on more than half the dataset.\n\nIn the gate simulation:\n\n| Policy | Auto-classified | Primary precision on scored accepted rows | \n|---|---|---|\n| Any flag forces review | 26.6% | 93.5% | \n| Record flags as attributes | **91.9%** | **93.8%** | \n\nThe stricter rule removed almost two thirds of the volume from automation and bought no measurable precision. When flags became attributes instead of automatic blockers, the remaining review queue was dominated by genuine Jev–Qwen disagreements.\n\nThis was one of the most useful results in the study: **workflow policy can matter more than another point of model accuracy.**\n\nQwen led fulfillment-mode accuracy at 71.0%, followed by Jev at 65.0% and Laya at 45.7%.\n\nThat still fell far short of the operational target. Configured-build precision ranged from 19% to 36%, and the mixed category was effectively unsolved.\n\nThe likely reason is structural: the distinction between distributor SKUs and an OEM-configured build often lives inside a bill of materials, configurator quote, or spreadsheet attachment—not in the notice text.\n\nThere is a second uncertainty: the fulfillment gold itself uses an unvalidated “eight or more lines from one OEM” heuristic. Human validation is the next required experiment.\n\nLaya's 78.0% result describes **the shipped defaults on this task**, not the model's ceiling.\n\nIt also did several things well:\n\nThe failure analysis produced concrete hypotheses rather than a vague “small model loses” conclusion.\n\nLaya's sequence packer reserves a 192-token question head. The primary-class question had seven options with definitions written to separate hardware, software, support, and services.\n\nWith seven options, each definition was cut to **25 tokens**, and the instruction could shrink to as few as eight tokens. The text designed to prevent Hardware-versus-Software confusion was exactly what got truncated.\n\nLaya called 105 of 572 Hardware rows “Software.” Jev made that error seven times; Qwen made it 20 times.\n\nLaya reached 81.5% accuracy on states below 600 characters but only 65.8% from 600 to 1,200 characters—roughly 150 to 300 tokens and still inside its 512-token window.\n\nJev and Qwen did not show the same decline. That means simple end-of-window truncation cannot explain the whole effect.\n\nLaya emitted an average of 4.05 components per row, compared with 1.95 for Jev and 1.16 for Qwen. Its RFI and text-insufficient flags fired on 63% and 71% of rows, while the discovery analysis found true RFI rates around 2–3%.\n\nThe public repository lists six follow-up experiments: increase the head budget, warn on truncation, try the typed-decision and multilingual checkpoints, fit per-question thresholds, split the primary question from the rest, and record actual token counts.\n\nThese are testable engineering questions—and contributions are welcome.\n\nFor all 12,000 opportunities:\n\n| Source | Cost or compute | Observed operation | \n|---|---|---|\n| Jev | **$0.78 at list input-token price** | p50 185 ms, p95 273 ms, zero errors | \n| Qwen3.5-35B | About 80 minutes of one shared GPU | 2.7 rows/s, 69 malformed responses | \n| Laya 421M | Local RTX 2000 Ada laptop | p50 299 ms, p95 576 ms, zero errors | \n\nThese are different deployment products, so the table is not a simple price race. Jev includes hosted operations. Qwen used an existing on-prem cluster. Laya ran unbatched at one row at a time; its separate CUDA study shows what a batched server can do.\n\nThe right architecture depends on volume, infrastructure, latency, privacy, and whether calibrated probabilities are part of the product requirement.\n\nJev and Qwen agreed on **91.0% of 11,931 paired rows**.\n\nThe 1,068 disagreements clustered around Hardware versus Other, Hardware versus Software, Software versus Maintenance & Support, and Services versus Software or Other.\n\nInstead of reviewing random rows, those disagreements form a much more valuable blind-adjudication pool. The next planned step is human review of a stratified sample and 300 blinded Jev–Qwen disagreements.\n\nThe results are useful because the boundaries are explicit—not because the benchmark pretends to be universal.\n\nEverything publishable is open:\n\nThe repository includes 52 passing tests, deterministic figure generation, a citation file, and the pre-registered selection rule.\n\nIf you build production classifiers, I would be especially interested in your answer to this:\n\n**Would you optimize for the highest raw accuracy—or for the largest slice of traffic that can be automated at a defensible precision bound?**", "url": "https://wpnews.pro/news/we-tested-a-35b-llm-against-typed-decision-models-on-12000-real-rfqs-confidence", "canonical_source": "https://dev.to/cookies_c9dc8b91f33d29250/we-tested-a-35b-llm-against-typed-decision-models-on-12000-real-rfqs-confidence-changed-the-winner-56hh", "published_at": "2026-09-24 19:39:35+00:00", "updated_at": "2026-09-24 19:58:59.141434+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-tools", "ai-research"], "entities": ["Jev", "Qwen3.5-35B-A3B", "Laya 421M", "Convai", "vLLM", "GSA MAS", "SEWP", "GSA 2GIT"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/we-tested-a-35b-llm-against-typed-decision-models-on-12000-real-rfqs-confidence", "markdown": "https://wpnews.pro/news/we-tested-a-35b-llm-against-typed-decision-models-on-12000-real-rfqs-confidence.md", "text": "https://wpnews.pro/news/we-tested-a-35b-llm-against-typed-decision-models-on-12000-real-rfqs-confidence.txt", "jsonld": "https://wpnews.pro/news/we-tested-a-35b-llm-against-typed-decision-models-on-12000-real-rfqs-confidence.jsonld"}}