We Tested a 35B LLM Against Typed-Decision Models on 12,000 Real RFQs—Confidence Changed the Winner

A developer benchmarked a typed-decision API (Jev), a 35B mixture-of-experts LLM (Qwen3.5-35B-A3B), and a 421M open-weight model (Laya) on 12,000 real U.S. federal IT solicitations, finding Jev led primary-class accuracy at 91.9% versus Qwen's 89.6% and Laya's 78.0%. The decisive factor for automation was calibration: Jev's expected calibration error of 0.049 allowed 86.5% of gold rows to be auto-accepted at a 95% precision target, while neither Qwen's three confidence levels nor Laya's scores reached that target at any useful cutoff.

91.9%. 89.6%. 78.0%. Those were the primary-class accuracies of a typed-decision API, a 35B mixture-of-experts LLM, and a 421M open-weight decision model on the same real classification job. But accuracy was not the result that changed the deployment decision. The decisive question was: Can the model tell us when its answer is safe enough to automate—and when a human needs to look? To find out, I compared Jev , Qwen3.5-35B-A3B , and Laya 421M on 12,000 U.S. federal IT solicitations. The models were graded against behavioral evidence: what a reseller's sales team actually quoted. The complete pipeline, aggregate metrics, figures, experiment design, and runbook are open in jev-laya-classification-bench https://github.com/bhushankinge/jev-laya-classification-bench . Privacy note: The public repository contains aggregate results only. Solicitation samples, quote identifiers, gold-label files, and per-row predictions remain private. A federal IT reseller receives solicitations from SEWP, GSA MAS, and GSA 2GIT. Every opportunity must be classified before it can move through the CRM: These decisions are operational. Hardware may trigger distributor price lookups. A configured build may need an engineer and an OEM portal. Software can require publisher authorization. Services go to a statement-of-work workflow. The goal was not to win a generic benchmark. It was to determine how much of this queue could be automated safely. All three paths produced one shared label schema: | Path | Model | Execution | |---|---|---| | Jev | TypeSafe System One, jev-1.13.0 | Hosted API with calibrated probabilities | | Qwen | Qwen3.5-35B-A3B-FP8 | On-prem vLLM with structured JSON | | Laya | Convai Laya, 421M parameters | Open weights on an RTX 2000 Ada laptop GPU | Jev and Laya received byte-identical typed-question bundles. Qwen received a strict JSON schema, and its answers were mapped onto the same taxonomy. All three labeled the 12,000-opportunity sample. Qwen produced 69 permanently malformed responses, leaving 11,931 rows in the all-source paired set. When a sales rep quoted an opportunity, the product types on the latest quote lines provided the composition label. That yielded 741 paired opportunities with a single unambiguous primary class. For fulfillment mode, configurator fingerprints and distributor information on quote lines produced 634 paired labels. This gold is useful because it reflects real downstream behavior, but it is not perfect truth. Quoted opportunities are a selected subset, Hardware represents 77% of the single-class rows, and the fulfillment labels include an unvalidated heuristic. Those limitations matter throughout the interpretation. | Source | Primary accuracy | Calibration error | Auto-accepted at a bounded 95% precision target | Fulfillment accuracy | |---|---|---|---|---| | Jev | 91.9% | 0.049 | 86.5% | 65.0% | | Qwen3.5-35B-A3B | 89.6% | n/a: three confidence levels | not reached | 71.0% | | Laya 421M | 78.0% | 0.322 | not reached | 45.7% | Jev beat the much larger LLM by 2.3 percentage points on the primary class. Laya's shipped defaults trailed by 13.9 points. Yet the biggest difference was not those two points between Jev and Qwen. It was calibration. Jev's expected calibration error was 0.049. At a confidence cutoff of 0.94, it accepted 641 of 741 gold rows— 86.5% coverage with 96.7% observed precision . The Wilson 95% lower bound remained at or above the 95% target along the cutoff envelope. Qwen reported three prompt-defined confidence levels. Its “high” bucket covered 97.8% of rows, but precision was 90.1%, so no bucket reached the 95% target. Laya produced 705 distinct confidence values on 741 rows, but those scores were not ranked well enough to reach the target at any useful cutoff. That distinction is operationally enormous. A classifier that is slightly more accurate but badly calibrated still leaves you without a defensible automation boundary. I tested nine Jev variants across: All nine landed between 91.2% and 91.9% on the same 741 rows—676 to 681 correct. The winner, A-S2, was chosen using a rule written before the experiments: accuracy first, then coverage at the bounded precision cutoff, then the cheaper input. The practical lesson was unexpected: elaborate question structure and a small attachment excerpt did not measurably improve this top-level task. The question bundle itself consumed most of the tokens, and the tested attachment excerpt existed on only 5.6% of rows. That is not proof that attachments never help. It is evidence that this particular excerpt strategy did not. The original design sent every row with any raised flag to human review. One flag—brand-name-only—fired on more than half the dataset. In the gate simulation: | Policy | Auto-classified | Primary precision on scored accepted rows | |---|---|---| | Any flag forces review | 26.6% | 93.5% | | Record flags as attributes | 91.9% | 93.8% | The stricter rule removed almost two thirds of the volume from automation and bought no measurable precision. When flags became attributes instead of automatic blockers, the remaining review queue was dominated by genuine Jev–Qwen disagreements. This was one of the most useful results in the study: workflow policy can matter more than another point of model accuracy. Qwen led fulfillment-mode accuracy at 71.0%, followed by Jev at 65.0% and Laya at 45.7%. That still fell far short of the operational target. Configured-build precision ranged from 19% to 36%, and the mixed category was effectively unsolved. The likely reason is structural: the distinction between distributor SKUs and an OEM-configured build often lives inside a bill of materials, configurator quote, or spreadsheet attachment—not in the notice text. There is a second uncertainty: the fulfillment gold itself uses an unvalidated “eight or more lines from one OEM” heuristic. Human validation is the next required experiment. Laya's 78.0% result describes the shipped defaults on this task , not the model's ceiling. It also did several things well: The failure analysis produced concrete hypotheses rather than a vague “small model loses” conclusion. Laya's sequence packer reserves a 192-token question head. The primary-class question had seven options with definitions written to separate hardware, software, support, and services. With seven options, each definition was cut to 25 tokens , and the instruction could shrink to as few as eight tokens. The text designed to prevent Hardware-versus-Software confusion was exactly what got truncated. Laya called 105 of 572 Hardware rows “Software.” Jev made that error seven times; Qwen made it 20 times. Laya reached 81.5% accuracy on states below 600 characters but only 65.8% from 600 to 1,200 characters—roughly 150 to 300 tokens and still inside its 512-token window. Jev and Qwen did not show the same decline. That means simple end-of-window truncation cannot explain the whole effect. Laya emitted an average of 4.05 components per row, compared with 1.95 for Jev and 1.16 for Qwen. Its RFI and text-insufficient flags fired on 63% and 71% of rows, while the discovery analysis found true RFI rates around 2–3%. The public repository lists six follow-up experiments: increase the head budget, warn on truncation, try the typed-decision and multilingual checkpoints, fit per-question thresholds, split the primary question from the rest, and record actual token counts. These are testable engineering questions—and contributions are welcome. For all 12,000 opportunities: | Source | Cost or compute | Observed operation | |---|---|---| | Jev | $0.78 at list input-token price | p50 185 ms, p95 273 ms, zero errors | | Qwen3.5-35B | About 80 minutes of one shared GPU | 2.7 rows/s, 69 malformed responses | | Laya 421M | Local RTX 2000 Ada laptop | p50 299 ms, p95 576 ms, zero errors | These are different deployment products, so the table is not a simple price race. Jev includes hosted operations. Qwen used an existing on-prem cluster. Laya ran unbatched at one row at a time; its separate CUDA study shows what a batched server can do. The right architecture depends on volume, infrastructure, latency, privacy, and whether calibrated probabilities are part of the product requirement. Jev and Qwen agreed on 91.0% of 11,931 paired rows . The 1,068 disagreements clustered around Hardware versus Other, Hardware versus Software, Software versus Maintenance & Support, and Services versus Software or Other. Instead of reviewing random rows, those disagreements form a much more valuable blind-adjudication pool. The next planned step is human review of a stratified sample and 300 blinded Jev–Qwen disagreements. The results are useful because the boundaries are explicit—not because the benchmark pretends to be universal. Everything publishable is open: The repository includes 52 passing tests, deterministic figure generation, a citation file, and the pre-registered selection rule. If you build production classifiers, I would be especially interested in your answer to this: Would you optimize for the highest raw accuracy—or for the largest slice of traffic that can be automated at a defensible precision bound?