{"slug": "fast-models-slow-evidence-a-paired-and-self-audited-evaluation-of-system-1-for", "title": "Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses", "summary": "A paired evaluation of two System-1 decision models for LLM agent harnesses found the hosted model Jev significantly more accurate than the open-weight model Laya on 9 of 11 agent decision points, by margins of +10.8 to +46.0 percentage points, according to the arXiv paper 2610.02267v1. The study tested 7,283 base cases plus 6,640 robustness variants across 18 public sources with byte-identical inputs, and reported that neither model beat chance on zero-shot model routing while Laya changed 30% of its answers when option order was reversed and dropped to 31% accuracy at 50 nearest-neighbour tools versus 98% for Jev on items with a unique correct tool. The authors' self-audit found three analysis errors and one design confound that distorted headline deployment claims, including an omitted pre-screen cost that cut a reported 23.9% saving to an actual 4.3% and gate accuracy reported as end-to-end quality (58% vs. 98%).", "body_md": "arXiv:2610.02267v1 Announce Type: new \nAbstract: Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input carries an injection. System-1 decision models answer such questions in a single forward pass with class probabilities, promising large cost and latency savings over LLM calls. We present a paired evaluation of an open-weight (Laya) and a hosted (Jev) System-1 model on 11 agent decision points built from 18 public sources: 7,283 base cases plus 6,640 robustness variants, with byte-identical inputs, paired tests, and cross-hardware and cross-day reproducibility checks. Jev is significantly more accurate on 9 of 11 decision points (+10.8 to +46.0 pp). Neither model beats chance on zero-shot model routing, and they tie on RAG relevance gating. Laya changes 30% of its answers when the option order is reversed and degrades sharply with many or similar candidates (31% at 50 nearest-neighbour tools, vs. 98% for Jev on items with a unique correct tool). We also audit our own pipeline. Three analysis errors and one design confound distorted headline deployment claims: an omitted pre-screen cost (reported 23.9% saving, actual 4.3%), gate accuracy reported as end-to-end quality (58% vs. 98%), in-sample thresholds (5% target, up to 17% held-out misses), and a \"channel effect\" on injection false positives that vanishes with channel-native content. Two other suspected confounds did not change the conclusions. All cases, raw outputs and analysis code are available at https://github.com/David-DL-Space/sys1-eval.", "url": "https://wpnews.pro/news/fast-models-slow-evidence-a-paired-and-self-audited-evaluation-of-system-1-for", "canonical_source": "https://arxiv.org/abs/2610.02267", "published_at": "2026-10-05 04:00:00+00:00", "updated_at": "2026-10-05 04:11:14.535197+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-research", "artificial-intelligence", "machine-learning"], "entities": ["Laya", "Jev", "arXiv", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/fast-models-slow-evidence-a-paired-and-self-audited-evaluation-of-system-1-for", "markdown": "https://wpnews.pro/news/fast-models-slow-evidence-a-paired-and-self-audited-evaluation-of-system-1-for.md", "text": "https://wpnews.pro/news/fast-models-slow-evidence-a-paired-and-self-audited-evaluation-of-system-1-for.txt", "jsonld": "https://wpnews.pro/news/fast-models-slow-evidence-a-paired-and-self-audited-evaluation-of-system-1-for.jsonld"}}