{"slug": "measure-before-you-train-typed-decision-models-vs-a-classical-baseline-vs-a-llm", "title": "Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox", "summary": "A measurement protocol applied to Foyer, a self-hosted personal inbox agent in daily use, found that its production keyword rules scored 28% on message type against a 58% majority class, according to an arXiv paper (2610.32955v1) evaluated on 739 real Italian/English emails labeled by the owner. Under thread-grouped cross-validation, a TF-IDF + logistic-regression baseline trained on roughly 200 labels had the highest accuracy on every field (type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90), though it never predicted the two rarest classes and was statistically tied with Gemma 4 on type. The authors report that the most accurate system, composed into actions, ignored emails needing action 18% of the time versus 4% for the best LLM, and that a cost-chosen threshold on its noise probability halved that rate while making it the cheapest system.", "body_md": "arXiv:2610.32955v1 Announce Type: new \nAbstract: Personal email agents decide, for every message, what it is, whether it is urgent, whether to draft a reply, create a task, or ignore it. Their error rate is rarely measured: the stored labels were written by the system itself. We present Foyer, a self-hosted personal inbox agent in daily use, and a measurement protocol (stratified sampling, frozen snapshots, thread-grouped splits, pre-registered hypotheses, human labels in the agent's review loop) that yields ground truth before any training. On 739 real Italian/English emails labelled by the owner (231 unbiased, 500 adversarially sampled), we compare nine systems under thread-grouped cross-validation: the production keyword rules, a TF-IDF + logistic-regression baseline, local typed decision models (Tev1, Nimble), a hosted decision model (Nimble v4-max), and hosted generative LLMs (Qwen2.5-7B, Gemma 4 26B/31B). Four results. The production rules score 28% on message type against a 58% majority class: the field shown most prominently carried no information. With ~200 labels, logistic regression has the highest accuracy on every field (type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90), though it never predicts the two rarest classes and is statistically tied with Gemma 4 on type; few-shot prompting does not close the gap. The sender's relationship (client, supplier, service) is recovered from other emails of the same contact with 98.6% accuracy where the contact has been seen, and is not inferable from a single message by any model: it belongs in the contact graph, not the prompt. Finally, accuracy is not the product metric: composed into actions, the most accurate system most often ignores emails needing action (18% vs 4% for the best LLM); a cost-chosen threshold on its noise probability halves that rate and makes it the cheapest system.", "url": "https://wpnews.pro/news/measure-before-you-train-typed-decision-models-vs-a-classical-baseline-vs-a-llm", "canonical_source": "https://www.machinebrief.com/news/measure-before-you-train-typed-decision-models-vs-a-classica-ersc", "published_at": "2026-10-26 04:00:00+00:00", "updated_at": "2026-10-26 05:42:11.321926+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-agents", "ai-research", "large-language-models"], "entities": ["Foyer", "Tev1", "Nimble", "Nimble v4-max", "Qwen2.5-7B", "Gemma 4", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/measure-before-you-train-typed-decision-models-vs-a-classical-baseline-vs-a-llm", "markdown": "https://wpnews.pro/news/measure-before-you-train-typed-decision-models-vs-a-classical-baseline-vs-a-llm.md", "text": "https://wpnews.pro/news/measure-before-you-train-typed-decision-models-vs-a-classical-baseline-vs-a-llm.txt", "jsonld": "https://wpnews.pro/news/measure-before-you-train-typed-decision-models-vs-a-classical-baseline-vs-a-llm.jsonld"}}