Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox

A measurement protocol applied to Foyer, a self-hosted personal inbox agent in daily use, found that its production keyword rules scored 28% on message type against a 58% majority class, according to an arXiv paper (2610.32955v1) evaluated on 739 real Italian/English emails labeled by the owner. Under thread-grouped cross-validation, a TF-IDF + logistic-regression baseline trained on roughly 200 labels had the highest accuracy on every field (type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90), though it never predicted the two rarest classes and was statistically tied with Gemma 4 on type. The authors report that the most accurate system, composed into actions, ignored emails needing action 18% of the time versus 4% for the best LLM, and that a cost-chosen threshold on its noise probability halved that rate while making it the cheapest system.

arXiv:2610.32955v1 Announce Type: new Abstract: Personal email agents decide, for every message, what it is, whether it is urgent, whether to draft a reply, create a task, or ignore it. Their error rate is rarely measured: the stored labels were written by the system itself. We present Foyer, a self-hosted personal inbox agent in daily use, and a measurement protocol stratified sampling, frozen snapshots, thread-grouped splits, pre-registered hypotheses, human labels in the agent's review loop that yields ground truth before any training. On 739 real Italian/English emails labelled by the owner 231 unbiased, 500 adversarially sampled , we compare nine systems under thread-grouped cross-validation: the production keyword rules, a TF-IDF + logistic-regression baseline, local typed decision models Tev1, Nimble , a hosted decision model Nimble v4-max , and hosted generative LLMs Qwen2.5-7B, Gemma 4 26B/31B . Four results. The production rules score 28% on message type against a 58% majority class: the field shown most prominently carried no information. With ~200 labels, logistic regression has the highest accuracy on every field type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90 , though it never predicts the two rarest classes and is statistically tied with Gemma 4 on type; few-shot prompting does not close the gap. The sender's relationship client, supplier, service is recovered from other emails of the same contact with 98.6% accuracy where the contact has been seen, and is not inferable from a single message by any model: it belongs in the contact graph, not the prompt. Finally, accuracy is not the product metric: composed into actions, the most accurate system most often ignores emails needing action 18% vs 4% for the best LLM ; a cost-chosen threshold on its noise probability halves that rate and makes it the cheapest system.