What actually breaks when you run LLM agents unattended for 58 days

An organization running 78 autonomous LLM agents for back-office tasks published a dataset of 6,768 failures recorded over 58 days, finding that 43% of errors were 'boring' shape mismatches—such as missing required headings or wrong-language output—rather than dramatic hallucinations or refusals. The most frequent failure, 'missing_required,' occurred 2,535 times and silently broke downstream pipelines, while dramatic failures like refusals and answering different questions accounted for only 1.7% of occurrences. The project also discovered a case where two stages both reported success but no work crossed between them, highlighting systemic blind spots in agent monitoring.

We run an organization made of agents. Not a demo — a company's back office. Bookkeeping, inbox triage, drafting, review, research. 78 agents, local models, no human in the loop between the schedule firing and the output landing in a ledger. For 58 days we recorded every failure. Not "the run errored" — what was wrong with the answer . 6768 records, published as a dataset. This is what we found. It is not what we expected. Every failure below returned HTTP 200. Every one had plausible length and fluent prose. Every one was recorded by our monitoring as a successful completion. On one day in this period our uptime metric read 97–100% for the full day, and the number of finished deliverables was zero. Both numbers were correct. If your dashboard says the agent ran, you have learned that the agent ran. Of the records that carry a stated reason, these are the types, by how often they occurred: | Count | Type | What it looks like | |---|---|---| | 4875 | other | 日本語が壊れている | | 4054 | too short | too short 0 chars, minimum 150 | | 2535 | missing required | missing required: 判定 :： | | 2330 | forbidden present | あってはならないものが含まれる: お客様/顧客 の声 | | 609 | broken language mix | Japanese and English mixed into nonsense 4 English function | | 243 | chinese in japanese | Chinese characters in Japanese output: 资 | | 220 | answer to other question | 別の仕事の答え（採点用のJSON）が返ってきている | | 172 | not japanese | not Japanese no kana at all | | 144 | unearned claim | claims customers it does not have ja | | 36 | refusal | 断り文句で終わっている | | 24 | reasoning leaked | 英語の思考が本文に漏れている | | 9 | invented specifics | 材料に無い具体を書いている（readme.json, MIT） | 6589 of the 15251 type occurrences — 43% — are boring. Not hallucination. Not jailbreaks. Not safety. The model answered — in the wrong shape . It skipped the heading the downstream parser keys on. It wrote 0 characters. It answered in the wrong language. The dramatic failures everyone writes evals for — refusing 36 , answering a different question 220 — are the rarest things in the dataset. Combined they are 1.7% . We had built our review pass around the dramatic ones. missing required is the largest single type, and it is the most expensive, because it is the one that silently breaks a pipeline rather than one output. Our single most frequent violation, 2535 occurrences, is one missing line: missing required: 判定 :： 判定 means "verdict". A review agent's contract requires a line starting with that word, because a later stage reads that line to decide whether the reviewed item advances. The model would write a thoughtful, correct, well-structured review, and put its verdict in prose instead of on a line with that prefix. The review is good. The reviewer is not wrong. The stage after it cannot read the answer. This is what makes shape failures expensive: they are invisible to a human spot-check. Someone reading the output sees a fine review and concludes the agent works. A related failure, which we found separately and which the dataset does not contain: 42 items were approved by the upstream stage and produced zero downstream work. That one was not a shape mismatch — the downstream stage was generating its input from four hard-coded sentences and never read the upstream output at all. It had been that way from the beginning. Both stages reported success for weeks. Different cause, same shape of blindness: every stage was green, and nothing crossed between them. We now record, on every produced artifact, which artifact it was made from, not japanese 172 + chinese in japanese 243 + broken language mix 609 + reasoning leaked 24 = 1048 occurrences , our third-largest cluster. Some samples, verbatim from the dataset: Chinese characters in Japanese output: 资 Japanese and English mixed into nonsense 4 English function words inside Japanese not Japanese no kana at all The last one is worth sitting with. In a model evaluation we ran while writing this — not in the dataset, but with the same contracts we run in production — a candidate model produced 1,680 characters of confident, well-formatted text for a Japanese review task containing not a single kana character . It was entirely Chinese. Length: passed. Structure: passed. A reader who does not read Japanese would see a completed deliverable. If you operate in a non-English language, your base model's training-data language will leak, and it will leak in a way that passes every structural check you have. We now measure kana presence as a hard gate, and we keep the writer model and the reviewer model in different families so a family-wide language failure cannot approve itself. Four things, in order of how much they mattered. 1. Declare the shape, machine-check it, before anything downstream reads it. Minimum length, required patterns, forbidden patterns, expected language, and — for anything with numbers — whether the number is labeled as measured, target, assumed, or estimated. This is the tool we extracted from that work: honto-contract https://www.npmjs.com/package/honto-contract . Zero dependencies, 2. When something fails, record why, not that. For our first 41 days we wrote "contract not satisfied" and threw the reasons away. 1381 of our 6768 records carry no reason, and we cannot reconstruct them — the outputs are gone. It took us three weeks to notice, because "0 failures with a reason" and "0 failures" look identical on a chart. Those 1381 rows are in the published dataset, labeled, because deleting them would misrepresent what we know. 3. Do not let the writer approve its own work. Our reviewer model shares no family with our writer model. When we briefly used the same family for both, the reviewer approved six consecutive outputs that were in the wrong language. 4. Measure produced output, not invocations. We spent weeks judging "is this agent working?" by whether it had started recently. 78 agents were permanently excluded from remediation because a scheduler touched their start-log every hour. The number was real. It measured the wrong thing. Both are free. The dataset is one organization's records, not a benchmark: 6768 rows over 58 days from a single deployment, mostly Japanese-language tasks, mostly small local models 7B–35B . It will not tell you what fails in your setup. It will tell you what kind of thing fails when nobody is watching, which is the part we could not find written down anywhere when we needed it. We are still running. The dataset updates as we fail. We keep running, so the dataset keeps growing, and the contract templates we use in production 76 of them are not in this repository. If you want either of those, or you think the data is wrong, write to us: sales@gxcafe.co.jp . We will tell you what we can and cannot do, in the same tone as Every number here is counted from the published dataset and reproducible from it. Where we do not know something, this article says so: