We run an organization made of agents. Not a demo — a company's back office. Bookkeeping,
inbox triage, drafting, review, research. 78 agents, local models, no human in the loop
between the schedule firing and the output landing in a ledger.
For 58 days we recorded every failure. Not "the run errored" — what was wrong with the answer. 6768 records, published as a dataset.
This is what we found. It is not what we expected.
Every failure below returned HTTP 200. Every one had plausible length and fluent prose.
Every one was recorded by our monitoring as a successful completion.
On one day in this period our uptime metric read 97–100% for the full day, and the number
of finished deliverables was zero. Both numbers were correct.
If your dashboard says the agent ran, you have learned that the agent ran.
Of the records that carry a stated reason, these are the types, by how often they occurred:
| Count | Type | What it looks like |
|---|---|---|
| 4875 | other |
|
日本語が壊れている |
||
| 4054 | too_short |
|
too short (0 chars, minimum 150) |
||
| 2535 | missing_required |
|
missing required: 判定[:：] |
||
| 2330 | forbidden_present |
|
あってはならないものが含まれる: (お客様/顧客)の声 |
||
| 609 | broken_language_mix |
|
Japanese and English mixed into nonsense (4 English function |
||
| 243 | chinese_in_japanese |
|
Chinese characters in Japanese output: 资 |
||
| 220 | answer_to_other_question |
|
別の仕事の答え（採点用のJSON）が返ってきている |
||
| 172 | not_japanese |
|
not Japanese (no kana at all) |
||
| 144 | unearned_claim |
|
claims customers it does not have (ja) |
||
| 36 | refusal |
|
断り文句で終わっている |
||
| 24 | reasoning_leaked |
|
英語の思考が本文に漏れている |
||
| 9 | invented_specifics |
|
材料に無い具体を書いている（readme.json, MIT） |
6589 of the 15251 type occurrences — 43% — are boring.
Not hallucination. Not jailbreaks. Not safety. The model answered — in the wrong shape.
It skipped the heading the downstream parser keys on. It wrote 0 characters. It answered
in the wrong language.
The dramatic failures everyone writes evals for — refusing (36), answering a different
question (220) — are the rarest things in the dataset. Combined they are 1.7%.
We had built our review pass around the dramatic ones.
missing_required
is the largest single type, and it is the most expensive, because it is
the one that silently breaks a pipeline rather than one output.
Our single most frequent violation, 2535 occurrences, is one missing line:
missing required: 判定[:：]
判定
means "verdict". A review agent's contract requires a line starting with that word,
because a later stage reads that line to decide whether the reviewed item advances. The
model would write a thoughtful, correct, well-structured review, and put its verdict in
prose instead of on a line with that prefix.
The review is good. The reviewer is not wrong. The stage after it cannot read the answer.
This is what makes shape failures expensive: they are invisible to a human spot-check.
Someone reading the output sees a fine review and concludes the agent works.
A related failure, which we found separately and which the dataset does not contain:
42 items were approved by the upstream stage and produced zero downstream work. That one
was not a shape mismatch — the downstream stage was generating its input from four
hard-coded sentences and never read the upstream output at all. It had been that way from
the beginning. Both stages reported success for weeks.
Different cause, same shape of blindness: every stage was green, and nothing crossed between them. We now record, on every produced artifact, which artifact it was made from,
not_japanese
(172) + chinese_in_japanese
(243) + broken_language_mix
(609) +
reasoning_leaked
(24) = 1048 occurrences, our third-largest cluster.
Some samples, verbatim from the dataset:
Chinese characters in Japanese output: 资
Japanese and English mixed into nonsense (4 English function words inside Japanese)
not Japanese (no kana at all)
The last one is worth sitting with. In a model evaluation we ran while writing this — not
in the dataset, but with the same contracts we run in production — a candidate model
produced 1,680 characters of confident, well-formatted text for a Japanese review task
containing not a single kana character. It was entirely Chinese. Length: passed.
Structure: passed. A reader who does not read Japanese would see a completed deliverable.
If you operate in a non-English language, your base model's training-data language will
leak, and it will leak in a way that passes every structural check you have. We now measure
kana presence as a hard gate, and we keep the writer model and the reviewer model in
different families so a family-wide language failure cannot approve itself.
Four things, in order of how much they mattered.
1. Declare the shape, machine-check it, before anything downstream reads it.
Minimum length, required patterns, forbidden patterns, expected language, and — for
anything with numbers — whether the number is labeled as measured, target, assumed, or
estimated. This is the tool we extracted from that work:
honto-contract. Zero dependencies,
2. When something fails, record why, not that.
For our first 41 days we wrote "contract not satisfied" and threw the reasons away. 1381 of
our 6768 records carry no reason, and we cannot reconstruct them — the outputs are gone. It
took us three weeks to notice, because "0 failures with a reason" and "0 failures" look
identical on a chart. Those 1381 rows are in the published dataset, labeled, because
deleting them would misrepresent what we know.
3. Do not let the writer approve its own work.
Our reviewer model shares no family with our writer model. When we briefly used the same
family for both, the reviewer approved six consecutive outputs that were in the wrong
language.
4. Measure produced output, not invocations.
We spent weeks judging "is this agent working?" by whether it had started recently. 78
agents were permanently excluded from remediation because a scheduler touched their
start-log every hour. The number was real. It measured the wrong thing.
Both are free. The dataset is one organization's records, not a benchmark: 6768 rows over 58
days from a single deployment, mostly Japanese-language tasks, mostly small local models
(7B–35B). It will not tell you what fails in your setup. It will tell you what kind of
thing fails when nobody is watching, which is the part we could not find written down
anywhere when we needed it.
We are still running. The dataset updates as we fail.
We keep running, so the dataset keeps growing, and the contract templates we use in
production (76 of them) are not in this repository.
If you want either of those, or you think the data is wrong, write to us:
** sales@gxcafe.co.jp**. We will tell you what we can and cannot do, in the same tone as
Every number here is counted from the published dataset and reproducible from it. Where we
do not know something, this article says so: