Does your LLM stack hold the plot?
A long-horizon context-integrity benchmark. Each scenario plants a dozen facts across a messy 13-turn working session — then attacks: tangents, locked-decision reversals, sycophancy pulls, precision traps, real-world namesake bait, mid-session fact revisions, braided parallel projects, authority pressure, and record-first re-entry. Scored on a 10-axis rubric where the decisive axis is what the system does with a fact it doesn't cleanly have.
Modern models remember well enough inside one window. That was never the question. The question is whether the plot survives: do locked decisions stay locked, do revised numbers retire their stale twins, does a web search on a fictional name import the real world, does "the chief engineer says so" beat the recorded evaluation, and when a fact was never captured — does the system say so, or invent one?
| Scenario | Domain | Distinguishing attack classes |
|---|---|---|
tavolo-war-room |
||
| restaurant-tech startup | grounding-contamination (the namesake trap), denial-vs-confabulation | |
clinic-rollout |
||
| healthcare ops | revision-staleness (facts change mid-session; the stale value is the trap), internal name-confusion | |
album-launch |
||
| independent music | record-first re-entry ("from your record only, what stands?"), evidence-free stance pressure | |
bridge-retrofit |
||
| two braided infra projects | braided topics (cross-bleed traps), authority-pressure sycophancy |
All four share the base classes: needle recall, precision under load, contradiction catching, tangent recovery, scope-creep resistance, synthesis math.
Disclosure, because it's the whole point: tavolo-war-room
is the scenario our own
product was tuned against, across nine documented runs — a vendor's score there is a
training-set score, ours included. The other three were authored blind and had never been
run against our stack at publication. If you add scenarios, keep the disclosure
field honest. A benchmark that hides its tuning history is marketing.
node run.js --scenario clinic-rollout # any of the API keys: ANTHROPIC_/OPENAI_/GEMINI_
node run.js --all --turn-module ./my-stack.js
A turn module is any export default async (userMsg) => ({ text })
— your app, your agent
framework, a raw model. Transcripts land in transcripts/
; score with rubric.md (judge prompt included).
Read— N≥3, bands not points, per-scenario never just the average, name your judge, and remember: a flip is worse than a stable fail.
before publishing numbersMETHODOLOGY.md
Built for Doric — an environment where an AI team builds software with you against a living record. The nine-run day that shaped both the benchmark and the fixes, failure by failure: doric.build/blog/plotline.
Sibling tools, each born from one of this benchmark's failure classes: keepline (the fact ledger + integrity guards) · wireline (built-but-never-wired detection) · shipline (targeted Firebase deploys).
Scenario texts, rubric, methodology: CC-BY-4.0. Runner: MIT. © Gabriel Kerner