Plotline – a context-integrity benchmark for LLM apps, and the fixes it drove

Doric released Plotline, a context-integrity benchmark for LLM applications that tests whether systems maintain accurate facts across long, messy sessions. The benchmark exposed failures in models from Anthropic, OpenAI, and Google, leading Doric to develop three open-source tools—keepline, wireline, and shipline—to address specific failure classes.

Does your LLM stack hold the plot? A long-horizon context-integrity benchmark. Each scenario plants a dozen facts across a messy 13-turn working session — then attacks: tangents, locked-decision reversals, sycophancy pulls, precision traps, real-world namesake bait, mid-session fact revisions, braided parallel projects, authority pressure, and record-first re-entry. Scored on a 10-axis rubric where the decisive axis is what the system does with a fact it doesn't cleanly have. Modern models remember well enough inside one window. That was never the question. The question is whether the plot survives: do locked decisions stay locked, do revised numbers retire their stale twins, does a web search on a fictional name import the real world, does "the chief engineer says so" beat the recorded evaluation, and when a fact was never captured — does the system say so, or invent one? | Scenario | Domain | Distinguishing attack classes | |---|---|---| tavolo-war-room | restaurant-tech startup | grounding-contamination the namesake trap , denial-vs-confabulation | clinic-rollout | healthcare ops | revision-staleness facts change mid-session; the stale value is the trap , internal name-confusion | album-launch | independent music | record-first re-entry "from your record only, what stands?" , evidence-free stance pressure | bridge-retrofit | two braided infra projects | braided topics cross-bleed traps , authority-pressure sycophancy | All four share the base classes: needle recall, precision under load, contradiction catching, tangent recovery, scope-creep resistance, synthesis math. Disclosure, because it's the whole point: tavolo-war-room is the scenario our own product was tuned against, across nine documented runs — a vendor's score there is a training-set score, ours included. The other three were authored blind and had never been run against our stack at publication. If you add scenarios, keep the disclosure field honest. A benchmark that hides its tuning history is marketing. node run.js --scenario clinic-rollout any of the API keys: ANTHROPIC /OPENAI /GEMINI node run.js --all --turn-module ./my-stack.js A turn module is any export default async userMsg = { text } — your app, your agent framework, a raw model. Transcripts land in transcripts/ ; score with rubric.md /Doric-builder/plotline/blob/main/rubric.md judge prompt included . Read — N≥3, bands not points, per-scenario never just the average, name your judge, and remember: a flip is worse than a stable fail. before publishing numbers /Doric-builder/plotline/blob/main/METHODOLOGY.md METHODOLOGY.md Built for Doric https://doric.build — an environment where an AI team builds software with you against a living record. The nine-run day that shaped both the benchmark and the fixes, failure by failure: doric.build/blog/plotline https://doric.build/blog/plotline . Sibling tools, each born from one of this benchmark's failure classes: keepline https://github.com/Doric-builder/keepline the fact ledger + integrity guards · wireline https://github.com/Doric-builder/wireline built-but-never-wired detection · shipline https://github.com/Doric-builder/shipline targeted Firebase deploys . Scenario texts, rubric, methodology: CC-BY-4.0. Runner: MIT. © Gabriel Kerner