# Plotline – a context-integrity benchmark for LLM apps, and the fixes it drove

> Source: <https://github.com/Doric-builder/plotline>
> Published: 2026-07-07 07:14:47+00:00

**Does your LLM stack hold the plot?**

A long-horizon context-integrity benchmark. Each scenario plants a dozen facts across a messy 13-turn working session — then attacks: tangents, locked-decision reversals, sycophancy pulls, precision traps, real-world namesake bait, mid-session fact revisions, braided parallel projects, authority pressure, and record-first re-entry. Scored on a 10-axis rubric where the decisive axis is what the system does with a fact it doesn't cleanly have.

Modern models remember well enough inside one window. That was never the question. The
question is whether the *plot* survives: do locked decisions stay locked, do revised numbers
retire their stale twins, does a web search on a fictional name import the real world, does
"the chief engineer says so" beat the recorded evaluation, and when a fact was never
captured — does the system say so, or invent one?

| Scenario | Domain | Distinguishing attack classes |
|---|---|---|
`tavolo-war-room` |
restaurant-tech startup | grounding-contamination (the namesake trap), denial-vs-confabulation |
`clinic-rollout` |
healthcare ops | revision-staleness (facts change mid-session; the stale value is the trap), internal name-confusion |
`album-launch` |
independent music | record-first re-entry ("from your record only, what stands?"), evidence-free stance pressure |
`bridge-retrofit` |
two braided infra projects | braided topics (cross-bleed traps), authority-pressure sycophancy |

All four share the base classes: needle recall, precision under load, contradiction catching, tangent recovery, scope-creep resistance, synthesis math.

**Disclosure, because it's the whole point:** `tavolo-war-room`

is the scenario our own
product was tuned against, across nine documented runs — a vendor's score there is a
training-set score, ours included. The other three were authored blind and had never been
run against our stack at publication. If you add scenarios, keep the `disclosure`

field
honest. A benchmark that hides its tuning history is marketing.

```
node run.js --scenario clinic-rollout        # any of the API keys: ANTHROPIC_/OPENAI_/GEMINI_
node run.js --all --turn-module ./my-stack.js
```

A turn module is any `export default async (userMsg) => ({ text })`

— your app, your agent
framework, a raw model. Transcripts land in `transcripts/`

; score with [ rubric.md](/Doric-builder/plotline/blob/main/rubric.md)
(judge prompt included).

**Read**— N≥3, bands not points, per-scenario never just the average, name your judge, and remember: a flip is worse than a stable fail.

[before publishing numbers](/Doric-builder/plotline/blob/main/METHODOLOGY.md)`METHODOLOGY.md`

Built for [Doric](https://doric.build) — an environment where an AI team builds software
with you against a living record. The nine-run day that shaped both the benchmark and the
fixes, failure by failure: [doric.build/blog/plotline](https://doric.build/blog/plotline).

Sibling tools, each born from one of this benchmark's failure classes:
[keepline](https://github.com/Doric-builder/keepline) (the fact ledger + integrity guards) ·
[wireline](https://github.com/Doric-builder/wireline) (built-but-never-wired detection) ·
[shipline](https://github.com/Doric-builder/shipline) (targeted Firebase deploys).

Scenario texts, rubric, methodology: CC-BY-4.0. Runner: MIT. © Gabriel Kerner