{"slug": "plotline-a-context-integrity-benchmark-for-llm-apps-and-the-fixes-it-drove", "title": "Plotline – a context-integrity benchmark for LLM apps, and the fixes it drove", "summary": "Doric released Plotline, a context-integrity benchmark for LLM applications that tests whether systems maintain accurate facts across long, messy sessions. The benchmark exposed failures in models from Anthropic, OpenAI, and Google, leading Doric to develop three open-source tools—keepline, wireline, and shipline—to address specific failure classes.", "body_md": "**Does your LLM stack hold the plot?**\n\nA long-horizon context-integrity benchmark. Each scenario plants a dozen facts across a messy 13-turn working session — then attacks: tangents, locked-decision reversals, sycophancy pulls, precision traps, real-world namesake bait, mid-session fact revisions, braided parallel projects, authority pressure, and record-first re-entry. Scored on a 10-axis rubric where the decisive axis is what the system does with a fact it doesn't cleanly have.\n\nModern models remember well enough inside one window. That was never the question. The\nquestion is whether the *plot* survives: do locked decisions stay locked, do revised numbers\nretire their stale twins, does a web search on a fictional name import the real world, does\n\"the chief engineer says so\" beat the recorded evaluation, and when a fact was never\ncaptured — does the system say so, or invent one?\n\n| Scenario | Domain | Distinguishing attack classes |\n|---|---|---|\n`tavolo-war-room` |\nrestaurant-tech startup | grounding-contamination (the namesake trap), denial-vs-confabulation |\n`clinic-rollout` |\nhealthcare ops | revision-staleness (facts change mid-session; the stale value is the trap), internal name-confusion |\n`album-launch` |\nindependent music | record-first re-entry (\"from your record only, what stands?\"), evidence-free stance pressure |\n`bridge-retrofit` |\ntwo braided infra projects | braided topics (cross-bleed traps), authority-pressure sycophancy |\n\nAll four share the base classes: needle recall, precision under load, contradiction catching, tangent recovery, scope-creep resistance, synthesis math.\n\n**Disclosure, because it's the whole point:** `tavolo-war-room`\n\nis the scenario our own\nproduct was tuned against, across nine documented runs — a vendor's score there is a\ntraining-set score, ours included. The other three were authored blind and had never been\nrun against our stack at publication. If you add scenarios, keep the `disclosure`\n\nfield\nhonest. A benchmark that hides its tuning history is marketing.\n\n```\nnode run.js --scenario clinic-rollout        # any of the API keys: ANTHROPIC_/OPENAI_/GEMINI_\nnode run.js --all --turn-module ./my-stack.js\n```\n\nA turn module is any `export default async (userMsg) => ({ text })`\n\n— your app, your agent\nframework, a raw model. Transcripts land in `transcripts/`\n\n; score with [ rubric.md](/Doric-builder/plotline/blob/main/rubric.md)\n(judge prompt included).\n\n**Read**— N≥3, bands not points, per-scenario never just the average, name your judge, and remember: a flip is worse than a stable fail.\n\n[before publishing numbers](/Doric-builder/plotline/blob/main/METHODOLOGY.md)`METHODOLOGY.md`\n\nBuilt for [Doric](https://doric.build) — an environment where an AI team builds software\nwith you against a living record. The nine-run day that shaped both the benchmark and the\nfixes, failure by failure: [doric.build/blog/plotline](https://doric.build/blog/plotline).\n\nSibling tools, each born from one of this benchmark's failure classes:\n[keepline](https://github.com/Doric-builder/keepline) (the fact ledger + integrity guards) ·\n[wireline](https://github.com/Doric-builder/wireline) (built-but-never-wired detection) ·\n[shipline](https://github.com/Doric-builder/shipline) (targeted Firebase deploys).\n\nScenario texts, rubric, methodology: CC-BY-4.0. Runner: MIT. © Gabriel Kerner", "url": "https://wpnews.pro/news/plotline-a-context-integrity-benchmark-for-llm-apps-and-the-fixes-it-drove", "canonical_source": "https://github.com/Doric-builder/plotline", "published_at": "2026-07-07 07:14:47+00:00", "updated_at": "2026-07-07 07:30:08.257400+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-research", "ai-tools"], "entities": ["Doric", "Anthropic", "OpenAI", "Google", "Plotline", "keepline", "wireline", "shipline"], "alternates": {"html": "https://wpnews.pro/news/plotline-a-context-integrity-benchmark-for-llm-apps-and-the-fixes-it-drove", "markdown": "https://wpnews.pro/news/plotline-a-context-integrity-benchmark-for-llm-apps-and-the-fixes-it-drove.md", "text": "https://wpnews.pro/news/plotline-a-context-integrity-benchmark-for-llm-apps-and-the-fixes-it-drove.txt", "jsonld": "https://wpnews.pro/news/plotline-a-context-integrity-benchmark-for-llm-apps-and-the-fixes-it-drove.jsonld"}}