{"slug": "token-compression-and-the-index-protocol", "title": "Token Compression and the Index Protocol", "summary": "Two repositories, agentic-framework and github-analysis, shipped token-compression and Index Protocol changes on 2026-08-22, according to the Context First field notes, with commit dc7a0b7 recording approximate corpus savings of about 110 KB from the research archive, 50 KB from graphify kit stubs and 12 KB from the README. The agentic-framework commit consolidated governance into an invariants.md registry and removed superseded research from the tree, while github-analysis commits 420fa79 and ed3f446 added and tightened the Index Protocol in graphify-search so agents query a live graph before falling back to Grep. The author frames the work as cutting the per-prompt \"context tax\" of dumping whole monorepos into an agent's window, and notes remaining backlog in still-large skills such as gh-stack and spec-builder.", "body_md": "[Field Notes](https://contextfirst.blog/tag/field-notes/)\n\n# Token Compression and the Index Protocol\n\nCompress the corpus; query the index first\n\nAgent context is finite. Dump a monorepo wiki into the window and the agent misses the one normative paragraph while you pay for duplicated governance prose. Documentation debt shows up as wrong edits and expensive sessions - the same [search exhaustion](https://contextfirst.blog/graphify-monorepo-navigation/) problem from the agent side.\n\n**Here's what this piece covers:** the context tax, what 2026-08-22 shipped in two repos, compression principles plus the Index Protocol, the named pattern, and three habits before you scale agentic delivery across more trees.\n\nPart of the [Field Notes: agentic delivery](https://contextfirst.blog/field-notes-agentic-delivery/) series - series closer on making repos navigable and dense for agents.\n\n## 1. The leadership problem\n\nAgents load entire standards trees and still invent process. Humans maintain fifty copies of \"how agents consume docs.\" Cost and quality degrade together.\n\n**Wrong:** \"Use smaller prompts.\"**Right:** \"`dc7a0b7` / `420fa79` (2026-08-22): what we index, what we omit, retrieval order before Grep - measured corpus cuts, not vibes.\"\n\nThe anti-pattern is **dump the whole monorepo** - every prompt pays full tree tax.\n\n## 2. What the git history shows\n\nSame day, two repos closed the loop:\n\n| Repo | Commit | When | What landed | \n|---|---|---|---|\n| **agentic-framework** | `dc7a0b7` | 2026-08-22 11:44 ET | Compress corpus: `invariants.md` registry, router-only README, deduplicated entry docs, research archive removed | \n| **github-analysis** | `420fa79` | 2026-08-22 10:18 ET | Index Protocol in `graphify-search` - corpus-first navigation | \n| **github-analysis** | `ed3f446` | 2026-08-22 10:19 ET | Tighten Index Protocol; block loading the full graphify skill as a dump | \n\n`dc7a0b7` recorded approximate savings in `token-compression-plan.md`: research archive ~110 KB, graphify kit stubs ~50 KB, README ~12 KB, plus smaller methodology trims. Coarse byte deltas - re-index after changes; do not treat KB as token-perfect without a fresh graphify estimate.\n\n## 3. What I built\n\n**Token compression principles** (agentic-framework):\n\n1. **Single source** - governance in`invariants.md` ; entry docs link, do not duplicate.\n2. **Reference, don't restate** - skills cite canonical sections.\n3. **Normative density** - RFC-2119 for requirements; cut marketing repetition.\n4. **Archive excision** - superseded research leaves the tree; history stays in git.\n5. **Kit stubs** - portable copies redirect to canonical`docs/graphify/` .\n\n**Index Protocol** (github-analysis `graphify-search`, mirrored across Graphify repos):\n\n1. Prefer live graph (`graphify update .` via Cursor`beforeSubmitPrompt` - AST-only, no API cost).\n2. Retrieval order: `query` →`path` →`explain` → lexical fallback**only on miss** .\n3. Subagents must propagate the rule - no bypass because paths feel \"familiar.\"\n\n```\n# Index Protocol - cheap structure before expensive reads\ngraphify query \"where is recurrence median computed?\"\ngraphify path \"detect_regressions\" \"write_summary_md\"\ngraphify explain \"Index Protocol\"\n# Grep/Read only if the graph returns nothing useful\n```\n\nBacklog after the Aug 22 pass: still-large skills (`gh-stack`, `spec-builder`) and duplicated standards intros - compression is a quarterly habit, not a one-commit event. Complements [Graphify for monorepo navigation](https://contextfirst.blog/graphify-monorepo-navigation/) (portable kit) with **density** of what gets indexed.\n\n## 4. Named pattern: Index Protocol (+ compression)\n\n**Index Protocol** - structured repo index first; **token compression** keeps the indexed corpus dense - omit noise, measure context cost, fall back to Grep only on miss.\n\n| Artifact | Who uses it | Decision it supports | \n|---|---|---|\n| `invariants.md` | Agents + humans | Single normative lattice | \n| `token-compression-plan.md` | Doc owners | What to cut next | \n| `graphify-search` skill | Every agent session | Retrieval order | \n| Hook `graphify-refresh.sh` | Cursor users | Fresh AST graph per prompt | \n\n**What broke:** skills that restate the whole operating model burn tokens every load. Skipping graphify because \"I know the file\" recreates search exhaustion. Measured % token savings beyond the plan's KB table still need a re-index comparison when you claim interview-grade precision.\n\n## 5. What I'd do as a lead\n\nThree habits - steal one this month:\n\n1. **Assign a doc owner for the agent-facing corpus** - same accountability as code ownership.\n2. **Enforce Index Protocol in `AGENTS.md` for repos past a size threshold.** Query before Grep is a team rule, not a preference.\n3. **Run a quarterly compression pass.** Delete duplicated \"how agents consume\" intros; keep one invariants registry.\n\n### Further reading\n\n- [Field notes: agentic delivery](https://contextfirst.blog/field-notes-agentic-delivery/) - series intro\n- Prior: [Blocker matrix](https://contextfirst.blog/blocker-matrix-fix-what-unblocks/) ·[Graphify monorepo navigation](https://contextfirst.blog/graphify-monorepo-navigation/)\n- Related: [Shipping the agentic framework](https://contextfirst.blog/shipping-agentic-software-delivery-framework/) ·[Specifications as contracts](https://contextfirst.blog/specifications-as-execution-contracts/)\n- Anchors: agentic-framework `dc7a0b7` · github-analysis`420fa79` /`ed3f446` · 2026-08-22", "url": "https://wpnews.pro/news/token-compression-and-the-index-protocol", "canonical_source": "https://contextfirst.blog/token-compression-index-protocol/", "published_at": "2026-09-20 13:00:00+00:00", "updated_at": "2026-10-19 02:50:08.545119+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools"], "entities": ["agentic-framework", "github-analysis", "graphify-search", "Graphify", "Cursor", "gh-stack", "spec-builder"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/token-compression-and-the-index-protocol", "markdown": "https://wpnews.pro/news/token-compression-and-the-index-protocol.md", "text": "https://wpnews.pro/news/token-compression-and-the-index-protocol.txt", "jsonld": "https://wpnews.pro/news/token-compression-and-the-index-protocol.jsonld"}}