Token Compression and the Index Protocol

Two repositories, agentic-framework and github-analysis, shipped token-compression and Index Protocol changes on 2026-08-22, according to the Context First field notes, with commit dc7a0b7 recording approximate corpus savings of about 110 KB from the research archive, 50 KB from graphify kit stubs and 12 KB from the README. The agentic-framework commit consolidated governance into an invariants.md registry and removed superseded research from the tree, while github-analysis commits 420fa79 and ed3f446 added and tightened the Index Protocol in graphify-search so agents query a live graph before falling back to Grep. The author frames the work as cutting the per-prompt "context tax" of dumping whole monorepos into an agent's window, and notes remaining backlog in still-large skills such as gh-stack and spec-builder.

Field Notes https://contextfirst.blog/tag/field-notes/ Token Compression and the Index Protocol Compress the corpus; query the index first Agent context is finite. Dump a monorepo wiki into the window and the agent misses the one normative paragraph while you pay for duplicated governance prose. Documentation debt shows up as wrong edits and expensive sessions - the same search exhaustion https://contextfirst.blog/graphify-monorepo-navigation/ problem from the agent side. Here's what this piece covers: the context tax, what 2026-08-22 shipped in two repos, compression principles plus the Index Protocol, the named pattern, and three habits before you scale agentic delivery across more trees. Part of the Field Notes: agentic delivery https://contextfirst.blog/field-notes-agentic-delivery/ series - series closer on making repos navigable and dense for agents. 1. The leadership problem Agents load entire standards trees and still invent process. Humans maintain fifty copies of "how agents consume docs." Cost and quality degrade together. Wrong: "Use smaller prompts." Right: " dc7a0b7 / 420fa79 2026-08-22 : what we index, what we omit, retrieval order before Grep - measured corpus cuts, not vibes." The anti-pattern is dump the whole monorepo - every prompt pays full tree tax. 2. What the git history shows Same day, two repos closed the loop: | Repo | Commit | When | What landed | |---|---|---|---| | agentic-framework | dc7a0b7 | 2026-08-22 11:44 ET | Compress corpus: invariants.md registry, router-only README, deduplicated entry docs, research archive removed | | github-analysis | 420fa79 | 2026-08-22 10:18 ET | Index Protocol in graphify-search - corpus-first navigation | | github-analysis | ed3f446 | 2026-08-22 10:19 ET | Tighten Index Protocol; block loading the full graphify skill as a dump | dc7a0b7 recorded approximate savings in token-compression-plan.md : research archive ~110 KB, graphify kit stubs ~50 KB, README ~12 KB, plus smaller methodology trims. Coarse byte deltas - re-index after changes; do not treat KB as token-perfect without a fresh graphify estimate. 3. What I built Token compression principles agentic-framework : 1. Single source - governance in invariants.md ; entry docs link, do not duplicate. 2. Reference, don't restate - skills cite canonical sections. 3. Normative density - RFC-2119 for requirements; cut marketing repetition. 4. Archive excision - superseded research leaves the tree; history stays in git. 5. Kit stubs - portable copies redirect to canonical docs/graphify/ . Index Protocol github-analysis graphify-search , mirrored across Graphify repos : 1. Prefer live graph graphify update . via Cursor beforeSubmitPrompt - AST-only, no API cost . 2. Retrieval order: query → path → explain → lexical fallback only on miss . 3. Subagents must propagate the rule - no bypass because paths feel "familiar." Index Protocol - cheap structure before expensive reads graphify query "where is recurrence median computed?" graphify path "detect regressions" "write summary md" graphify explain "Index Protocol" Grep/Read only if the graph returns nothing useful Backlog after the Aug 22 pass: still-large skills gh-stack , spec-builder and duplicated standards intros - compression is a quarterly habit, not a one-commit event. Complements Graphify for monorepo navigation https://contextfirst.blog/graphify-monorepo-navigation/ portable kit with density of what gets indexed. 4. Named pattern: Index Protocol + compression Index Protocol - structured repo index first; token compression keeps the indexed corpus dense - omit noise, measure context cost, fall back to Grep only on miss. | Artifact | Who uses it | Decision it supports | |---|---|---| | invariants.md | Agents + humans | Single normative lattice | | token-compression-plan.md | Doc owners | What to cut next | | graphify-search skill | Every agent session | Retrieval order | | Hook graphify-refresh.sh | Cursor users | Fresh AST graph per prompt | What broke: skills that restate the whole operating model burn tokens every load. Skipping graphify because "I know the file" recreates search exhaustion. Measured % token savings beyond the plan's KB table still need a re-index comparison when you claim interview-grade precision. 5. What I'd do as a lead Three habits - steal one this month: 1. Assign a doc owner for the agent-facing corpus - same accountability as code ownership. 2. Enforce Index Protocol in AGENTS.md for repos past a size threshold. Query before Grep is a team rule, not a preference. 3. Run a quarterly compression pass. Delete duplicated "how agents consume" intros; keep one invariants registry. Further reading - Field notes: agentic delivery https://contextfirst.blog/field-notes-agentic-delivery/ - series intro - Prior: Blocker matrix https://contextfirst.blog/blocker-matrix-fix-what-unblocks/ · Graphify monorepo navigation https://contextfirst.blog/graphify-monorepo-navigation/ - Related: Shipping the agentic framework https://contextfirst.blog/shipping-agentic-software-delivery-framework/ · Specifications as contracts https://contextfirst.blog/specifications-as-execution-contracts/ - Anchors: agentic-framework dc7a0b7 · github-analysis 420fa79 / ed3f446 · 2026-08-22