Token Compression and the Index Protocol

Field Notes

Compress the corpus; query the index first

Agent context is finite. Dump a monorepo wiki into the window and the agent misses the one normative paragraph while you pay for duplicated governance prose. Documentation debt shows up as wrong edits and expensive sessions - the same search exhaustion problem from the agent side.

Here's what this piece covers: the context tax, what 2026-08-22 shipped in two repos, compression principles plus the Index Protocol, the named pattern, and three habits before you scale agentic delivery across more trees.

Part of the Field Notes: agentic delivery series - series closer on making repos navigable and dense for agents.

1. The leadership problem #

Agents load entire standards trees and still invent process. Humans maintain fifty copies of "how agents consume docs." Cost and quality degrade together.

Wrong: "Use smaller prompts."Right: "dc7a0b7 / 420fa79 (2026-08-22): what we index, what we omit, retrieval order before Grep - measured corpus cuts, not vibes."

The anti-pattern is dump the whole monorepo - every prompt pays full tree tax.

2. What the git history shows #

Same day, two repos closed the loop:

dc7a0b7 recorded approximate savings in token-compression-plan.md: research archive ~110 KB, graphify kit stubs ~50 KB, README ~12 KB, plus smaller methodology trims. Coarse byte deltas - re-index after changes; do not treat KB as token-perfect without a fresh graphify estimate.

3. What I built #

Token compression principles (agentic-framework):

1. Single source - governance ininvariants.md ; entry docs link, do not duplicate.
2. Reference, don't restate - skills cite canonical sections.
3. Normative density - RFC-2119 for requirements; cut marketing repetition.
4. Archive excision - superseded research leaves the tree; history stays in git.
5. Kit stubs - portable copies redirect to canonicaldocs/graphify/ .

Index Protocol (github-analysis graphify-search, mirrored across Graphify repos):

1. Prefer live graph (graphify update . via CursorbeforeSubmitPrompt - AST-only, no API cost).
2. Retrieval order: query →path →explain → lexical fallbackonly on miss .
3. Subagents must propagate the rule - no bypass because paths feel "familiar."

graphify query "where is recurrence median computed?"
graphify path "detect_regressions" "write_summary_md"
graphify explain "Index Protocol"

Backlog after the Aug 22 pass: still-large skills (gh-stack, spec-builder) and duplicated standards intros - compression is a quarterly habit, not a one-commit event. Complements Graphify for monorepo navigation (portable kit) with density of what gets indexed.

4. Named pattern: Index Protocol (+ compression) #

Index Protocol - structured repo index first; token compression keeps the indexed corpus dense - omit noise, measure context cost, fall back to Grep only on miss.

What broke: skills that restate the whole operating model burn tokens every load. Skipping graphify because "I know the file" recreates search exhaustion. Measured % token savings beyond the plan's KB table still need a re-index comparison when you claim interview-grade precision.

5. What I'd do as a lead #

Three habits - steal one this month:

1. Assign a doc owner for the agent-facing corpus - same accountability as code ownership.
2. Enforce Index Protocol in AGENTS.md for repos past a size threshold. Query before Grep is a team rule, not a preference.
3. Run a quarterly compression pass. Delete duplicated "how agents consume" intros; keep one invariants registry.

Further reading

• Field notes: agentic delivery - series intro
• Prior: Blocker matrix ·Graphify monorepo navigation
• Related: Shipping the agentic framework ·Specifications as contracts
• Anchors: agentic-framework dc7a0b7 · github-analysis420fa79 /ed3f446 · 2026-08-22