# Token Compression and the Index Protocol

> Source: <https://contextfirst.blog/token-compression-index-protocol/>
> Published: 2026-09-20 13:00:00+00:00

[Field Notes](https://contextfirst.blog/tag/field-notes/)

# Token Compression and the Index Protocol

Compress the corpus; query the index first

Agent context is finite. Dump a monorepo wiki into the window and the agent misses the one normative paragraph while you pay for duplicated governance prose. Documentation debt shows up as wrong edits and expensive sessions - the same [search exhaustion](https://contextfirst.blog/graphify-monorepo-navigation/) problem from the agent side.

**Here's what this piece covers:** the context tax, what 2026-08-22 shipped in two repos, compression principles plus the Index Protocol, the named pattern, and three habits before you scale agentic delivery across more trees.

Part of the [Field Notes: agentic delivery](https://contextfirst.blog/field-notes-agentic-delivery/) series - series closer on making repos navigable and dense for agents.

## 1. The leadership problem

Agents load entire standards trees and still invent process. Humans maintain fifty copies of "how agents consume docs." Cost and quality degrade together.

**Wrong:** "Use smaller prompts."**Right:** "`dc7a0b7` / `420fa79` (2026-08-22): what we index, what we omit, retrieval order before Grep - measured corpus cuts, not vibes."

The anti-pattern is **dump the whole monorepo** - every prompt pays full tree tax.

## 2. What the git history shows

Same day, two repos closed the loop:

| Repo | Commit | When | What landed | 
|---|---|---|---|
| **agentic-framework** | `dc7a0b7` | 2026-08-22 11:44 ET | Compress corpus: `invariants.md` registry, router-only README, deduplicated entry docs, research archive removed | 
| **github-analysis** | `420fa79` | 2026-08-22 10:18 ET | Index Protocol in `graphify-search` - corpus-first navigation | 
| **github-analysis** | `ed3f446` | 2026-08-22 10:19 ET | Tighten Index Protocol; block loading the full graphify skill as a dump | 

`dc7a0b7` recorded approximate savings in `token-compression-plan.md`: research archive ~110 KB, graphify kit stubs ~50 KB, README ~12 KB, plus smaller methodology trims. Coarse byte deltas - re-index after changes; do not treat KB as token-perfect without a fresh graphify estimate.

## 3. What I built

**Token compression principles** (agentic-framework):

1. **Single source** - governance in`invariants.md` ; entry docs link, do not duplicate.
2. **Reference, don't restate** - skills cite canonical sections.
3. **Normative density** - RFC-2119 for requirements; cut marketing repetition.
4. **Archive excision** - superseded research leaves the tree; history stays in git.
5. **Kit stubs** - portable copies redirect to canonical`docs/graphify/` .

**Index Protocol** (github-analysis `graphify-search`, mirrored across Graphify repos):

1. Prefer live graph (`graphify update .` via Cursor`beforeSubmitPrompt` - AST-only, no API cost).
2. Retrieval order: `query` →`path` →`explain` → lexical fallback**only on miss** .
3. Subagents must propagate the rule - no bypass because paths feel "familiar."

```
# Index Protocol - cheap structure before expensive reads
graphify query "where is recurrence median computed?"
graphify path "detect_regressions" "write_summary_md"
graphify explain "Index Protocol"
# Grep/Read only if the graph returns nothing useful
```

Backlog after the Aug 22 pass: still-large skills (`gh-stack`, `spec-builder`) and duplicated standards intros - compression is a quarterly habit, not a one-commit event. Complements [Graphify for monorepo navigation](https://contextfirst.blog/graphify-monorepo-navigation/) (portable kit) with **density** of what gets indexed.

## 4. Named pattern: Index Protocol (+ compression)

**Index Protocol** - structured repo index first; **token compression** keeps the indexed corpus dense - omit noise, measure context cost, fall back to Grep only on miss.

| Artifact | Who uses it | Decision it supports | 
|---|---|---|
| `invariants.md` | Agents + humans | Single normative lattice | 
| `token-compression-plan.md` | Doc owners | What to cut next | 
| `graphify-search` skill | Every agent session | Retrieval order | 
| Hook `graphify-refresh.sh` | Cursor users | Fresh AST graph per prompt | 

**What broke:** skills that restate the whole operating model burn tokens every load. Skipping graphify because "I know the file" recreates search exhaustion. Measured % token savings beyond the plan's KB table still need a re-index comparison when you claim interview-grade precision.

## 5. What I'd do as a lead

Three habits - steal one this month:

1. **Assign a doc owner for the agent-facing corpus** - same accountability as code ownership.
2. **Enforce Index Protocol in `AGENTS.md` for repos past a size threshold.** Query before Grep is a team rule, not a preference.
3. **Run a quarterly compression pass.** Delete duplicated "how agents consume" intros; keep one invariants registry.

### Further reading

- [Field notes: agentic delivery](https://contextfirst.blog/field-notes-agentic-delivery/) - series intro
- Prior: [Blocker matrix](https://contextfirst.blog/blocker-matrix-fix-what-unblocks/) ·[Graphify monorepo navigation](https://contextfirst.blog/graphify-monorepo-navigation/)
- Related: [Shipping the agentic framework](https://contextfirst.blog/shipping-agentic-software-delivery-framework/) ·[Specifications as contracts](https://contextfirst.blog/specifications-as-execution-contracts/)
- Anchors: agentic-framework `dc7a0b7` · github-analysis`420fa79` /`ed3f446` · 2026-08-22