TokenZip v2 — PRD, HLD, LLD
type: NewsArticle
description: TokenZip v2 is a token compression engine that reduces LLM input token costs by up to 95% for coding copilots like Claude Code and Codex by transforming an entire codebase into a multi-level, queryable knowledge graph stored locally in `.tokenzip/db`. It auto-detects module boundaries, supports nested monorepo structures, and stores symbols with relationships (CALLS, IMPLEMENTS, INHERITS, etc.) using SurrealDB with RocksDB storage, enabling incremental parsing and fast queries under 100ms for repos up to 100K files. The system is exposed as an MCP server for AI copilots, kept fresh via git hooks, and includes structured markdown parsing with Mermaid block conversion and cross-reference resolution.
date published: 2026-05-11T08:22:43+00:00
date modified: 2026-05-23T14:07:30+00:00
language: en
is accessible for free: true
article body: # TokenZip — PRD, HLD, LLD --- # 📋 PRD — Product Requirements Document ## 1. Executive Summary **TokenZip v2** transforms Karpathy's llm wiki concept into a gzip like **token compression engine** on top of entire codebase, which can reduce the LLM input token cost upto by 95% when using with Coding Copilots like Claude Code, Codex etc. Instead of generating a flat text summary, it builds a multi-level, queryable, chainable knowledge graph — from repo → modules → files → symbols — stored locally in `.tokenzip/db`, exposed as an **MCP server** for any AI copilot, and kept fresh via **git hooks** ## 2. Problem Statement | Problem | Impact | |---|---| | AI copilots lack structural awareness of large codebases | They hallucinate imports, miss dependencies, suggest changes in wrong modules | | Text-based token references are flat and non-queryable | Cannot ask "which functions depend on this interface?" or "what modules does this feature span?" | | No persistent code intelligence layer | Every session re-parses from scratch, wasting tokens and time | | Documentation (PRD/HLD/LLD/README) is unstructured | AI can't extract workflows, sequence diagrams, or release plans from markdown | | Cross-language dependency tracking is manual | A SQL schema change affecting 3 TS files is invisible until runtime | | Cross-repository dependency tracking is manual | The current repository has no awareness of dependent or upstream repositories, including shared interfaces, API contracts, endpoint usage, schema dependencies, or cross-repo integrations — making impact analysis and coordinated changes error-prone | | Version-aware dependency conflicts are difficult to detect | AI copilots and developers lack visibility into incompatible interface versions, breaking API/schema changes, SDK mismatches, or transitive dependency drift across repositories — causing silent integration failures and upgrade risks ## POC Results ### Under 30 seconds indexing time for a codebase with ~1950 files ### Un…
publisher: Web Pulse
keywords: developer-tools, large-language-models, artificial-intelligence, open-source, products
mentions: TokenZip
mentions: Karpathy
mentions: Claude Code
mentions: Codex
mentions: OpenCode
mentions: Kilo Code
mentions: MCP
citation: CreativeWork