{"slug": "two-knowledge-hierarchies-structuring-context-for-ai-agents-and-llms", "title": "Two Knowledge Hierarchies: Structuring Context for AI Agents and LLMs", "summary": "TestSmith has implemented a two-tier knowledge hierarchy to provide AI agents and LLMs with structured, task-specific context. For AI agents working on the TestSmith codebase, the system uses a `CLAUDE.md` hierarchy with per-package files that agents load automatically when touching specific directories, ensuring they receive only relevant context. For runtime test generation, TestSmith merges a root `TESTSMITH.md` file with optional per-directory overrides into the system prompt, exempting this project knowledge from the configurable token budget that trims other content.", "body_md": "TestSmith has two distinct audiences that need context about the project: AI agents that work *on* the TestSmith codebase (helping develop and extend it), and the LLM that generates test code *for your project* at runtime. These are different problems with different solutions.\n\nWhen an AI agent opens TestSmith to fix a bug or add a feature, it needs to understand the codebase structure without reading every file. A single large context file doesn't work well — an agent fixing a retry bug doesn't need to know the Java driver's fixture generation logic.\n\nThe solution is a `CLAUDE.md`\n\nhierarchy:\n\n```\nCLAUDE.md                              ← package map, invariants, dependency direction\ninternal/domain/CLAUDE.md             ← interfaces, key types, \"add a field\" checklist\ninternal/generation/CLAUDE.md         ← pipeline data flow, verifier selection\ninternal/llm/CLAUDE.md                ← middleware stack, batch vs fan-out, cache key\ninternal/projectknowledge/CLAUDE.md   ← TESTSMITH.md hierarchy, budget tiers\ninternal/drivers/CLAUDE.md            ← how to add an adapter or language driver\n```\n\nThe root file is the map. The per-package files are the territory. An agent touching the LLM retry logic loads `internal/llm/CLAUDE.md`\n\n— it never sees the driver or generation docs.\n\nThe root file contains three things that every agent needs regardless of task:\n\n`domain`\n\nnever imports other internal packages; `drivers`\n\nnever import `generation`\n\n)`GeneratedFile.Language`\n\nmust always be set; `resolveAction`\n\nhas specific rules for fixture vs. non-fixture files)Per-package files contain the \"read this before touching this package\" context: data flow diagrams for the pipeline, the middleware stack for the LLM layer, the adapter registration pattern for drivers.\n\nWhen Claude Code loads a file in a package, it automatically reads that package's `CLAUDE.md`\n\n. The agent gets exactly what it needs, nothing more.\n\nThis is what TestSmith injects into prompts when generating tests for *your* project. It's a conventions file you maintain alongside your source code.\n\nTwo levels are merged at generation time:\n\n```\n<project-root>/TESTSMITH.md     ← always loaded; project-wide framework, mock style\n<source-dir>/TESTSMITH.md       ← optional; package-level overrides\n```\n\nExample root `TESTSMITH.md`\n\n:\n\n```\n# Project conventions\n\nFramework: pytest\nMock style: pytest-mock (use `mocker.patch`, not `unittest.mock.patch`)\nAssertion style: plain assert statements\n\n# Module structure\nServices are in `src/services/`. Each service has a single public class.\nTests go in `tests/` mirroring the `src/` structure.\n```\n\nExample per-directory override in `src/services/payment/TESTSMITH.md`\n\n:\n\n```\n# Payment service conventions\nThis module integrates with Stripe. Mock all `stripe.*` calls.\nUse `pytest.mark.vcr` for HTTP interaction tests.\n```\n\nThe root file is loaded once at startup and cached in `ProjectContext`\n\n. The per-directory file is merged lazily — only when a file in that directory is being generated. A large monorepo never loads context it doesn't need.\n\n**Both files go into the system prompt, not the user prompt.** This matters because the user prompt is subject to a configurable token budget (`PromptTokenBudget`\n\n, default 6,000 tokens) with a priority-based trim:\n\n| Priority | Content | Dropped when? |\n|---|---|---|\n| 1 (never) | Source code | Never |\n| 2 | Internal dep signatures | Budget exceeded after source |\n| 3 | Style snippet from nearby tests | Dropped first |\n\nProject knowledge is exempt from this budget entirely — it stays in the system prompt regardless of how large the source file is.\n\nBeyond `TESTSMITH.md`\n\n, TestSmith also mines conventions from existing tests in the same directory — up to 5 files, capped at 80 lines total. This gives the model real examples of the project's test style without requiring the developer to maintain a conventions doc.\n\nThis is cheaper and more accurate than a hand-written guide: it automatically reflects the actual test patterns in use, and it updates itself as tests evolve. If your team starts using a new assertion pattern, the next generation run picks it up.\n\nThe third piece is the dep index: at the start of a `--all`\n\nrun, TestSmith analyses every source file once and builds a `modulePath → SourceAnalysis`\n\nmap. When generating tests for `payment.go`\n\n, it can pull the public API signature of `discount.go`\n\n(which `payment.go`\n\nimports) from memory:\n\n```\n// In the prompt:\n// Internal dependency signatures:\n// discount.ApplyPromoCode(order Order, code string) (Order, error)\n// discount.ValidateCode(code string) bool\n```\n\nThis tells the model what the real interface looks like so it generates test doubles that match the actual signatures — not invented ones.\n\nIn watch mode, when a file changes, only that file's entry is refreshed. The rest of the index stays warm between regens.\n\nThe two layers solve different problems:\n\n**Agent context** is about *development-time* navigation. It's hierarchical, human-readable, and loaded selectively. It describes architecture and invariants. It lives in the repo and is maintained alongside the code it describes.\n\n**Runtime LLM context** is about *generation-time* quality. It's merged from two levels, injected into system prompts, and exempt from token budgets. It describes conventions and patterns specific to the target project — things an LLM can't infer from source code alone.\n\nConflating the two leads to either bloated system prompts (dumping agent context into every generation request) or under-informed agents (giving them only the user-facing conventions doc with no architectural guidance). Keeping them separate means each audience gets exactly what it needs.\n\n*Next: the cross-platform bugs we hit shipping a Go CLI — detector boundary escapes and Windows path separators.*", "url": "https://wpnews.pro/news/two-knowledge-hierarchies-structuring-context-for-ai-agents-and-llms", "canonical_source": "https://dev.to/orieken/two-knowledge-hierarchies-structuring-context-for-ai-agents-and-llms-2o6c", "published_at": "2026-05-27 05:22:10+00:00", "updated_at": "2026-05-27 05:22:38.398963+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["TestSmith", "CLAUDE.md"], "alternates": {"html": "https://wpnews.pro/news/two-knowledge-hierarchies-structuring-context-for-ai-agents-and-llms", "markdown": "https://wpnews.pro/news/two-knowledge-hierarchies-structuring-context-for-ai-agents-and-llms.md", "text": "https://wpnews.pro/news/two-knowledge-hierarchies-structuring-context-for-ai-agents-and-llms.txt", "jsonld": "https://wpnews.pro/news/two-knowledge-hierarchies-structuring-context-for-ai-agents-and-llms.jsonld"}}