{"slug": "why-ctxd-let-the-agent-grep", "title": "Why ctxd: Let the agent grep", "summary": "A blog post titled \"Why ctxd: Let the agent grep\" argues that ctxd's grep-first retrieval, which keeps selected folders searchable through ctxfs and lets the calling agent supply search terms and read source files, is simpler and cheaper than embedding-based RAG. The post's comparison table states ctxd requires no embedding generation or embedding API fees and makes updated text searchable instantly after sync, while embedding RAG needs chunking, an embedding model, a vector index and re-embedding on file changes. The post concedes that grep-first search depends on the agent guessing useful vocabulary, since a search for \"RUNBOOK\" will not match a file merely because a reader would call it a runbook, and says automatic vocabulary expansion is not part of the local search engine.", "body_md": "[← Back to blog](https://ctxd.dev/blog)\n\n# Why ctxd: Let the agent grep\n\n## TL;DR\n\n**Use ctxd. It is simpler and cheaper, and it works just as well, if not better.**\n\nFrequent updates, tight budgets, latency targets. Fewer moving parts, fewer ways to break.\n\nScroll sideways to compare.\n\n| Dimension | ctxd | Embedding RAG | \n|---|---|---|\n| Setup complexity | **Simple.** Copy text and retain paths. | **Multiple stages.** Split text, embed chunks and maintain a vector index. | \n| Tuning | **None.** | Chunk size and overlap, embedding model, retrieval depth and optional reranking. | \n| Embedding compute and API cost | **Free.** No embedding generation or embedding API fees. | **Paid compute.** CPU/GPU time for embedding and re-embedding. Hosted APIs and indexes add service fees. | \n| Query latency | **No embedding step.** Text scans and repeated agent turns can add latency. | **Indexed lookup.** Query embedding and optional reranking add latency. | \n| File updates | **Instant after sync.** Updated text is searchable without re-embedding. | **Time + compute cost.** Rebuild affected chunks, re-embed changed text and update the index. | \n| Vocabulary matching | **Pattern-based.** The agent must supply useful terms or try alternatives. | **Similarity-based.** Can retrieve related wording without an exact term match. | \n\nFree software still uses compute and storage. The agent’s model can add usage charges with either approach.\n\nSpeed depends on corpus size, hardware, caching and the agent’s search strategy. These are architectural trade-offs; end-to-end performance still needs benchmarking.\n\nYou ask an agent how to roll back a deployment after its health checks fail. It finds a line about restoring the previous image. But the instruction immediately above it says to stop the rollout first.\n\nThe agent needs a way to read beyond the match. **Grep first search. No embeddings.** That is our starting point for ctxd: searchable text, source paths and tools the agent can call again.\n\nctxd’s local tracker keeps selected folders available through ctxfs. Your application gives the agent tools to inspect paths, search text and read source files. The agent chooses the queries.\n\n## What an embedding engine adds\n\nEmbedding-based retrieval usually splits text into chunks, encodes them as vectors and maintains an index. Each layer brings decisions: where chunks end, which embedding model to use and how to keep the index current.\n\nWe want to start with searchable text and source reads. Follow the same runbook through preparation, a question and a source edit.\n\n## Finding the words to search\n\nThe weakness of grep-first search is the starting vocabulary. “Undo a deploy” may miss a file that says “rollback.” Searching for `RUNBOOK` will not find a file just because a reader would call it a runbook. Regex matches text patterns; it does not infer that meaning.\n\nToday, an agent can list folders and paths, read headings, and try alternatives such as `rollback|revert|undo`. It can use terms discovered in one source to search the next. ctxd exposes those operations; the calling agent supplies the vocabulary and judges the evidence.\n\nOur direction is to make that discovery loop easier to guide and inspect: choose a useful scope, try candidate terms, read evidence, then refine. Automatic vocabulary expansion is not part of the local search engine. The [ctxfs guide](https://ctxd.dev/docs/ctxfs/#read-operations) describes the available read operations.\n\n## Read around the match\n\nA regex can find an error code, a service name or a phrase. The result gives the caller a file path and line number to inspect.\n\nIn this example, searching for `rollback` finds line 3. Reading the surrounding passage reveals the step that comes before it.\n\n**Search supplies a lead.** This query completes and finds line 3. The instruction on line 2 contains no match for “rollback,” so it is absent.\n\n**Reading reveals the prerequisite.** The surrounding passage establishes the order: stop the rollout, restore the image, then recheck health.\n\n**Figure 2. A complete search is not a complete answer.** Both panels use the reproducible file below. The highlighted prerequisite changes the procedure the agent can infer. Text wraps within each numbered source line.\n\nThat read may reveal another file or a term worth searching. A caller can narrow the next query to a folder or path prefix. The agent follows the evidence, and developers can inspect its sequence of calls.\n\n## Try the two calls\n\nUse the local service from the [quickstart](https://ctxd.dev/docs/quickstart/). The example calls its [ctxfs API](https://ctxd.dev/docs/api/) to find a line, then read its context.\n\n## Create and track the example file\n\nOn your workstation, as your normal user, start in a directory where `ctxd-thesis-example` does not exist:\n\n```\nmkdir ctxd-thesis-example && cat > ctxd-thesis-example/runbook.md <<'TEXT'# Deployment notesIf deployment health checks fail, stop the rollout.Rollback: restore the previous image tag, then recheck health.TEXT\n```\n\nAfter installing ctxd, start local ctxfs and track this example folder:\n\n```\nctxd backend set ctxfsctxd folders add ./ctxd-thesis-example --name ThesisExamplectxd tracker start\n```\n\nWait for a successful scan. The CLI resolves `ThesisExample` to its tracked path\nand connects through the configured local socket. See the [quickstart](https://ctxd.dev/docs/quickstart/)\nfor service checks.\n\nSearch for the matching line, then read the three-line passage:\n\n```\nctxd files grep '(?i)rollback' --prefix ThesisExample --limit 20\nctxd files read-lines ThesisExample/runbook.md --start 1 --end 3\n```\n\nThe search returns one matching line. The read returns all three lines, including the instruction to stop the rollout. An agent can use that evidence before deciding what to do next.\n\n## Know what a result leaves out\n\n- **Different wording can be missed.** Regex matches text patterns; it does not rank by meaning. A synonym may need another query.\n- **A response can be partial.** Inspect`complete` and`stopped_by` before treating search results as exhaustive.\n- **Tracked files take time to refresh.** Changes become searchable after a successful scan. See the[local tracker guide](https://ctxd.dev/docs/local-tracker/) for service controls.\n\nctxd keeps local copies of tracked text. See the [security and data guide](https://ctxd.dev/docs/security/) for retention and sharing details.\n\n## Build on an inspectable interface\n\nThe [local implementation](https://github.com/ctxd-dev/ctxd) is [MIT licensed](https://github.com/ctxd-dev/ctxd/blob/main/LICENSE). Developers can inspect how files are tracked and what each call returns.\n\nDirect file access may already serve your application. ctxd adds a common interface and folder tracking, with a service and storage to manage. We want to make that trade-off useful for agents that need to follow the evidence across files.\n\n## Glossary\n\n| Term | Meaning | \n|---|---|\n| Embedding | A vector representation used to compare queries and passages by similarity. | \n| Chunk | A portion of a document indexed as a retrieval unit. | \n| Reranking | Scoring retrieved candidates against the query to reorder them. | \n| Context expansion | Fetching neighboring passages or a parent document around a retrieved chunk. | \n\n## References\n\nResearch and technical sources behind the design:\n\n1. [*Is Grep All You Need?*](https://arxiv.org/html/2605.15184v1) : retrieval tools and how results reach the agent.\n2. [*SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering*](https://arxiv.org/abs/2405.15793v3) : interfaces for navigating, searching and editing files.\n3. [*Towards Retrieving Interaction Spaces for Agentic Search*](https://arxiv.org/html/2606.06880v1) : selecting a workspace for an agent to explore.\n4. [Retrieval architectures](https://docs.langchain.com/oss/python/deepagents/retrieval) : retrieval components and agentic workflows.\n5. [Vector indexing](https://developers.llamaindex.ai/python/framework/module_guides/indexing/vector_store_index/) ,[embedding models](https://developers.llamaindex.ai/python/framework/module_guides/models/embeddings/) and[retrieval tuning](https://developers.llamaindex.ai/python/framework/optimizing/basic_strategies/basic_strategies/) : chunks, vectors, model choices and retrieval settings.\n6. [Document management](https://developers.llamaindex.ai/python/framework/module_guides/indexing/document_management/) : refreshing an index when source documents change.\n7. [Parent-document retrieval](https://reference.langchain.com/python/langchain-classic/retrievers/parent_document_retriever/ParentDocumentRetriever) and[retrieval postprocessing](https://developers.llamaindex.ai/python/framework/module_guides/querying/node_postprocessors/) : fetching broader context and reranking candidate passages.\n\nThe papers evaluate their own systems and conditions. Choosing no embeddings is our design decision; these references do not establish a universal retrieval winner or measure ctxd.\n\nRead the source. Try the local workflow. Help shape the project.\n\n[Explore ctxd on GitHub](https://github.com/ctxd-dev/ctxd)", "url": "https://wpnews.pro/news/why-ctxd-let-the-agent-grep", "canonical_source": "https://ctxd.dev/blog/why-ctxd", "published_at": "2026-11-03 22:39:56+00:00", "updated_at": "2026-11-03 22:50:10.663310+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-search"], "entities": ["ctxd", "ctxfs"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/why-ctxd-let-the-agent-grep", "markdown": "https://wpnews.pro/news/why-ctxd-let-the-agent-grep.md", "text": "https://wpnews.pro/news/why-ctxd-let-the-agent-grep.txt", "jsonld": "https://wpnews.pro/news/why-ctxd-let-the-agent-grep.jsonld"}}