{"slug": "poler-harvest-a-content-addressed-search-tool-in-rust-with-2ms-warm-cache-for-ai", "title": "Poler-harvest: A Content-Addressed Search Tool in Rust with 2ms Warm Cache for AI Agents", "summary": "A developer built poler-harvest, a Rust-based content-addressed search tool for AI coding agents that adds persistent L1–L3 caching on top of multi-threaded search using rayon, aho-corasick, and regex. Benchmarked with hyperfine on Linux x86_64 with NVMe, warm repeated searches dropped to 2 ms versus 633–760 ms cold, a claimed 364× speedup over GNU grep and ripgrep, which the developer notes have no memory between runs. Cache invalidation uses size, mtime, and content hash with selective re-scanning of changed files, validated by 7/7 regression tests, and the tool is released under a custom source-available license.", "body_md": "I love `ripgrep` and `grep` — they are legendary for one-off manual terminal lookups. But while designing local AI coding agents and automated developer loops, I hit a fundamental bottleneck: **traditional search tools have zero memory**.\n\nWhen an autonomous coding agent or CI test runner performs dozens of searches across the exact same 100,000-file codebase within a few minutes, re-traversing the directory tree and re-scanning gigabytes of files from disk on every single turn wastes seconds of user latency.\n\nTo solve this, I built **[poler-harvest](https://github.com/poler-engine-org/poler-harvest)** in Rust.\n\n`poler-harvest` combines high-throughput multi-threaded search with content-addressed L1–L3 persistent caching:\n\n**Cold Run (Zero Cache):**\n\n`rayon`, `aho-corasick`, and `regex`.`.t5a` / `.tar.poler`).\n**Warm Run (Instant Lookups):**\n\n**Honest Invalidation (7/7 Regression Tests):**\n\n`size`, `mtime`, and content hash, selectively re-scanning only modified, added, or deleted files in sub-milliseconds.\n| Tool | Cold Search | Warm Search (Repeated) | Speedup | \n|---|---|---|---|\n| **GNU grep 3.8** (`grep -r` ) | 407–451 ms | 407–451 ms | 1× | \n| **ripgrep 13.0** (`rg` ) | 728–760 ms | 728–760 ms | 1× | \n| **poler-harvest** | 633–760 ms | **2 ms** | **364× faster** 🚀 | \n\n*(Measurements taken with `hyperfine` on Linux 6.x, x86_64, NVMe).*\n\n```\ncurl -sSL https://raw.githubusercontent.com/poler-engine-org/poler-harvest/main/scripts/install.sh | bash\nsudo pacman -U https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest-0.96.1-1-x86_64.pkg.tar.zst\ncurl -LO https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest_0.96.1_amd64.deb\nsudo dpkg -i poler-harvest_0.96.1_amd64.deb\npip install https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler_harvest-0.96.1-py3-none-manylinux_2_34_x86_64.whl\n# Exact search with 2ms warm lookups:\npoler-harvest . \"TODO: fix this bug\" address\n\n# Contextual markdown search for documentation:\npoler-harvest src/ \"network timeout\" markdown -c 2\n\n# Regex search with machine-readable JSONL stream:\npoler-harvest src/ \"struct.*Error\" -r -f json\n\n# One-off scan without touching persistent cache:\npoler-harvest /var/log \"kernel error\" --no-cache\n```\n\nThe project is published under the **POLER Custom Source-Available License v1.0** — completely free for personal use, security audits, academic research, and community developer tooling:\n\n⭐ **GitHub Repository:** [https://github.com/poler-engine-org/poler-harvest](https://github.com/poler-engine-org/poler-harvest)\n\nI would love to hear feedback, architectural suggestions, or questions about the FastCDC chunking pipeline from the community!", "url": "https://wpnews.pro/news/poler-harvest-a-content-addressed-search-tool-in-rust-with-2ms-warm-cache-for-ai", "canonical_source": "https://dev.to/kotokvit/poler-harvest-a-content-addressed-search-tool-in-rust-with-2ms-warm-cache-for-ai-agents-87d", "published_at": "2026-11-20 00:10:02+00:00", "updated_at": "2026-11-20 00:13:07.188918+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools"], "entities": ["poler-harvest", "Rust", "ripgrep", "GNU grep", "rayon", "aho-corasick", "hyperfine", "POLER Custom Source-Available License"], "also_reported_by": [], "takeaway": "Caching search results by content hash with size/mtime invalidation cuts repeated codebase searches from roughly 633–760 ms cold to 2 ms warm, a 364× speedup over grep and ripgrep for agents issuing many queries against the same tree.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/poler-harvest-a-content-addressed-search-tool-in-rust-with-2ms-warm-cache-for-ai", "markdown": "https://wpnews.pro/news/poler-harvest-a-content-addressed-search-tool-in-rust-with-2ms-warm-cache-for-ai.md", "text": "https://wpnews.pro/news/poler-harvest-a-content-addressed-search-tool-in-rust-with-2ms-warm-cache-for-ai.txt", "jsonld": "https://wpnews.pro/news/poler-harvest-a-content-addressed-search-tool-in-rust-with-2ms-warm-cache-for-ai.jsonld"}}