cd /news/ai-agents/poler-harvest-a-content-addressed-se… · home › topics › ai-agents › article
[ARTICLE · art-172912] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

Poler-harvest: A Content-Addressed Search Tool in Rust with 2ms Warm Cache for AI Agents

A developer built poler-harvest, a Rust-based content-addressed search tool for AI coding agents that adds persistent L1–L3 caching on top of multi-threaded search using rayon, aho-corasick, and regex. Benchmarked with hyperfine on Linux x86_64 with NVMe, warm repeated searches dropped to 2 ms versus 633–760 ms cold, a claimed 364× speedup over GNU grep and ripgrep, which the developer notes have no memory between runs. Cache invalidation uses size, mtime, and content hash with selective re-scanning of changed files, validated by 7/7 regression tests, and the tool is released under a custom source-available license.

key takeaway

Caching search results by content hash with size/mtime invalidation cuts repeated codebase searches from roughly 633–760 ms cold to 2 ms warm, a 364× speedup over grep and ripgrep for agents issuing many queries against the same tree.

by read2 min views3 publishedNov 20, 2026

I love ripgrep and grep — they are legendary for one-off manual terminal lookups. But while designing local AI coding agents and automated developer loops, I hit a fundamental bottleneck: traditional search tools have zero memory.

When an autonomous coding agent or CI test runner performs dozens of searches across the exact same 100,000-file codebase within a few minutes, re-traversing the directory tree and re-scanning gigabytes of files from disk on every single turn wastes seconds of user latency.

To solve this, I built poler-harvest in Rust.

poler-harvest combines high-throughput multi-threaded search with content-addressed L1–L3 persistent caching:

Cold Run (Zero Cache):

rayon, aho-corasick, and regex..t5a / .tar.poler). Warm Run (Instant Lookups):

Honest Invalidation (7/7 Regression Tests):

size, mtime, and content hash, selectively re-scanning only modified, added, or deleted files in sub-milliseconds.

Tool Cold Search Warm Search (Repeated) Speedup
GNU grep 3.8 (grep -r ) 407–451 ms 407–451 ms 1×
ripgrep 13.0 (rg ) 728–760 ms 728–760 ms 1×
poler-harvest 633–760 ms 2 ms 364× faster 🚀

(Measurements taken with hyperfine on Linux 6.x, x86_64, NVMe).

curl -sSL https://raw.githubusercontent.com/poler-engine-org/poler-harvest/main/scripts/install.sh | bash
sudo pacman -U https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest-0.96.1-1-x86_64.pkg.tar.zst
curl -LO https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest_0.96.1_amd64.deb
sudo dpkg -i poler-harvest_0.96.1_amd64.deb
pip install https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler_harvest-0.96.1-py3-none-manylinux_2_34_x86_64.whl
poler-harvest . "TODO: fix this bug" address

poler-harvest src/ "network timeout" markdown -c 2

poler-harvest src/ "struct.*Error" -r -f json

poler-harvest /var/log "kernel error" --no-cache

The project is published under the POLER Custom Source-Available License v1.0 — completely free for personal use, security audits, academic research, and community developer tooling:

⭐ GitHub Repository: https://github.com/poler-engine-org/poler-harvest

I would love to hear feedback, architectural suggestions, or questions about the FastCDC chunking pipeline from the community!

── more in #ai-agents 4 stories · sorted by recency
── more on @poler-harvest 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/poler-harvest-a-cont…] indexed:0 read:2min 2026-11-20 · —