I love ripgrep and grep — they are legendary for one-off manual terminal lookups. But while designing local AI coding agents and automated developer loops, I hit a fundamental bottleneck: traditional search tools have zero memory.
When an autonomous coding agent or CI test runner performs dozens of searches across the exact same 100,000-file codebase within a few minutes, re-traversing the directory tree and re-scanning gigabytes of files from disk on every single turn wastes seconds of user latency.
To solve this, I built poler-harvest in Rust.
poler-harvest combines high-throughput multi-threaded search with content-addressed L1–L3 persistent caching:
Cold Run (Zero Cache):
rayon, aho-corasick, and regex..t5a / .tar.poler).
Warm Run (Instant Lookups):
Honest Invalidation (7/7 Regression Tests):
size, mtime, and content hash, selectively re-scanning only modified, added, or deleted files in sub-milliseconds.
| Tool | Cold Search | Warm Search (Repeated) | Speedup |
|---|---|---|---|
GNU grep 3.8 (grep -r ) |
407–451 ms | 407–451 ms | 1× |
ripgrep 13.0 (rg ) |
728–760 ms | 728–760 ms | 1× |
| poler-harvest | 633–760 ms | 2 ms | 364× faster 🚀 |
(Measurements taken with hyperfine on Linux 6.x, x86_64, NVMe).
curl -sSL https://raw.githubusercontent.com/poler-engine-org/poler-harvest/main/scripts/install.sh | bash
sudo pacman -U https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest-0.96.1-1-x86_64.pkg.tar.zst
curl -LO https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest_0.96.1_amd64.deb
sudo dpkg -i poler-harvest_0.96.1_amd64.deb
pip install https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler_harvest-0.96.1-py3-none-manylinux_2_34_x86_64.whl
poler-harvest . "TODO: fix this bug" address
poler-harvest src/ "network timeout" markdown -c 2
poler-harvest src/ "struct.*Error" -r -f json
poler-harvest /var/log "kernel error" --no-cache
The project is published under the POLER Custom Source-Available License v1.0 — completely free for personal use, security audits, academic research, and community developer tooling:
⭐ GitHub Repository: https://github.com/poler-engine-org/poler-harvest
I would love to hear feedback, architectural suggestions, or questions about the FastCDC chunking pipeline from the community!