# Poler-harvest: A Content-Addressed Search Tool in Rust with 2ms Warm Cache for AI Agents

> Source: <https://dev.to/kotokvit/poler-harvest-a-content-addressed-search-tool-in-rust-with-2ms-warm-cache-for-ai-agents-87d>
> Published: 2026-11-20 00:10:02+00:00

**Key takeaway:** Caching search results by content hash with size/mtime invalidation cuts repeated codebase searches from roughly 633–760 ms cold to 2 ms warm, a 364× speedup over grep and ripgrep for agents issuing many queries against the same tree.

I love `ripgrep` and `grep` — they are legendary for one-off manual terminal lookups. But while designing local AI coding agents and automated developer loops, I hit a fundamental bottleneck: **traditional search tools have zero memory**.

When an autonomous coding agent or CI test runner performs dozens of searches across the exact same 100,000-file codebase within a few minutes, re-traversing the directory tree and re-scanning gigabytes of files from disk on every single turn wastes seconds of user latency.

To solve this, I built **[poler-harvest](https://github.com/poler-engine-org/poler-harvest)** in Rust.

`poler-harvest` combines high-throughput multi-threaded search with content-addressed L1–L3 persistent caching:

**Cold Run (Zero Cache):**

`rayon`, `aho-corasick`, and `regex`.`.t5a` / `.tar.poler`).
**Warm Run (Instant Lookups):**

**Honest Invalidation (7/7 Regression Tests):**

`size`, `mtime`, and content hash, selectively re-scanning only modified, added, or deleted files in sub-milliseconds.
| Tool | Cold Search | Warm Search (Repeated) | Speedup | 
|---|---|---|---|
| **GNU grep 3.8** (`grep -r` ) | 407–451 ms | 407–451 ms | 1× | 
| **ripgrep 13.0** (`rg` ) | 728–760 ms | 728–760 ms | 1× | 
| **poler-harvest** | 633–760 ms | **2 ms** | **364× faster** 🚀 | 

*(Measurements taken with `hyperfine` on Linux 6.x, x86_64, NVMe).*

```
curl -sSL https://raw.githubusercontent.com/poler-engine-org/poler-harvest/main/scripts/install.sh | bash
sudo pacman -U https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest-0.96.1-1-x86_64.pkg.tar.zst
curl -LO https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest_0.96.1_amd64.deb
sudo dpkg -i poler-harvest_0.96.1_amd64.deb
pip install https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler_harvest-0.96.1-py3-none-manylinux_2_34_x86_64.whl
# Exact search with 2ms warm lookups:
poler-harvest . "TODO: fix this bug" address

# Contextual markdown search for documentation:
poler-harvest src/ "network timeout" markdown -c 2

# Regex search with machine-readable JSONL stream:
poler-harvest src/ "struct.*Error" -r -f json

# One-off scan without touching persistent cache:
poler-harvest /var/log "kernel error" --no-cache
```

The project is published under the **POLER Custom Source-Available License v1.0** — completely free for personal use, security audits, academic research, and community developer tooling:

⭐ **GitHub Repository:** [https://github.com/poler-engine-org/poler-harvest](https://github.com/poler-engine-org/poler-harvest)

I would love to hear feedback, architectural suggestions, or questions about the FastCDC chunking pipeline from the community!
