You're on-call. A payment failed. You pull the logs:
kubectl logs deployment/payment-svc --since=1h | wc -l
Now what? You could grep for "timeout" — but that gives you 47 exact matches out of context. You could pipe it into an LLM, but 100k lines blows past any context window. You need the 200 lines that matter.
That's what logcompress does. Give it a natural language query and a log file, and it returns only the relevant lines — scored, ranked, and expanded with trace context.
logcompress search -q "why did payment 42 timeout" -f app.log --stats
Output: the 200 lines that tell the story, from the initial request through retries to the final timeout error.
Every line is tokenized into hash tokens using FNV-1a. No String allocation — we work with u64 hashes directly:
fn hash_token(bytes: &[u8]) -> u64 {
let mut h: u64 = 0xcbf29ce484222325; // FNV offset basis
for &b in bytes {
let b = if b.is_ascii_uppercase() { b + 32 } else { b };
h ^= b as u64;
h = h.wrapping_mul(0x100000001b3); // FNV prime
}
h
}
This runs in parallel via rayon. 100k lines tokenize in under 50ms.
Each candidate line gets a TF-IDF cosine similarity score against the query. This is the standard information retrieval approach — terms that appear rarely in the corpus but match the query get high scores.
But raw TF-IDF has a problem with structured logs: if every JSON line contains "service":"payment", then "payment" appears in every document and gets zero discriminative power. The solution is a score gap filter: after scoring, we drop lines scoring less than 25% of the top score. This cleanly separates signal from noise.
If the query mentions "trace-abc123" and a log line has "trace_id":"trace-abc123", that's a stronger signal than the same string appearing in a message. We give configurable multipliers to structured fields:
error_id: 3.0x
trace_id: 3.0x
span_id: 2.5x
order_id: 2.0x
This is where logcompress differs from a simple TF-IDF search.
Consider a timeout incident:
Line 1: "Initiating payment request to gateway" (trace_id: tr-001)
Line 2: "Retrying gateway connection, attempt 1" (trace_id: tr-001)
Line 3: "Gateway response slow, approaching timeout" (trace_id: tr-001)
Line 4: "Payment gateway timeout after 30000ms" (trace_id: tr-001)
Only Line 4 matches the query "timeout". Lines 1-3 use different vocabulary. But they share the same trace_id. After scoring, we extract trace IDs from high-scoring hits and pull all lines sharing those IDs.
This takes recall from ~70% to 100% on our benchmark.
Finally, we expand ±N lines around each hit and merge overlapping ranges. This catches log lines immediately before/after an incident that don't share a trace ID — like the request that triggered the timeout, or the recovery action that followed.
We tested against a 100k-line synthetic dataset with 50 hidden incidents (timeout, OOM, deadlock, auth failure, rate limit):
| Metric | Value |
|---|---|
| Recall | 100% (55/55 queries) |
| Latency (100k lines) | < 250ms |
| Compression | 100k → 100-200 lines |
| Tool | Approach | Relevance |
|---|---|---|
| grep | Exact string match | None |
| jq | Field filtering | None |
| ripgrep | Fast regex | None |
| logcompress | TF-IDF + trace expansion | Ranked |
Install from crates.io:
cargo install logcompress-cli
As a library:
use logcompress::{compress, CompressConfig};
let result = compress("payment timeout", &logs, &CompressConfig::default());
println!("Found {} relevant lines out of {}",
result.stats.output_lines, result.stats.input_lines);
pip install logcompress)
The code is MIT/Apache-2.0: github.com/TimurRakhmatullin86/logcompress
What query patterns would you need for your production logs? What edge cases would break this?