{"slug": "building-a-query-aware-log-compressor-in-rust-from-100k-lines-to-200", "title": "Building a Query-Aware Log Compressor in Rust: From 100k Lines to 200", "summary": "Developer Timur Rakhmatullin has released logcompress, an open-source Rust tool that compresses large log files down to the most relevant lines by using TF-IDF scoring, trace ID expansion, and context windowing. The tool, available on GitHub and crates.io, can reduce a 100,000-line log to 100-200 relevant lines in under 250ms, achieving 100% recall on a synthetic benchmark.", "body_md": "You're on-call. A payment failed. You pull the logs:\n\n```\nkubectl logs deployment/payment-svc --since=1h | wc -l\n# 100,000\n```\n\nNow what? You could grep for \"timeout\" — but that gives you 47 exact matches out of context. You could pipe it into an LLM, but 100k lines blows past any context window. You need the 200 lines that matter.\n\nThat's what [logcompress](https://github.com/TimurRakhmatullin86/logcompress) does. Give it a natural language query and a log file, and it returns only the relevant lines — scored, ranked, and expanded with trace context.\n\n```\nlogcompress search -q \"why did payment 42 timeout\" -f app.log --stats\n```\n\nOutput: the 200 lines that tell the story, from the initial request through retries to the final timeout error.\n\nEvery line is tokenized into hash tokens using FNV-1a. No String allocation — we work with `u64` hashes directly:\n\n``` php\nfn hash_token(bytes: &[u8]) -> u64 {\n    let mut h: u64 = 0xcbf29ce484222325; // FNV offset basis\n    for &b in bytes {\n        let b = if b.is_ascii_uppercase() { b + 32 } else { b };\n        h ^= b as u64;\n        h = h.wrapping_mul(0x100000001b3); // FNV prime\n    }\n    h\n}\n```\n\nThis runs in parallel via rayon. 100k lines tokenize in under 50ms.\n\nEach candidate line gets a TF-IDF cosine similarity score against the query. This is the standard information retrieval approach — terms that appear rarely in the corpus but match the query get high scores.\n\nBut raw TF-IDF has a problem with structured logs: if every JSON line contains `\"service\":\"payment\"`, then \"payment\" appears in every document and gets zero discriminative power. The solution is a **score gap filter**: after scoring, we drop lines scoring less than 25% of the top score. This cleanly separates signal from noise.\n\nIf the query mentions \"trace-abc123\" and a log line has `\"trace_id\":\"trace-abc123\"`, that's a stronger signal than the same string appearing in a message. We give configurable multipliers to structured fields:\n\n```\nerror_id: 3.0x\ntrace_id: 3.0x\nspan_id: 2.5x\norder_id: 2.0x\n```\n\nThis is where logcompress differs from a simple TF-IDF search.\n\nConsider a timeout incident:\n\n```\nLine 1: \"Initiating payment request to gateway\" (trace_id: tr-001)\nLine 2: \"Retrying gateway connection, attempt 1\" (trace_id: tr-001)\nLine 3: \"Gateway response slow, approaching timeout\" (trace_id: tr-001)\nLine 4: \"Payment gateway timeout after 30000ms\" (trace_id: tr-001)\n```\n\nOnly Line 4 matches the query \"timeout\". Lines 1-3 use different vocabulary. But they share the same `trace_id`. After scoring, we extract trace IDs from high-scoring hits and pull all lines sharing those IDs.\n\nThis takes recall from ~70% to 100% on our benchmark.\n\nFinally, we expand ±N lines around each hit and merge overlapping ranges. This catches log lines immediately before/after an incident that don't share a trace ID — like the request that triggered the timeout, or the recovery action that followed.\n\nWe tested against a 100k-line synthetic dataset with 50 hidden incidents (timeout, OOM, deadlock, auth failure, rate limit):\n\n| Metric | Value | \n|---|---|\n| Recall | 100% (55/55 queries) | \n| Latency (100k lines) | < 250ms | \n| Compression | 100k → 100-200 lines | \n\n| Tool | Approach | Relevance | \n|---|---|---|\n| grep | Exact string match | None | \n| jq | Field filtering | None | \n| ripgrep | Fast regex | None | \n| **logcompress** | TF-IDF + trace expansion | Ranked | \n\nInstall from crates.io:\n\n```\ncargo install logcompress-cli\n```\n\nAs a library:\n\n``` js\nuse logcompress::{compress, CompressConfig};\n\nlet result = compress(\"payment timeout\", &logs, &CompressConfig::default());\nprintln!(\"Found {} relevant lines out of {}\", \n    result.stats.output_lines, result.stats.input_lines);\n```\n\n`pip install logcompress`)\nThe code is MIT/Apache-2.0: [github.com/TimurRakhmatullin86/logcompress](https://github.com/TimurRakhmatullin86/logcompress)\n\nWhat query patterns would you need for your production logs? What edge cases would break this?", "url": "https://wpnews.pro/news/building-a-query-aware-log-compressor-in-rust-from-100k-lines-to-200", "canonical_source": "https://dev.to/tim860/building-a-query-aware-log-compressor-in-rust-from-100k-lines-to-200-2nj", "published_at": "2026-09-07 22:01:47+00:00", "updated_at": "2026-09-07 22:31:19.467420+00:00", "lang": "en", "topics": ["developer-tools", "machine-learning", "natural-language-processing"], "entities": ["Timur Rakhmatullin", "logcompress", "GitHub", "crates.io", "Rust"], "alternates": {"html": "https://wpnews.pro/news/building-a-query-aware-log-compressor-in-rust-from-100k-lines-to-200", "markdown": "https://wpnews.pro/news/building-a-query-aware-log-compressor-in-rust-from-100k-lines-to-200.md", "text": "https://wpnews.pro/news/building-a-query-aware-log-compressor-in-rust-from-100k-lines-to-200.txt", "jsonld": "https://wpnews.pro/news/building-a-query-aware-log-compressor-in-rust-from-100k-lines-to-200.jsonld"}}