{"slug": "how-to-let-gzip-find-the-signal-in-a-pile-of-documents", "title": "How to Let gzip Find the Signal in a Pile of Documents", "summary": "A developer demonstrates a technique using gzip compression ratios to rank text documents by information density, helping to surface potentially valuable files in large collections. The method measures compressibility as a proxy for redundancy, with lower ratios indicating more repetitive content and higher ratios suggesting more varied material. The approach is presented as a first-pass filtering tool rather than a measure of semantic quality.", "body_md": "Suppose you have a directory full of text documents.\n\nMost are repetitive, padded with boilerplate, or otherwise low-signal. A few contain the useful material. You could read every file manually, feed them all into an embedding pipeline, or ask an LLM to rank them.\n\nOr you could ask **gzip**.\n\nThe basic idea is simple:\n\nRepetitive text compresses well. Varied text usually does not.\n\nThat makes compression ratio a crude but surprisingly useful proxy for redundancy.\n\nIt will not tell you which document is *best*. But it can help you identify which documents contain less repetition and deserve a closer look.\n\nFor each document:\n\n`gzip`\n\n.\n\n```\ncompressed size / original size\n```\n\nA lower ratio means the document compressed well, which usually indicates more repetition.\n\nA higher ratio means the document was harder to compress, which may indicate more varied or information-dense content.\n\nIn other words:\n\nHere is a small Bash pipeline that ranks `.txt`\n\nfiles by compression ratio:\n\n```\nfind ./documents -type f -name '*.txt' -print0 |\nwhile IFS= read -r -d '' file; do\n  raw=$(wc -c < \"$file\")\n  compressed=$(gzip -n -c -- \"$file\" | wc -c)\n\n  awk -v file=\"$file\" -v raw=\"$raw\" -v gz=\"$compressed\" '\n    raw > 0 {\n      printf \"%.3f\\t%8d\\t%8d\\t%s\\n\", gz/raw, raw, gz, file\n    }\n  '\ndone | sort -nr\n```\n\nExample output:\n\n```\n0.642      18432      11834  ./documents/research-notes.txt\n0.417      30211      12600  ./documents/project-summary.txt\n0.091      27102       2467  ./documents/standard-contract.txt\n```\n\nThe columns are:\n\n```\nratio    original bytes    compressed bytes    filename\n```\n\nBecause the output is sorted in descending order, the least compressible files appear first.\n\nThose are the files I would inspect first when looking for the possible “gems.”\n\nTo find the most repetitive documents instead, reverse the sort:\n\n```\nsort -n\n```\n\n`gzip -n`\n\n?\nThe `-n`\n\nflag prevents `gzip`\n\nfrom storing the original filename and timestamp in its output.\n\nThat makes the compressed sizes more comparable across files and across runs.\n\nWithout it, a small amount of unrelated metadata can leak into the measurement.\n\nThis technique does not measure truth, relevance, writing quality, or semantic importance.\n\nIt measures compressibility.\n\nThose things sometimes correlate, but they are not the same.\n\nA document full of repeated boilerplate will usually compress extremely well. A document with more distinct vocabulary, sentence structure, numbers, and ideas may compress less efficiently.\n\nThat makes the ratio useful as a first-pass ranking signal.\n\nIt is closer to a metal detector than a treasure map.\n\n`gzip`\n\nadds headers and other fixed overhead. For tiny files, that overhead can dominate the result.\n\nYou may want to ignore documents below a minimum size:\n\n```\nfind ./documents -type f -name '*.txt' -size +1k -print0\n```\n\nRunning this directly against PDF, DOCX, ZIP, JPG, or other compressed formats mostly measures the compression characteristics of the container format.\n\nExtract the text first.\n\nFor example, with PDFs:\n\n```\npdftotext input.pdf output.txt\n```\n\nEncrypted data, random identifiers, hashes, minified code, and corrupted text are all difficult to compress.\n\nThey may score highly while containing little useful information.\n\nContracts, API documentation, technical specifications, and scientific papers may repeat terminology because precision requires it.\n\nA lower ratio can indicate redundancy, but it can also indicate consistency.\n\nCompression ratios can be affected by:\n\nFor a fairer comparison, normalize the documents first.\n\nFor example:\n\n```\ntr -s '[:space:]' ' ' < input.txt\n```\n\nYou could also strip HTML, remove headers and footers, or convert everything to lowercase before compression.\n\nJust remember that normalization changes what you are measuring.\n\nFor larger collections, I would filter out tiny files and print the percentage saved:\n\n```\nfind ./documents -type f -name '*.txt' -size +1k -print0 |\nwhile IFS= read -r -d '' file; do\n  raw=$(wc -c < \"$file\")\n  compressed=$(gzip -n -c -- \"$file\" | wc -c)\n\n  awk -v file=\"$file\" -v raw=\"$raw\" -v gz=\"$compressed\" '\n    raw > 0 {\n      ratio = gz / raw\n      saved = 100 * (1 - ratio)\n\n      printf \"%6.2f%% saved\\t%8d bytes\\t%s\\n\",\n             saved, raw, file\n    }\n  '\ndone | sort -n\n```\n\nThis sorts the files with the lowest percentage saved first, meaning the least compressible documents rise to the top.\n\nThis trick can be handy for quickly triaging:\n\nIt is especially useful when you want a fast local heuristic without setting up a database, embedding model, or external API.\n\nThe broader idea is more interesting than the Bash command.\n\nCompression ratio can be treated as a lightweight feature in a ranking system.\n\nYou could combine it with:\n\nCompression alone is crude.\n\nCompression plus a few other signals could become a genuinely useful document-triage tool.\n\nThere are sophisticated ways to rank a pile of documents.\n\nSometimes, though, a 40-year-old compression algorithm is enough to tell you which files keep repeating themselves.\n\nAnd that is often a very good place to start.", "url": "https://wpnews.pro/news/how-to-let-gzip-find-the-signal-in-a-pile-of-documents", "canonical_source": "https://dev.to/jlmartel/how-to-let-gzip-find-the-signal-in-a-pile-of-documents-2o9g", "published_at": "2026-08-03 23:19:53+00:00", "updated_at": "2026-08-04 00:09:01.189054+00:00", "lang": "en", "topics": ["developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/how-to-let-gzip-find-the-signal-in-a-pile-of-documents", "markdown": "https://wpnews.pro/news/how-to-let-gzip-find-the-signal-in-a-pile-of-documents.md", "text": "https://wpnews.pro/news/how-to-let-gzip-find-the-signal-in-a-pile-of-documents.txt", "jsonld": "https://wpnews.pro/news/how-to-let-gzip-find-the-signal-in-a-pile-of-documents.jsonld"}}