# How to Let gzip Find the Signal in a Pile of Documents > Source: > Published: 2026-08-03 23:19:53+00:00 Suppose you have a directory full of text documents. Most are repetitive, padded with boilerplate, or otherwise low-signal. A few contain the useful material. You could read every file manually, feed them all into an embedding pipeline, or ask an LLM to rank them. Or you could ask **gzip**. The basic idea is simple: Repetitive text compresses well. Varied text usually does not. That makes compression ratio a crude but surprisingly useful proxy for redundancy. It will not tell you which document is *best*. But it can help you identify which documents contain less repetition and deserve a closer look. For each document: `gzip` . ``` compressed size / original size ``` A lower ratio means the document compressed well, which usually indicates more repetition. A higher ratio means the document was harder to compress, which may indicate more varied or information-dense content. In other words: Here is a small Bash pipeline that ranks `.txt` files by compression ratio: ``` find ./documents -type f -name '*.txt' -print0 | while IFS= read -r -d '' file; do raw=$(wc -c < "$file") compressed=$(gzip -n -c -- "$file" | wc -c) awk -v file="$file" -v raw="$raw" -v gz="$compressed" ' raw > 0 { printf "%.3f\t%8d\t%8d\t%s\n", gz/raw, raw, gz, file } ' done | sort -nr ``` Example output: ``` 0.642 18432 11834 ./documents/research-notes.txt 0.417 30211 12600 ./documents/project-summary.txt 0.091 27102 2467 ./documents/standard-contract.txt ``` The columns are: ``` ratio original bytes compressed bytes filename ``` Because the output is sorted in descending order, the least compressible files appear first. Those are the files I would inspect first when looking for the possible “gems.” To find the most repetitive documents instead, reverse the sort: ``` sort -n ``` `gzip -n` ? The `-n` flag prevents `gzip` from storing the original filename and timestamp in its output. That makes the compressed sizes more comparable across files and across runs. Without it, a small amount of unrelated metadata can leak into the measurement. This technique does not measure truth, relevance, writing quality, or semantic importance. It measures compressibility. Those things sometimes correlate, but they are not the same. A document full of repeated boilerplate will usually compress extremely well. A document with more distinct vocabulary, sentence structure, numbers, and ideas may compress less efficiently. That makes the ratio useful as a first-pass ranking signal. It is closer to a metal detector than a treasure map. `gzip` adds headers and other fixed overhead. For tiny files, that overhead can dominate the result. You may want to ignore documents below a minimum size: ``` find ./documents -type f -name '*.txt' -size +1k -print0 ``` Running this directly against PDF, DOCX, ZIP, JPG, or other compressed formats mostly measures the compression characteristics of the container format. Extract the text first. For example, with PDFs: ``` pdftotext input.pdf output.txt ``` Encrypted data, random identifiers, hashes, minified code, and corrupted text are all difficult to compress. They may score highly while containing little useful information. Contracts, API documentation, technical specifications, and scientific papers may repeat terminology because precision requires it. A lower ratio can indicate redundancy, but it can also indicate consistency. Compression ratios can be affected by: For a fairer comparison, normalize the documents first. For example: ``` tr -s '[:space:]' ' ' < input.txt ``` You could also strip HTML, remove headers and footers, or convert everything to lowercase before compression. Just remember that normalization changes what you are measuring. For larger collections, I would filter out tiny files and print the percentage saved: ``` find ./documents -type f -name '*.txt' -size +1k -print0 | while IFS= read -r -d '' file; do raw=$(wc -c < "$file") compressed=$(gzip -n -c -- "$file" | wc -c) awk -v file="$file" -v raw="$raw" -v gz="$compressed" ' raw > 0 { ratio = gz / raw saved = 100 * (1 - ratio) printf "%6.2f%% saved\t%8d bytes\t%s\n", saved, raw, file } ' done | sort -n ``` This sorts the files with the lowest percentage saved first, meaning the least compressible documents rise to the top. This trick can be handy for quickly triaging: It is especially useful when you want a fast local heuristic without setting up a database, embedding model, or external API. The broader idea is more interesting than the Bash command. Compression ratio can be treated as a lightweight feature in a ranking system. You could combine it with: Compression alone is crude. Compression plus a few other signals could become a genuinely useful document-triage tool. There are sophisticated ways to rank a pile of documents. Sometimes, though, a 40-year-old compression algorithm is enough to tell you which files keep repeating themselves. And that is often a very good place to start.