Why Your LLM Pipeline Is Burning 60% of Its Token Budget on Noise (and How to Fix It)

A developer built LLMSlim, a Python library for surgical prompt compression in RAG pipelines, to address the problem of filler tokens burning up to 60% of token budgets. The library uses a deterministic 6-step pipeline including TF-IDF vector graph construction and LexRank centrality scoring to reduce token count without losing critical information. LLMSlim offers both extractive and hybrid compression modes, with the extractive pipeline running in under 30ms fully offline.

I run a lot of RAG pipelines. And for a while, I was doing what most people do: retrieve the top-k documents, concatenate them into the context, and ship the whole thing to the LLM. It worked. But when I started looking closely at what was actually in those contexts, I noticed something uncomfortable: a huge fraction of every prompt was filler. Background sentences that restated the obvious. Transition paragraphs that connected ideas the model didn't need connected. Prose padding that existed because technical documents are written for humans who need continuity, not for models that need information density. And I was paying for every token of it. Here's what makes this particularly painful: it's not just a billing issue. Long-context prompts have real performance implications. Quadratic attention cost. Transformer attention scales as O n^2 with sequence length. Adding 2,000 filler tokens to a 4,000-token context doesn't increase cost by 50% - it increases the prefill computation by nearly 225%. Lost in the Middle. Research consistently shows that LLMs underweight information in the middle of long contexts. If your most relevant retrieved chunk lands in position 12 of 20, the model may not give it the attention it deserves. Instruction drift. This is the one that bit me hardest. When system directives are diluted across thousands of tokens of prose, models sometimes drop low-frequency rules. A JSON schema requirement buried after 3,000 tokens of context is easier to forget than one sitting at token 200. I spent about three months building LLMSlim, a Python library for surgical prompt compression. The goal was simple: reduce the token count of prompts significantly without losing any critical information and without touching system instructions. The core of the library is a 6-step deterministic pipeline: python from llmslim import compress One line to compress your context result = compress your massive rag context, target ratio=0.5, Keep 50% of tokens strategy="extractive" 100% offline print result.compressed text Compressed context ready to use print result.token reduction e.g., 0.52 52% reduction Step 1: Protected Sentence Splitting The text is split into sentences using regex patterns that avoid breaking on code fences , URLs, or Markdown title markers. AST fence content is preserved as atomic units. Step 2: TF-IDF Vector Graph Construction Each sentence becomes a vector in TF-IDF space. We compute pairwise cosine similarities to build a weighted graph where edges represent semantic similarity. Step 3: LexRank Centrality Scoring We run power iteration over the stochastic transition matrix derived from the similarity graph. Sentences with higher stationary probability scores are more informationally central to the document. Step 4: Priority Tier Hard Locking Before any pruning happens, a deterministic rule pass identifies and hard-locks: These sentences are immune to the scoring algorithm. They survive regardless of centrality. Step 5: Two-Pass Budget Allocation Pass 1 allocates token budgets proportionally across semantic chunks. Pass 2 rebalances globally to hit the target ratio, using a priority-aware knapsack allocation. Step 6: Ordered Reassembly Selected sentences are reassembled in their original document order. This preserves logical flow and causal reasoning chains. The extractive pipeline is fast sub-30ms and fully offline. But for contexts where you want even higher compression quality, v0.3.0 introduces hybrid mode: python from llmslim import compress, CallableProvider, RewriteRequest def my rewriter req: RewriteRequest - str: return openai client.chat.completions.create model="gpt-4o-mini", messages= {"role": "user", "content": req.user prompt} .choices 0 .message.content provider = CallableProvider my rewriter result = compress context, target ratio=0.45, strategy="hybrid", provider=provider The hybrid mode runs extractive pre-pruning first removing obvious noise , then passes the reduced context to the LLM rewriter for semantic compression. This achieves higher information density than pure extraction while keeping the LLM call focused on a smaller input. All benchmarks are reproducible and run on N=500 prompts per dataset: | Dataset | Token Reduction | Latency | Directive Retention | |---|---|---|---| | System Directives | 51.4% | 24.8ms | 100.0% | | 50k Long Context GPT-5 | 55.0% | 26.0ms | 100.0% | | XML Mode Claude 3.5 | 50.0% | 24.0ms | 100.0% | | 100k Megabyte RAG Gemini | 65.0% | 38.0ms | 100.0% | The 100% directive retention across all test domains is the number I'm most proud of. It's not coincidental - it's the Priority Tier system doing exactly what it was designed to do. A few things surprised me: TF-IDF is surprisingly competitive with embeddings for this task. I initially assumed I'd need neural embeddings for good centrality scoring. TF-IDF vector space turned out to work remarkably well for identifying genuinely redundant prose, largely because redundant sentences tend to share vocabulary. The two-pass budget allocator matters more than I expected. A naive single-pass selection consistently underperforms because it doesn't account for how information clusters within documents. The second pass global rebalancing step is responsible for maybe 8-10% of the quality improvement over simpler approaches. Instruction fidelity is a hard constraint, not a soft one. Early versions of the pipeline treated all sentences equally by score. The moment I realized I needed a hard exclusion layer not just upweighting for directives, the results became dramatically more reliable. bash pip install llmslim Full documentation, integration guides for LangChain, LlamaIndex, FastAPI, Ollama, and more are at https://www.llmslim.app https://www.llmslim.app The source is on GitHub and the benchmark methodology is fully open if you want to reproduce or challenge the numbers. I'd genuinely love to hear from anyone running production RAG pipelines about what compression approaches you've tried and what's worked. This is still a largely unsolved problem at the edges.