# Why Your LLM Pipeline Is Burning 60% of Its Token Budget on Noise (and How to Fix It) > Source: > Published: 2026-07-18 22:13:03+00:00 I run a lot of RAG pipelines. And for a while, I was doing what most people do: retrieve the top-k documents, concatenate them into the context, and ship the whole thing to the LLM. It worked. But when I started looking closely at what was actually in those contexts, I noticed something uncomfortable: a huge fraction of every prompt was filler. Background sentences that restated the obvious. Transition paragraphs that connected ideas the model didn't need connected. Prose padding that existed because technical documents are written for humans who need continuity, not for models that need information density. And I was paying for every token of it. Here's what makes this particularly painful: it's not just a billing issue. Long-context prompts have real performance implications. **Quadratic attention cost.** Transformer attention scales as O(n^2) with sequence length. Adding 2,000 filler tokens to a 4,000-token context doesn't increase cost by 50% - it increases the prefill computation by nearly 225%. **Lost in the Middle.** Research consistently shows that LLMs underweight information in the middle of long contexts. If your most relevant retrieved chunk lands in position 12 of 20, the model may not give it the attention it deserves. **Instruction drift.** This is the one that bit me hardest. When system directives are diluted across thousands of tokens of prose, models sometimes drop low-frequency rules. A JSON schema requirement buried after 3,000 tokens of context is easier to forget than one sitting at token 200. I spent about three months building LLMSlim, a Python library for surgical prompt compression. The goal was simple: reduce the token count of prompts significantly without losing any critical information and without touching system instructions. The core of the library is a 6-step deterministic pipeline: ``` python from llmslim import compress # One line to compress your context result = compress( your_massive_rag_context, target_ratio=0.5, # Keep 50% of tokens strategy="extractive" # 100% offline ) print(result.compressed_text) # Compressed context ready to use print(result.token_reduction) # e.g., 0.52 (52% reduction) ``` **Step 1: Protected Sentence Splitting** The text is split into sentences using regex patterns that avoid breaking on code fences (``` `), URLs, or Markdown title markers. AST fence content is preserved as atomic units.` **Step 2: TF-IDF Vector Graph Construction** Each sentence becomes a vector in TF-IDF space. We compute pairwise cosine similarities to build a weighted graph where edges represent semantic similarity. **Step 3: LexRank Centrality Scoring** We run power iteration over the stochastic transition matrix derived from the similarity graph. Sentences with higher stationary probability scores are more informationally central to the document. **Step 4: Priority Tier Hard Locking** Before any pruning happens, a deterministic rule pass identifies and hard-locks: These sentences are immune to the scoring algorithm. They survive regardless of centrality. **Step 5: Two-Pass Budget Allocation** Pass 1 allocates token budgets proportionally across semantic chunks. Pass 2 rebalances globally to hit the target ratio, using a priority-aware knapsack allocation. **Step 6: Ordered Reassembly** Selected sentences are reassembled in their original document order. This preserves logical flow and causal reasoning chains. The extractive pipeline is fast (sub-30ms) and fully offline. But for contexts where you want even higher compression quality, v0.3.0 introduces hybrid mode: ``python` from llmslim import compress, CallableProvider, RewriteRequest def my_rewriter(req: RewriteRequest) -> str: return openai_client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": req.user_prompt}] ).choices[0].message.content provider = CallableProvider(my_rewriter) result = compress( context, target_ratio=0.45, strategy="hybrid", provider=provider ) ``` The hybrid mode runs extractive pre-pruning first (removing obvious noise), then passes the reduced context to the LLM rewriter for semantic compression. This achieves higher information density than pure extraction while keeping the LLM call focused on a smaller input. All benchmarks are reproducible and run on N=500 prompts per dataset: | Dataset | Token Reduction | Latency | Directive Retention | |---|---|---|---| | System Directives | 51.4% | 24.8ms | 100.0% | | 50k Long Context (GPT-5) | 55.0% | 26.0ms | 100.0% | | XML Mode (Claude 3.5) | 50.0% | 24.0ms | 100.0% | | 100k Megabyte RAG (Gemini) | 65.0% | 38.0ms | 100.0% | The 100% directive retention across all test domains is the number I'm most proud of. It's not coincidental - it's the Priority Tier system doing exactly what it was designed to do. A few things surprised me: **TF-IDF is surprisingly competitive with embeddings for this task.** I initially assumed I'd need neural embeddings for good centrality scoring. TF-IDF vector space turned out to work remarkably well for identifying genuinely redundant prose, largely because redundant sentences tend to share vocabulary. **The two-pass budget allocator matters more than I expected.** A naive single-pass selection consistently underperforms because it doesn't account for how information clusters within documents. The second pass global rebalancing step is responsible for maybe 8-10% of the quality improvement over simpler approaches. **Instruction fidelity is a hard constraint, not a soft one.** Early versions of the pipeline treated all sentences equally by score. The moment I realized I needed a hard exclusion layer (not just upweighting) for directives, the results became dramatically more reliable. ``bash` pip install llmslim ` Full documentation, integration guides for LangChain, LlamaIndex, FastAPI, Ollama, and more are at [https://www.llmslim.app](https://www.llmslim.app) The source is on GitHub and the benchmark methodology is fully open if you want to reproduce or challenge the numbers. I'd genuinely love to hear from anyone running production RAG pipelines about what compression approaches you've tried and what's worked. This is still a largely unsolved problem at the edges.