{"slug": "building-llmslim-architecture-deep-dive-into-deterministic-prompt-compression", "title": "Building LLMSlim: Architecture Deep-Dive into Deterministic Prompt Compression", "summary": "A developer built LLMSlim, a Python library for deterministic prompt compression that handles edge cases like dropped system instructions and truncated JSON schemas. The library uses a six-stage DAG pipeline with TF-IDF vectorization and LexRank scoring, combined with deterministic rule-based classification to protect critical content. It achieves consistent compression within 2-3% of target ratios while preserving original sentence ordering.", "body_md": "Most prompt compression discussions focus on the happy path: you have a long RAG context, you trim it to 50% of tokens, and your API bill halves. What rarely gets discussed are the failure modes: dropped system instructions, truncated JSON schemas, broken code fences, and entity names that get quietly pruned because they scored low in some similarity metric.\n\nLLMSlim ([https://www.llmslim.app](https://www.llmslim.app)) is a Python library I've spent the last few months building to handle these edge cases properly. This post goes deep on the architecture decisions that make it work.\n\nLLM input prompts have a structural problem. The instructions that govern model behavior (system roles, JSON schemas, MUST/NEVER directives) are typically a tiny fraction of the total token count. The rest is context: retrieved documents, conversation history, background information. And within that context, a significant fraction is prose that exists for human readability, not informational density.\n\nThe challenge is distinguishing high-value informational sentences from low-value filler without a model call, without embeddings, and in under 30ms.\n\nLLMSlim processes prompts through a deterministic DAG (Directed Acyclic Graph) with six stages:\n\nNaive sentence splitting on periods breaks code blocks and URLs. The first stage uses regex-based splitting that respects:\n\n```\n# Pseudocode: core split logic\nsentences = regex_split(text)\nfor i, sent in enumerate(sentences):\n    if is_code_fence(sent): protected[i] = True  # immune to scoring\n```\n\nEach sentence becomes a TF-IDF vector. We compute the full pairwise cosine similarity matrix over these vectors to build a weighted undirected graph where edge weight(i,j) = cosine_similarity(v_i, v_j).\n\nThe choice of TF-IDF over neural embeddings was deliberate. TF-IDF runs in microseconds per sentence. It captures vocabulary overlap well, which is exactly what matters for identifying redundant prose. Neural embeddings add 50-100ms of latency and model loading overhead that isn't justified for this task.\n\nWe apply the LexRank algorithm: convert the similarity graph to a stochastic transition matrix M, then find the stationary distribution via power iteration:\n\n```\np_t+1 = d * M * p_t + (1-d) / n\n```\n\nwhere d is a damping factor (typically 0.85) and n is the sentence count. Convergence typically happens in 20-30 iterations. The resulting stationary probability vector gives each sentence a centrality score representing how informationally central it is to the document.\n\nThis is the stage that separates LLMSlim from naive extraction approaches. Before any sentence can be pruned, a deterministic rule pass classifies every sentence into one of four tiers:\n\n**Tier 4 (Inviolable)**: Sentences containing:\n\n`system:`\n\n, `developer:`\n\n, `user:`\n\n`MUST`\n\n, `NEVER`\n\n, `ALWAYS`\n\n, `REQUIRED`\n\n, `DO NOT`\n\n**Tier 3 (Protected)**: Sentences containing:\n\n**Tier 2 (Standard)**: Regular content sentences\n\n**Tier 1 (Candidate for removal)**: Filler phrases, transition sentences\n\nTier 4 sentences are hardcoded to survive the compression pass regardless of their LexRank score. A sentence saying \"You MUST respond only in JSON\" will score low in a document full of prose paragraphs - but it's the most important sentence in the prompt.\n\nPass 1 divides the document into semantic chunks and allocates token budgets proportionally:\n\n```\nchunk_budget_i = target_tokens * (chunk_tokens_i / total_tokens)\n```\n\nPass 2 applies a global rebalancing step. Each chunk may have selected sentences that, together, slightly exceed or undershoot its budget. The second pass collects the surplus/deficit across all chunks and redistributes tokens using a priority-aware greedy knapsack:\n\nThis two-pass approach is why the library consistently hits within 2-3% of the target ratio even on highly variable document structures.\n\nSelected sentences are sorted by their original document position and concatenated. Original ordering is preserved because causal reasoning and logical flow in documents depends on sequence, not just content.\n\nv0.3.0 adds a generative compression layer on top of the extractive pipeline. The flow:\n\nThe pluggable provider model means you don't need a specific LLM API key:\n\n``` python\nfrom llmslim import compress, CallableProvider, RewriteRequest\n\ndef my_provider(req: RewriteRequest) -> str:\n    # req.system_prompt, req.user_prompt, req.target_ratio all available\n    return your_llm_function(req.user_prompt)\n\nprovider = CallableProvider(my_provider)\nresult = compress(context, target_ratio=0.4, strategy=\"hybrid\", provider=provider)\n```\n\nAll benchmarks are reproducible. Hardware: AMD EPYC 7763, 64GB RAM, Ubuntu 24.04, Python 3.12.3, tiktoken cl100k_base. N=500 prompts per dataset, 100 iterations per sample.\n\n| Dataset | Token Reduction | Latency (mean) | Directive Retention | Entity Preservation |\n|---|---|---|---|---|\n| System Directives | 51.4% ± 1.2% | 24.8ms ± 2.1ms | 100.0% | 95.1% ± 1.1% |\n| 50k Context (GPT-5) | 55.0% ± 1.1% | 26.0ms ± 2.4ms | 100.0% | 94.9% ± 1.2% |\n| XML Mode (Claude 3.5) | 50.0% ± 0.8% | 24.0ms ± 1.9ms | 100.0% | 96.4% ± 0.9% |\n| 100k RAG (Gemini) | 65.0% ± 1.3% | 38.0ms ± 3.1ms | 100.0% | 94.8% ± 1.3% |\n\n**Attempt 1: Pure TF-IDF cutoff.** Setting a similarity threshold and dropping sentences below it sounds simple. In practice, document sections with specialized vocabulary score low across the board even when they contain critical information. Threshold tuning became document-specific.\n\n**Attempt 2: Summarization for compression.** Running a smaller model to summarize chunks seems appealing. It adds a full model inference call (50-500ms), requires an API dependency, and summaries tend to drop entity specifics that matter for downstream model performance.\n\n**Attempt 3: Single-pass budget allocation.** Works fine for uniform documents. Fails badly on documents with mixed density: technical specs mixed with background narrative. The two-pass approach emerged from debugging these failures.\n\n```\npip install llmslim\n```\n\nThe benchmark scripts, raw JSON payloads, and full methodology are open. If you find issues with the numbers, open an issue - I'd genuinely like to know.", "url": "https://wpnews.pro/news/building-llmslim-architecture-deep-dive-into-deterministic-prompt-compression", "canonical_source": "https://dev.to/yashvardhan_thanvi_6762e7/building-llmslim-architecture-deep-dive-into-deterministic-prompt-compression-2ip0", "published_at": "2026-07-18 22:23:27+00:00", "updated_at": "2026-07-18 22:57:25.553496+00:00", "lang": "en", "topics": ["developer-tools", "large-language-models", "natural-language-processing", "ai-infrastructure"], "entities": ["LLMSlim", "LexRank", "TF-IDF"], "alternates": {"html": "https://wpnews.pro/news/building-llmslim-architecture-deep-dive-into-deterministic-prompt-compression", "markdown": "https://wpnews.pro/news/building-llmslim-architecture-deep-dive-into-deterministic-prompt-compression.md", "text": "https://wpnews.pro/news/building-llmslim-architecture-deep-dive-into-deterministic-prompt-compression.txt", "jsonld": "https://wpnews.pro/news/building-llmslim-architecture-deep-dive-into-deterministic-prompt-compression.jsonld"}}