ContextFusion: The Context Brain Your LLM Apps Are Missing

ContextFusion, an open-source middleware tool from developer rotsl, claims to reduce LLM token usage by 60–99% while maintaining answer quality, using a multi-objective knapsack optimizer and delta fusion for agent conversations. Benchmarks were run with Claude Sonnet 4.6 on production-like workloads, and the tool supports providers like OpenAI and Anthropic, with installation via npm or pip.

Normal users: Install context-portfolio-optimizer, run cpo compile ./your-docs --budget 4000, and stop overpaying for tokens. Developers: Middleware pipeline that ingests heterogeneous sources → normalizes → precomputes → optimizes via multi-objective knapsack → compiles provider-specific payloads with delta fusion for agents. Both groups get 60–99% token reduction with identical answer quality. You’re building with LLMs. Maybe it’s a chatbot over your company docs. Maybe it’s a coding assistant. Maybe it’s an agent that needs to remember context across 20 turns. You keep hitting the same frustrations: You’ve tried RAG. You’ve tried chunking. But you’re still blindly stuffing retrieved chunks into prompts without knowing which ones actually matter. Think of it like a smart travel packer for your LLM trips. You have a weight limit token budget . You have dozens of items documents, code, images . Some items are essential. Some are nice-to-have. Some are duplicates. Some are risky outdated, untrusted . ContextFusion: Benchmarks run with Claude Sonnet 4.6 on production-like workloads. Full methodology at github.com/rotsl/context-fusion/benchmarks One-time setupnpm install -g @rotsl/contextfusionnpx @rotsl/contextfusion setup Create API keys filenpx @rotsl/contextfusion env Edit .env with your OPENAI API KEY or ANTHROPIC API KEY Run optimizationnpx @rotsl/contextfusion run ./my-documents \ --query "Summarize key findings" \ --provider anthropic \ --model claude-sonnet-4-6 \ --budget 4000 Launch Web UInpx @rotsl/contextfusion ui --port 8080 pip install context-portfolio-optimizer Set up environmentcat .env << 'EOF'ANTHROPIC API KEY=your key hereOPENAI API KEY=your key hereEOF Run CLIcpo run ./my-documents --budget 4000 --query "What are the main points?" Or compile for specific task typecpo compile ./my-codebase \ --task "Explain this function" \ --provider openai \ --model gpt-5-mini \ --mode code \ --budget 3000 docker build -t context-fusion:latest .docker run --rm -it -v "$ pwd ":/app context-fusion:latest run ./data --budget 3000 Run cpo ui --port 8080 and open your browser. You'll see: This transparency is rare. Most RAG tools are black boxes. ContextFusion shows its work. ✅ Multi-provider setups — Same pipeline, different output formats ✅ Cost-sensitive production — 60–99% token reduction ✅ Agent conversations — Delta fusion prevents token churn ✅ Complex ingestion — PDFs, images, code, spreadsheets unified ✅ Latency requirements — Precomputation + caching ❌ Simple single-turn Q&A with tiny documents ❌ You’re already heavily invested in a specific RAG framework and happy with costs ❌ You need real-time streaming with sub-100ms latency ContextFusion adds 50–200ms optimization overhead ┌─────────────────────────────────────────────────────────────────┐│ INGESTION LAYER ││ PDF │ DOCX │ CSV │ JSON │ Images OCR │ Code │ Markdown │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ NORMALIZATION LAYER ││ Convert all sources to uniform ContextBlock objects ││ - source type, content hash, created at, metadata │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ REPRESENTATION LAYER ││ Precompute compact variants per block: ││ - universal summary general purpose ││ - qa extractive question-answering focused ││ - code signature functions, classes, dependencies ││ - agent condensed working memory format │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ PRECOMPUTE PIPELINE ││ Store: fingerprints, summaries, token stats, ││ retrieval features, compact variants in .cpo cache/ │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ RETRIEVAL LAYER ││ Query classification → Lexical retrieval top-100 ││ → Fast rerank top-20/25 → Candidate set │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ MULTI-OBJECTIVE PLANNER Core ││ ││ maximize Σ w u·utility - w r·risk - w t·token cost ││ - w l·latency + w c·cacheability + w d·diversity ││ ││ subject to: Σ token i ≤ budget ││ ││ Selects optimal representation variant per block │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ COMPRESSION LAYER ││ - JSON minification ││ - Citation compaction Source URI → id ││ - Schema field pruning ││ Levels: none │ light │ medium │ aggressive │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ DELTA FUSION Agent Mode ││ Compute ContextDelta: ││ - added blocks: new since last turn ││ - updated blocks: changed content ││ - removed blocks: no longer relevant ││ - unchanged block ids: reuse from cache │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ PROVIDER ADAPTER LAYER ││ Compile provider-specific payloads: ││ - openai: chat.completions format ││ - anthropic: messages with XML citations ││ - ollama: local API structure ││ - openai compatible: generic wrapper │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ CACHE-AWARE ASSEMBLY ││ Segment into: ││ - stable: system instructions, citation maps, cacheable blocks ││ - dynamic: volatile content, real-time data │└─────────────────────────────────────────────────────────────────┘ Most RAG tools use semantic similarity: embed query, embed chunks, return top-k. This fails when: ContextFusion’s planner treats this as a constrained optimization problem: Pseudocode of the core algorithmdef select context blocks candidates, budget, weights : """ candidates: List ContextBlock with multiple representation variants budget: int token limit weights: dict str, float utility, risk, latency, cacheability, diversity """ Generate all block, variant pairs with scores items = for block in candidates: for variant in block.representations: score = weights 'utility' variant.utility score - weights 'risk' block.risk score - weights 'token cost' variant.token count - weights 'latency' variant.latency estimate + weights 'cacheability' block.cache score + weights 'diversity' diversity bonus block, selected items.append block.id, variant, score, variant.token count Solve 0/1 knapsack for maximum score within budget selected = knapsack 01 items, budget return selected This is NP-hard, but with proper indexing and heuristics, it runs in <100ms for typical workloads. python from context portfolio optimizer import PipelineRunner, Configfrom context portfolio optimizer.providers import AnthropicAdapter Custom configurationconfig = Config.from yaml """budget: instructions: 1000 retrieval: 3000 memory: 2000 examples: 1500 tool trace: 1000 output reserve: 1000scoring: utility weights: retrieval: 0.25 trust: 0.20 freshness: 0.15 structure: 0.15 diversity: 0.15 token cost: -0.10provider: name: anthropic model: claude-sonnet-4-6""" Initialize pipelinerunner = PipelineRunner config=config Run full pipelineresult = runner.run sources= "./docs/architecture.pdf", "./src/api.py", "./data/metrics.csv" , query="How does the authentication flow work?", task mode="qa", chat | qa | code | agent budget=4000, use precomputed=True, compute delta=False Set True for agent loops Inspect resultsprint f"Selected {result 'stats' 'blocks selected' } blocks" print f"Total tokens: {result 'stats' 'total tokens' }" print f"Context preview:\n{result 'context' :500 }..." Direct provider compilationadapter = AnthropicAdapter config.provider payload = adapter.compile packet context blocks=result 'selected blocks' , task="Answer with citations", model="claude-sonnet-4-6" payload is ready for anthropic.messages.create payload Standard agent implementations re-send the entire conversation history + retrieved context on every turn. With 10 turns × 4,000 tokens = 40,000 tokens wasted. ContextFusion’s delta tracking: Turn 1: Full contextturn1 result = runner.run sources, query="Step 1...", task mode="agent" turn1 packet = turn1 result 'context packet' Turn 2: Only send what changedturn2 result = runner.run sources, query="Step 2...", task mode="agent", previous packet=turn1 packet, Enable delta computation compute delta=True turn2 result 'context delta' contains: { 'added blocks': new retrieved content , 'updated blocks': changed blocks , 'removed blocks': no longer relevant , 'unchanged block ids': ids to reuse from cache , 'full context hash': 'abc123...' For cache validation } The provider adapter assembles: For production workloads, precompute expensive operations: One-time setup can run offline, on CI, or scheduled cpo precompute ./corpus \ --store-dir .cpo cache/precompute \ --semantic-dedup \ --generate-all-representations Runtime query uses precomputed artifactscpo compile ./corpus \ --precomputed-only \ --query "Quick question" \ --budget 2000 Precomputed artifacts: Expose ContextFusion as an MCP Model Context Protocol server: cpo serve-mcp --host localhost --port 8765 MCP clients can now call: LangChain: python from context portfolio optimizer.integrations import ContextFusionRetrieverretriever = ContextFusionRetriever sources= "./docs" , budget=3000, task mode="qa" Use in any LangChain chainfrom langchain.chains import RetrievalQAqa = RetrievalQA.from chain type llm=chat model, chain type="stuff", retriever=retriever LlamaIndex: python from context portfolio optimizer.integrations import ContextFusionNodeParserparser = ContextFusionNodeParser budget per query=4000, precompute dir=".cpo cache" Use with LlamaIndex index constructionfrom llama index.core import VectorStoreIndexindex = VectorStoreIndex.from documents documents, node parser=parser git clone https://github.com/rotsl/context-fusion.gitcd context-fusionmake bootstrap Install dev dependencies Development workflowmake test Run test suite 49 tests make lint Ruff + mypymake type-check Strict type checkingmake format Auto-format code Local serversmake ui Web UI on :8080make serve-mcp MCP server on :8765 Benchmarkingmake benchmark Run full benchmark suite Custom representation: python from context portfolio optimizer.representations import Representation, register representation@register representation "my custom" class MyCustomRepresentation Representation : def generate self, block: ContextBlock - str: Your custom summarization logic return custom summarize block.content def estimate tokens self, text: str - int: return len text.split 1.3 Rough heuristic Custom provider adapter: python from context portfolio optimizer.providers import BaseProviderAdapter, register adapter@register adapter "my provider" class MyProviderAdapter BaseProviderAdapter : def compile packet self, context blocks, task, model, kwargs : Format for your custom LLM API return { "model": model, "messages": {"role": "system", "content": self.format system }, {"role": "user", "content": self.format context context blocks, task } } Q: How is this different from LangChain’s ContextualCompressionRetriever? LangChain’s version compresses after retrieval using an LLM call. ContextFusion optimizes which content to retrieve and which representation to use, without requiring an LLM for compression. It’s also provider-agnostic and handles delta fusion for agents. Q: Does this replace my vector database? No. ContextFusion sits after retrieval. Use Pinecone, Weaviate, pgvector, or FAISS for initial retrieval — then pass candidates through ContextFusion for optimization. Q: What about streaming responses? ContextFusion optimizes the input context. Streaming the LLM’s output is unaffected. The optimization adds 50–200ms overhead, which is usually offset by reduced LLM latency from shorter prompts. Q: Can I use this with local models? Yes. The Ollama adapter works with any OpenAI-compatible local server. Budget planning and compression are even more valuable with slower local hardware. Q: How do I debug suboptimal context selection? Run cpo ui and inspect the "Selected Blocks" panel. Each block shows its utility score, risk score, token count, and why it was included/excluded. Run cpo ablate ./data to see which blocks contribute most to answer quality. ContextFusion isn’t just another RAG tool. It’s a bet that context optimization — treating token budgets as scarce resources to be allocated intelligently — will become as essential as retrieval itself. For normal users: Install it, run it, pay less. For developers: Extend it, integrate it, build smarter systems. Fuse less context. Keep more signal. Ship faster answers. ⭐️ Star the repo, ⚠️file issues, ㊣ submit PRs. ContextFusion is Apache-2.0 and built for production. ContextFusion: The Context Brain Your LLM Apps Are Missing https://pub.towardsai.net/contextfusion-the-context-brain-your-llm-apps-are-missing-c2fd9e632b99 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.