Normal users: Install context-portfolio-optimizer, run cpo compile ./your-docs --budget 4000, and stop overpaying for tokens.
Developers: Middleware pipeline that ingests heterogeneous sources → normalizes → precomputes → optimizes via multi-objective knapsack → compiles provider-specific payloads with delta fusion for agents.
Both groups get 60–99% token reduction with identical answer quality.
You’re building with LLMs. Maybe it’s a chatbot over your company docs. Maybe it’s a coding assistant. Maybe it’s an agent that needs to remember context across 20 turns.
You keep hitting the same frustrations:
You’ve tried RAG. You’ve tried chunking. But you’re still blindly stuffing retrieved chunks into prompts without knowing which ones actually matter.
Think of it like a smart travel packer for your LLM trips.
You have a weight limit (token budget). You have dozens of items (documents, code, images). Some items are essential. Some are nice-to-have. Some are duplicates. Some are risky (outdated, untrusted).
ContextFusion:
*Benchmarks run with Claude Sonnet 4.6 on production-like workloads. Full methodology at *github.com/rotsl/context-fusion/benchmarks
pip install context-portfolio-optimizer# Set up environmentcat > .env << 'EOF'ANTHROPIC_API_KEY=your_key_hereOPENAI_API_KEY=your_key_hereEOF# Run CLIcpo run ./my-documents --budget 4000 --query "What are the main points?"# Or compile for specific task typecpo compile ./my-codebase \ --task "Explain this function" \ --provider openai \ --model gpt-5-mini \ --mode code \ --budget 3000
docker build -t context-fusion:latest .docker run --rm -it -v "$(pwd)":/app context-fusion:latest run ./data --budget 3000
Run cpo ui --port 8080 and open your browser. You'll see:
This transparency is rare. Most RAG tools are black boxes. ContextFusion shows its work.
✅ Multi-provider setups — Same pipeline, different output formats
✅ Cost-sensitive production — 60–99% token reduction
✅ Agent conversations — Delta fusion prevents token churn
✅ Complex ingestion — PDFs, images, code, spreadsheets unified
✅ Latency requirements — Precomputation + caching
❌ Simple single-turn Q&A with tiny documents
❌ You’re already heavily invested in a specific RAG framework and happy with costs
❌ You need real-time streaming with sub-100ms latency (ContextFusion adds 50–200ms optimization overhead)
┌─────────────────────────────────────────────────────────────────┐│ INGESTION LAYER ││ PDF │ DOCX │ CSV │ JSON │ Images (OCR) │ Code │ Markdown │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ NORMALIZATION LAYER ││ Convert all sources to uniform ContextBlock objects ││ - source_type, content_hash, created_at, metadata │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ REPRESENTATION LAYER ││ Precompute compact variants per block: ││ - universal_summary (general purpose) ││ - qa_extractive (question-answering focused) ││ - code_signature (functions, classes, dependencies) ││ - agent_condensed (working memory format) │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ PRECOMPUTE PIPELINE ││ Store: fingerprints, summaries, token stats, ││ retrieval features, compact variants in .cpo_cache/ │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ RETRIEVAL LAYER ││ Query classification → Lexical retrieval (top-100) ││ → Fast rerank (top-20/25) → Candidate set │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ MULTI-OBJECTIVE PLANNER (Core) ││ ││ maximize Σ( w_u·utility - w_r·risk - w_t·token_cost ││ - w_l·latency + w_c·cacheability + w_d·diversity ) ││ ││ subject to: Σ(token_i) ≤ budget ││ ││ Selects optimal representation variant per block │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ COMPRESSION LAYER ││ - JSON minification ││ - Citation compaction (Source URI → [id]) ││ - Schema field pruning ││ Levels: none │ light │ medium │ aggressive │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ DELTA FUSION (Agent Mode) ││ Compute ContextDelta: ││ - added_blocks: new since last turn ││ - updated_blocks: changed content ││ - removed_blocks: no longer relevant ││ - unchanged_block_ids: reuse from cache │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ PROVIDER ADAPTER LAYER ││ Compile provider-specific payloads: ││ - openai: chat.completions format ││ - anthropic: messages with XML citations ││ - ollama: local API structure ││ - openai_compatible: generic wrapper │└─────────────────────────────────────────────────────────────────┘ ↓┌─────────────────────────────────────────────────────────────────┐│ CACHE-AWARE ASSEMBLY ││ Segment into: ││ - stable: system instructions, citation maps, cacheable blocks ││ - dynamic: volatile content, real-time data │└─────────────────────────────────────────────────────────────────┘
Most RAG tools use semantic similarity: embed query, embed chunks, return top-k. This fails when:
ContextFusion’s planner treats this as a constrained optimization problem:
This is NP-hard, but with proper indexing and heuristics, it runs in <100ms for typical workloads.
from context_portfolio_optimizer import PipelineRunner, Configfrom context_portfolio_optimizer.providers import AnthropicAdapter# Custom configurationconfig = Config.from_yaml("""budget: instructions: 1000 retrieval: 3000 memory: 2000 examples: 1500 tool_trace: 1000 output_reserve: 1000scoring: utility_weights: retrieval: 0.25 trust: 0.20 freshness: 0.15 structure: 0.15 diversity: 0.15 token_cost: -0.10provider: name: anthropic model: claude-sonnet-4-6""")# Initialize pipelinerunner = PipelineRunner(config=config)# Run full pipelineresult = runner.run( sources=["./docs/architecture.pdf", "./src/api.py", "./data/metrics.csv"], query="How does the authentication flow work?", task_mode="qa", # chat | qa | code | agent budget=4000, use_precomputed=True, compute_delta=False # Set True for agent loops)# Inspect resultsprint(f"Selected {result['stats']['blocks_selected']} blocks")print(f"Total tokens: {result['stats']['total_tokens']}")print(f"Context preview:\n{result['context'][:500]}...")# Direct provider compilationadapter = AnthropicAdapter(config.provider)payload = adapter.compile_packet( context_blocks=result['selected_blocks'], task="Answer with citations", model="claude-sonnet-4-6")# payload is ready for anthropic.messages.create(**payload)
Standard agent implementations re-send the entire conversation history + retrieved context on every turn. With 10 turns × 4,000 tokens = 40,000 tokens wasted.
ContextFusion’s delta tracking:
The provider adapter assembles:
For production workloads, precompute expensive operations:
Precomputed artifacts:
Expose ContextFusion as an MCP (Model Context Protocol) server:
cpo serve-mcp --host localhost --port 8765
MCP clients can now call:
LangChain:
from context_portfolio_optimizer.integrations import ContextFusionRetrieverretriever = ContextFusionRetriever( sources=["./docs"], budget=3000, task_mode="qa")# Use in any LangChain chainfrom langchain.chains import RetrievalQAqa = RetrievalQA.from_chain_type( llm=chat_model, chain_type="stuff", retriever=retriever)
LlamaIndex:
from context_portfolio_optimizer.integrations import ContextFusionNodeParserparser = ContextFusionNodeParser( budget_per_query=4000, precompute_dir=".cpo_cache")# Use with LlamaIndex index constructionfrom llama_index.core import VectorStoreIndexindex = VectorStoreIndex.from_documents( documents, node_parser=parser)
git clone https://github.com/rotsl/context-fusion.gitcd context-fusionmake bootstrap # Install dev dependencies# Development workflowmake test # Run test suite (49 tests)make lint # Ruff + mypymake type-check # Strict type checkingmake format # Auto-format code# Local serversmake ui # Web UI on :8080make serve-mcp # MCP server on :8765# Benchmarkingmake benchmark # Run full benchmark suite
Custom representation:
from context_portfolio_optimizer.representations import Representation, register_representation@register_representation("my_custom")class MyCustomRepresentation(Representation): def generate(self, block: ContextBlock) -> str: # Your custom summarization logic return custom_summarize(block.content) def estimate_tokens(self, text: str) -> int: return len(text.split()) * 1.3 # Rough heuristic
Custom provider adapter:
from context_portfolio_optimizer.providers import BaseProviderAdapter, register_adapter@register_adapter("my_provider")class MyProviderAdapter(BaseProviderAdapter): def compile_packet(self, context_blocks, task, model, **kwargs): # Format for your custom LLM API return { "model": model, "messages": [ {"role": "system", "content": self.format_system()}, {"role": "user", "content": self.format_context(context_blocks, task)} ] }
*Q: How is this different from LangChain’s *ContextualCompressionRetriever?
LangChain’s version compresses after retrieval using an LLM call. ContextFusion optimizes which content to retrieve and which representation to use, without requiring an LLM for compression. It’s also provider-agnostic and handles delta fusion for agents.
Q: Does this replace my vector database?
No. ContextFusion sits after retrieval. Use Pinecone, Weaviate, pgvector, or FAISS for initial retrieval — then pass candidates through ContextFusion for optimization.
Q: What about streaming responses?
ContextFusion optimizes the input context. Streaming the LLM’s output is unaffected. The optimization adds 50–200ms overhead, which is usually offset by reduced LLM latency from shorter prompts.
Q: Can I use this with local models?
Yes. The Ollama adapter works with any OpenAI-compatible local server. Budget planning and compression are even more valuable with slower local hardware.
Q: How do I debug suboptimal context selection?
*Run **cpo ui and inspect the "Selected Blocks" panel. Each block shows its utility score, risk score, token count, and why it was included/excluded. Run *cpo ablate ./data to see which blocks contribute most to answer quality.
ContextFusion isn’t just another RAG tool. It’s a bet that context optimization — treating token budgets as scarce resources to be allocated intelligently — will become as essential as retrieval itself.
For normal users: Install it, run it, pay less.
For developers: Extend it, integrate it, build smarter systems.
Fuse less context. Keep more signal. Ship faster answers.
⭐️ Star the repo, ⚠️file issues, ㊣ submit PRs. ContextFusion is Apache-2.0 and built for production.
ContextFusion: The Context Brain Your LLM Apps Are Missing was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.