Show HN: MemStitch – Zero-copy context bridging for vLLM (25x TTFT speedup)

MemStitch, a zero-copy context bridging gateway for vLLM, achieves a 25x speedup in time-to-first-token by allowing multiple GPU agents to share KV cache blocks instead of repeating the prefill phase. The system uses topological hashing and secure memory stitching to reduce latency from 1200ms to 48ms and cut GPU memory usage by 43.4% in multi-agent workflows.

Zero-Copy Context Bridging Gateway for Multi-Agent GPU Inference. In multi-agent collaborative workflows, separate agents often process the same long text context sequentially. For example: Agent A Legal Auditor : Reads a 200-page contract and runs compliance analyses populating the GPU KV Cache . Agent B Financial Compliance : Reads the same 200-page contract and audits financial liabilities. Under standard inference engines, Agent B is forced to repeat the expensive prefill phase , duplicate GPU activations, and suffer from high Time-to-First-Token TTFT latency. Context-Stitcher solves this by bridging caches at the memory level: Context Topological Hashing : Segmenting prompts into physical block-sizes and mapping them to cryptographic fingerprints Merkle-chains . Zero-Copy Block Stitching : Bypassing prefill for matched prefixes by mapping the logical attention table of Agent B directly to the physical GPU memory address of Agent A's cache blocks. Zero-Trust Secure Gate : Enforcing boundary control lists so unauthorized agent sessions cannot access shared physical blocks. Below is the benchmark analysis of Context-Stitcher compared to standard vLLM cold-prefills when executing consecutive agents over a shared 200-page document: ⚡ TTFT Prefill Latency Agent B Response Time — Lower is better Baseline vLLM Cold : ██████████████████████████████ 1200 ms Context-Stitcher: █ 48 ms 25.0x Prefill Speedup 🚀 💾 GPU Physical Cache Blocks Allocated Total VRAM — Lower is better Baseline vLLM Cold : ██████████████████████████████ 53 blocks No sharing Context-Stitcher: ████████████████ 30 blocks 43.4% Memory Saved 📉 pip install -r requirements.txt python run.py Once booted, the gateway routes are active on http://localhost:8000 : API Proxy Gateway : http://localhost:8000/v1/chat/completions Real-time Developer Console : Open http://localhost:8000 in your web browser. Context-Stitcher includes a responsive developer portal to monitor physical cache block states idle, private allocations, shared/stitched pages, security alarms in real time. Context-Stitcher supports Python SDK Decorators and OpenAI-Compatible REST APIs for cross-application integrations: If your agent pipelines are written in Python, you can utilize the StitcherMesh and @stitch agent decorators to link context memory: python from context stitcher import StitcherMesh, stitch agent 1. Initialize the pointer-sharing inference control mesh ref: vLLM mesh = StitcherMesh backend="vllm", model="meta-llama/Llama-3.1-8B-Instruct" Configure secure gate policy: allow Agent B to read Agent A's KV Cache blocks mesh.sg.add policy "agent a", "agent b" 2. Decorate your agent workflows @stitch agent mesh def agent a : prompt = " Long Document Context... \nAnalyze the intellectual property clauses." First execution: Prefills the cache and registers topological hashes res = mesh.generate prompt=prompt, fingerprint="legal doc v1" return "legal doc v1" @stitch agent mesh def agent b context fingerprint : prompt = " Long Document Context... \nEvaluate the financial compliance risk." Subsequent execution: Automatically stitches blocks, bypassing prefill phase res = mesh.generate prompt=prompt, fingerprint=context fingerprint print f"Agent B Response: {res 'generated text' }" print f"Time Saved: {res 'prefill time saved ms' }ms" The gateway exposes standard OpenAI endpoints. Point your LLM client base URL to Context-Stitcher to activate sharing. python from openai import OpenAI Direct client to the Context-Stitcher proxy gateway client = OpenAI base url="http://localhost:8000/v1", api key="not-needed" Send standard completions request with agent id metadata inside extra body response = client.chat.completions.create model="context-stitcher-sim", messages= {"role": "user", "content": " Long Document Context... \nEvaluate compliance risk."} , extra body={ "agent id": "AgentB", Identifies the requesting Agent "session id": "session legal audit" Identifies the shared session cache } print "Generated Output:", response.choices 0 .message.content curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "context-stitcher-sim", "messages": {"role": "user", "content": " Long Document Context... \nEvaluate compliance risk."} , "agent id": "AgentB", "session id": "session legal audit" }' You can inspect, add, or revoke access authorization rules dynamically between agents. curl -X GET http://localhost:8000/policies curl -X POST http://localhost:8000/policy \ -H "Content-Type: application/json" \ -d '{"owner agent": "AgentA", "allowed reader": "AgentB"}' curl -X DELETE http://localhost:8000/policy \ -H "Content-Type: application/json" \ -d '{"owner agent": "AgentA", "allowed reader": "AgentB"}'