{"slug": "show-hn-memstitch-zero-copy-context-bridging-for-vllm-25x-ttft-speedup", "title": "Show HN: MemStitch – Zero-copy context bridging for vLLM (25x TTFT speedup)", "summary": "MemStitch, a zero-copy context bridging gateway for vLLM, achieves a 25x speedup in time-to-first-token by allowing multiple GPU agents to share KV cache blocks instead of repeating the prefill phase. The system uses topological hashing and secure memory stitching to reduce latency from 1200ms to 48ms and cut GPU memory usage by 43.4% in multi-agent workflows.", "body_md": "**Zero-Copy Context Bridging Gateway for Multi-Agent GPU Inference.**\n\nIn multi-agent collaborative workflows, separate agents often process the same long text context sequentially. For example:\n\n**Agent A (Legal Auditor)**: Reads a 200-page contract and runs compliance analyses (populating the GPU KV Cache).** Agent B (Financial Compliance)**: Reads the same 200-page contract and audits financial liabilities.\n\nUnder standard inference engines, **Agent B is forced to repeat the expensive prefill phase**, duplicate GPU activations, and suffer from high Time-to-First-Token (TTFT) latency.\n\n**Context-Stitcher** solves this by bridging caches at the memory level:\n\n**Context Topological Hashing**: Segmenting prompts into physical block-sizes and mapping them to cryptographic fingerprints (Merkle-chains).** Zero-Copy Block Stitching**: Bypassing prefill for matched prefixes by mapping the logical attention table of Agent B directly to the physical GPU memory address of Agent A's cache blocks.**Zero-Trust Secure Gate**: Enforcing boundary control lists so unauthorized agent sessions cannot access shared physical blocks.\n\nBelow is the benchmark analysis of Context-Stitcher compared to standard vLLM cold-prefills when executing consecutive agents over a shared 200-page document:\n\n```\n⚡ TTFT Prefill Latency (Agent B Response Time) — Lower is better\nBaseline (vLLM Cold):   ██████████████████████████████  1200 ms\nContext-Stitcher:       █  48 ms ( 25.0x Prefill Speedup! 🚀 )\n\n💾 GPU Physical Cache Blocks Allocated (Total VRAM) — Lower is better\nBaseline (vLLM Cold):   ██████████████████████████████  53 blocks (No sharing)\nContext-Stitcher:       ████████████████  30 blocks ( 43.4% Memory Saved! 📉 )\npip install -r requirements.txt\npython run.py\n```\n\nOnce booted, the gateway routes are active on `http://localhost:8000`\n\n:\n\n**API Proxy Gateway**:`http://localhost:8000/v1/chat/completions`\n\n**Real-time Developer Console**: Open`http://localhost:8000`\n\nin your web browser.\n\nContext-Stitcher includes a responsive developer portal to monitor physical cache block states (idle, private allocations, shared/stitched pages, security alarms) in real time.\n\nContext-Stitcher supports **Python SDK Decorators** and **OpenAI-Compatible REST APIs** for cross-application integrations:\n\nIf your agent pipelines are written in Python, you can utilize the `StitcherMesh`\n\nand `@stitch_agent`\n\ndecorators to link context memory:\n\n``` python\nfrom context_stitcher import StitcherMesh, stitch_agent\n\n# 1. Initialize the pointer-sharing inference control mesh (ref: vLLM)\nmesh = StitcherMesh(backend=\"vllm\", model=\"meta-llama/Llama-3.1-8B-Instruct\")\n\n# Configure secure gate policy: allow Agent B to read Agent A's KV Cache blocks\nmesh.sg.add_policy(\"agent_a\", \"agent_b\")\n\n# 2. Decorate your agent workflows\n@stitch_agent(mesh)\ndef agent_a():\n    prompt = \"[Long Document Context...]\\nAnalyze the intellectual property clauses.\"\n    # First execution: Prefills the cache and registers topological hashes\n    res = mesh.generate(prompt=prompt, fingerprint=\"legal_doc_v1\")\n    return \"legal_doc_v1\"\n\n@stitch_agent(mesh)\ndef agent_b(context_fingerprint):\n    prompt = \"[Long Document Context...]\\nEvaluate the financial compliance risk.\"\n    # Subsequent execution: Automatically stitches blocks, bypassing prefill phase\n    res = mesh.generate(prompt=prompt, fingerprint=context_fingerprint)\n    print(f\"Agent B Response: {res['generated_text']}\")\n    print(f\"Time Saved: {res['prefill_time_saved_ms']}ms\")\n```\n\nThe gateway exposes standard OpenAI endpoints. Point your LLM client base URL to Context-Stitcher to activate sharing.\n\n``` python\nfrom openai import OpenAI\n\n# Direct client to the Context-Stitcher proxy gateway\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"not-needed\")\n\n# Send standard completions request with agent_id metadata inside extra_body\nresponse = client.chat.completions.create(\n    model=\"context-stitcher-sim\",\n    messages=[\n        {\"role\": \"user\", \"content\": \"[Long Document Context...]\\nEvaluate compliance risk.\"}\n    ],\n    extra_body={\n        \"agent_id\": \"AgentB\",                # Identifies the requesting Agent\n        \"session_id\": \"session_legal_audit\"  # Identifies the shared session cache\n    }\n)\n\nprint(\"Generated Output:\", response.choices[0].message.content)\ncurl -X POST http://localhost:8000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"context-stitcher-sim\",\n    \"messages\": [\n      {\"role\": \"user\", \"content\": \"[Long Document Context...]\\nEvaluate compliance risk.\"}\n    ],\n    \"agent_id\": \"AgentB\",\n    \"session_id\": \"session_legal_audit\"\n  }'\n```\n\nYou can inspect, add, or revoke access authorization rules dynamically between agents.\n\n```\ncurl -X GET http://localhost:8000/policies\ncurl -X POST http://localhost:8000/policy \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"owner_agent\": \"AgentA\", \"allowed_reader\": \"AgentB\"}'\ncurl -X DELETE http://localhost:8000/policy \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"owner_agent\": \"AgentA\", \"allowed_reader\": \"AgentB\"}'\n```\n\n", "url": "https://wpnews.pro/news/show-hn-memstitch-zero-copy-context-bridging-for-vllm-25x-ttft-speedup", "canonical_source": "https://github.com/DaqulaLin/MemStitch", "published_at": "2026-07-14 01:04:02+00:00", "updated_at": "2026-07-14 01:18:35.568897+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "large-language-models", "ai-tools", "developer-tools"], "entities": ["MemStitch", "vLLM", "Context-Stitcher", "StitcherMesh", "meta-llama/Llama-3.1-8B-Instruct"], "alternates": {"html": "https://wpnews.pro/news/show-hn-memstitch-zero-copy-context-bridging-for-vllm-25x-ttft-speedup", "markdown": "https://wpnews.pro/news/show-hn-memstitch-zero-copy-context-bridging-for-vllm-25x-ttft-speedup.md", "text": "https://wpnews.pro/news/show-hn-memstitch-zero-copy-context-bridging-for-vllm-25x-ttft-speedup.txt", "jsonld": "https://wpnews.pro/news/show-hn-memstitch-zero-copy-context-bridging-for-vllm-25x-ttft-speedup.jsonld"}}