Show HN: MemStitch – Zero-copy context bridging for vLLM (25x TTFT speedup)

Zero-Copy Context Bridging Gateway for Multi-Agent GPU Inference.

In multi-agent collaborative workflows, separate agents often process the same long text context sequentially. For example:

Agent A (Legal Auditor): Reads a 200-page contract and runs compliance analyses (populating the GPU KV Cache).** Agent B (Financial Compliance)**: Reads the same 200-page contract and audits financial liabilities.

Under standard inference engines, Agent B is forced to repeat the expensive prefill phase, duplicate GPU activations, and suffer from high Time-to-First-Token (TTFT) latency.

Context-Stitcher solves this by bridging caches at the memory level:

Context Topological Hashing: Segmenting prompts into physical block-sizes and mapping them to cryptographic fingerprints (Merkle-chains).** Zero-Copy Block Stitching**: Bypassing prefill for matched prefixes by mapping the logical attention table of Agent B directly to the physical GPU memory address of Agent A's cache blocks.Zero-Trust Secure Gate: Enforcing boundary control lists so unauthorized agent sessions cannot access shared physical blocks.

Below is the benchmark analysis of Context-Stitcher compared to standard vLLM cold-prefills when executing consecutive agents over a shared 200-page document:

⚡ TTFT Prefill Latency (Agent B Response Time) — Lower is better
Baseline (vLLM Cold):   ██████████████████████████████  1200 ms
Context-Stitcher:       █  48 ms ( 25.0x Prefill Speedup! 🚀 )

💾 GPU Physical Cache Blocks Allocated (Total VRAM) — Lower is better
Baseline (vLLM Cold):   ██████████████████████████████  53 blocks (No sharing)
Context-Stitcher:       ████████████████  30 blocks ( 43.4% Memory Saved! 📉 )
pip install -r requirements.txt
python run.py

Once booted, the gateway routes are active on http://localhost:8000

:

API Proxy Gateway:http://localhost:8000/v1/chat/completions

Real-time Developer Console: Openhttp://localhost:8000

in your web browser.

Context-Stitcher includes a responsive developer portal to monitor physical cache block states (idle, private allocations, shared/stitched pages, security alarms) in real time.

Context-Stitcher supports Python SDK Decorators and OpenAI-Compatible REST APIs for cross-application integrations:

If your agent pipelines are written in Python, you can utilize the StitcherMesh

and @stitch_agent

decorators to link context memory:

from context_stitcher import StitcherMesh, stitch_agent

mesh = StitcherMesh(backend="vllm", model="meta-llama/Llama-3.1-8B-Instruct")

mesh.sg.add_policy("agent_a", "agent_b")

@stitch_agent(mesh)
def agent_a():
    prompt = "[Long Document Context...]\nAnalyze the intellectual property clauses."
    res = mesh.generate(prompt=prompt, fingerprint="legal_doc_v1")
    return "legal_doc_v1"

@stitch_agent(mesh)
def agent_b(context_fingerprint):
    prompt = "[Long Document Context...]\nEvaluate the financial compliance risk."
    res = mesh.generate(prompt=prompt, fingerprint=context_fingerprint)
    print(f"Agent B Response: {res['generated_text']}")
    print(f"Time Saved: {res['prefill_time_saved_ms']}ms")

The gateway exposes standard OpenAI endpoints. Point your LLM client base URL to Context-Stitcher to activate sharing.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="context-stitcher-sim",
    messages=[
        {"role": "user", "content": "[Long Document Context...]\nEvaluate compliance risk."}
    ],
    extra_body={
        "agent_id": "AgentB",                # Identifies the requesting Agent
        "session_id": "session_legal_audit"  # Identifies the shared session cache
    }
)

print("Generated Output:", response.choices[0].message.content)
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "context-stitcher-sim",
    "messages": [
      {"role": "user", "content": "[Long Document Context...]\nEvaluate compliance risk."}
    ],
    "agent_id": "AgentB",
    "session_id": "session_legal_audit"
  }'

You can inspect, add, or revoke access authorization rules dynamically between agents.

curl -X GET http://localhost:8000/policies
curl -X POST http://localhost:8000/policy \
  -H "Content-Type: application/json" \
  -d '{"owner_agent": "AgentA", "allowed_reader": "AgentB"}'
curl -X DELETE http://localhost:8000/policy \
  -H "Content-Type: application/json" \
  -d '{"owner_agent": "AgentA", "allowed_reader": "AgentB"}'