{"slug": "same-agent-tasks-76-fewer-llm-calls-we-moved-semantic-cache-inside-the-graph", "title": "Same agent tasks, 76% fewer LLM calls – we moved semantic cache inside the graph", "summary": "ChorusGraph, a new open-source agent runtime, claims to reduce LLM calls by 76% by moving semantic caching inside the graph. The native BSP graph engine includes a two-stage semantic cache, swappable retrieval, auditable memory, and enterprise hardening, all installable via a single pip command. The project aims to simplify production LLM agent development by bundling orchestration, caching, retrieval, and observability into one runtime.", "body_md": "**Native agent runtime with semantic cache, swappable retrieval (PrismRAG), auditable memory, and enterprise hardening — one pip install, five plug-in ports.**\n\n```\npip install chorusgraph\nchorusgraph-demo\n```\n\n**Interactive demo (Product Hunt / launch):** [insightitsGit.github.io/ChorusGraph/demo.html](https://insightitsGit.github.io/ChorusGraph/demo.html) — click-through walkthrough, no API key for steps 1–3.\n\nChorusGraph= native engine + Prism stack ·LangGraph= optional baseline for A/B comparison only ([)]`docs/TERMINOLOGY.md`\n\nChorusGraph is **not** a LangGraph wrapper. It ships a **native BSP graph engine** (`chorusgraph.core.Graph`\n\n) with the Prism product stack attached by default: semantic cache, L2 retrieval, L3 memory, Route Ledger, checkpoints, and observability.\n\nYou define nodes, edges, and conditional routing on the native engine. Cache, retrieval, memory, and tools plug in through explicit ports on `ChorusStack`\n\n— swap Redis, vector RAG, or custom tool registries without rewriting orchestration.\n\n**ChorusGraph's own code has no LangGraph dependency on the product path.** The scheduler and all plug-in ports never import LangGraph. (Core dependency `prismlang`\n\nuses LangGraph internally for its own checkpointing utilities — it appears in `pip show`\n\n, but the ChorusGraph engine never calls it.) Install `chorusgraph[benchmark]`\n\nonly when running FL*/HL* comparison scenarios.\n\nBuilding production LLM agents usually means gluing six systems: orchestration, semantic cache, vector DB, reranker, checkpointing, and audit logs. ChorusGraph ships them as **one runtime** with explicit plug-in ports.\n\n| Pain | ChorusGraph answer |\n|---|---|\n| Repeat questions burn tokens | Two-stage semantic cache (coarse 64-d recall → full verify) |\n| RAG is another integration project | `RetrievalBackend` plug-in — keyword default, PrismRAG vector opt-in |\n| “Why did the agent say that?” | Route Ledger + `rule_chain` on every hop |\n| Orchestration + ops duct tape | Native scheduler, health endpoints, Docker/k8s packaging |\n| “Will this save us money?” | `chorusgraph-audit` — cold log simulation + pilot ledger reports |\n\n```\npip install chorusgraph\npython\nfrom chorusgraph import Graph, START, END, ChorusStack\nfrom chorusgraph.core.node import dict_node_adapter\n\nstack = ChorusStack.defaults(tenant_id=\"demo\")\n\ng = Graph(tenant_id=\"demo\", graph_id=\"hello\")\ng.add_node(\n    \"echo\",\n    dict_node_adapter(lambda s: {\"reply\": f\"Hello, {s.get('name', 'world')}\"}, hop=\"echo\"),\n)\ng.add_edge(START, \"echo\")\ng.add_edge(\"echo\", END)\n\nout = g.compile(stack=stack).invoke({\"name\": \"ChorusGraph\"})\nprint(out)  # {'reply': 'Hello, ChorusGraph'}\nchorusgraph-demo                              # routing + ledger (LLM-free)\nchorusgraph-finance-patterns                # ReAct / Plan-Solve / Reflection (needs GEMINI_API_KEY)\nchorusgraph-audit --log your_queries.jsonl    # simulated cache hit rate (no API key)\n```\n\n**Developer guide:** [ docs/DEVELOPER_GUIDE.md](/insightitsGit/ChorusGraph/blob/master/docs/DEVELOPER_GUIDE.md) — planning & reasoning, domain performance (finance vs healthcare), code examples.\n\nFull install guide: [ docs/INSTALL.md](/insightitsGit/ChorusGraph/blob/master/docs/INSTALL.md) · AI IDE prompts:\n\n`docs/AI_IDE_PROMPTS.md`\n\n| Feature | Description |\n|---|---|\nNative graph engine |\nBSP scheduler, envelope channels, conditional routing — no LangGraph on product paths |\nSemantic cache (L1) |\nTwo-stage gate: coarse recall → full verify; safe replay policies per domain |\nRetrieval (L2) |\nKeyword default; `PrismRAGRetrievalBackend` for vector + taxonomy (opt-in extra) |\nMemory (L3) |\nPrismCortex structured, replayable memory |\nRoute Ledger |\nPer-hop audit trail: cache hits, scores, durations, `rule_chain` |\nCheckpoints |\nSQLite default; Postgres enterprise persistence (license-gated) |\nTool registry |\nAllowlisted tools with sandbox; MCP-compatible patterns |\nResilience |\nTimeouts, retries, circuit breakers, graceful node failure |\nObservability |\nStructured JSON logs, OpenTelemetry traces, health/metrics endpoints |\nMulti-tenant guards |\nTenant isolation, resource limits, leakage tests |\nCold audit CLI |\n`chorusgraph-audit` — estimate cache savings from query logs (no LLM calls) |\nAgent patterns |\nReAct, Plan-Solve, Reflection via `chorusgraph.agents.Agent` — graph = plan |\nBenchmark matrix |\n8 scenarios (FL/FC/HL/HC) with fairness disclosure |\nDeploy packaging |\nDockerfile, docker-compose, k8s manifests |\n\nLangGraph alone |\nDIY stack (orchestrator + Redis + vector DB + reranker + logs) |\nChorusGraph |\n|\n|---|---|---|---|\n| Orchestration | ✅ StateGraph | You integrate | ✅ Native `Graph` |\n| Semantic cache | ❌ Roll your own | Separate service + glue | ✅ Built-in L1, swappable |\n| Retrieval / RAG | ❌ External | Chroma/Pinecone + custom code | ✅ `RetrievalBackend` port |\n| Audit / explainability | Limited | Custom logging | ✅ Route Ledger per hop |\n| Safe cache replay | Your problem | Your problem | ✅ Domain profiles (e.g. facts-only in healthcare) |\n| Benchmark proof | N/A | N/A | ✅ Published A/B vs LangGraph |\n| LangGraph dependency | Required | Optional | None on product path |\n\nChorusGraph **includes** LangGraph baselines (`benchmark/fl*`\n\n, `benchmark/hl*`\n\n) for fair apples-to-apples comparison — same model, tools, prompts, workload. Native scenarios (`benchmark/fc*`\n\n, `benchmark/hc*`\n\n) compile with `chorusgraph.core.Graph`\n\nonly.\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│  Your graph — nodes, edges, conditional routing              │\n├─────────────────────────────────────────────────────────────┤\n│  ChorusStack — swappable ports                               │\n│  ┌──────────┬──────────┬──────────┬──────────────────────┐  │\n│  │ Cache    │ Memory   │ Tools    │ Retrieval (L2)       │  │\n│  │ Prism /  │ Cortex   │ Registry │ Keyword / PrismRAG   │  │\n│  │ Redis    │          │          │                      │  │\n│  └──────────┴──────────┴──────────┴──────────────────────┘  │\n├─────────────────────────────────────────────────────────────┤\n│  Engine (fixed): BSP scheduler · envelopes · Resonance · JL  │\n├─────────────────────────────────────────────────────────────┤\n│  Route Ledger · checkpoints · tenant guards · observability  │\n└─────────────────────────────────────────────────────────────┘\n```\n\nDetails: [ docs/COMPOSE.md](/insightitsGit/ChorusGraph/blob/master/docs/COMPOSE.md) ·\n\n`docs/DEVELOPER_GUIDE.md`\n\nFour swappable ports on [ ChorusStack](/insightitsGit/ChorusGraph/blob/master/chorusgraph/compose/stack.py) — engine and scheduler stay fixed.\n\n| Port | Default | Swap examples | Method |\n|---|---|---|---|\nCache (`CacheBackend` ) |\n`PrismCacheBackend` |\n`RedisCacheBackend` |\n`with_cache()` |\nMemory (`MemoryBackend` ) |\n`CortexMemoryBackend` |\nDisable with `enable_memory=False` |\nstack field |\nTools (`ToolBackend` ) |\nFinance tool registry | Custom `ToolRegistry` , MCP |\n`resolve_tools()` |\nRetrieval (`RetrievalBackend` ) |\n`KeywordRetrievalBackend` |\n`PrismRAGRetrievalBackend` |\n`with_retrieval()` |\nPersistence (enterprise) |\n`SqlitePersistenceBackend` |\n`PostgresPersistenceBackend` |\nlicense-gated 5th port |\n\n``` python\nfrom chorusgraph.compose import ChorusStack, PrismRAGRetrievalBackend, RedisCacheBackend\nfrom chorusgraph.embedders import PrismlangOnnxEmbedder\n\nbackend = PrismRAGRetrievalBackend(\n    embedder=PrismlangOnnxEmbedder(),\n    mapping={\"categories\": [...], \"rules\": [...]},\n)\nbackend.index(your_corpus)\n\nstack = (\n    ChorusStack.defaults(tenant_id=\"acme\")\n    .with_retrieval(backend)\n    .with_cache(RedisCacheBackend(tenant_id=\"acme\", redis_url=\"redis://localhost:6379/0\"))\n)\n```\n\nFull plug-in guide: `docs/PLUGINS.md`\n\n| Layer | Component | Role |\n|---|---|---|\n| L0 — hop | PrismLang | 64-d state compression + `rule_chain` audit |\n| L1 — cache | PrismCache | Semantic gate, Resonance-scored recall |\n| L2 — knowledge | Retrieval plug-in | Keyword default · vector + taxonomy opt-in |\n| rerank | PrismResonance | Shared substrate rerank |\n| L3 — memory | PrismCortex | Structured, replayable memory |\n| transport | CHORUS / PrismAPI | Cross-node envelopes · federation hooks |\n\nChorusGraph is the **integration runtime** for the Prism family — PrismLang, PrismCache, PrismCortex, PrismRAG ship as defaults or opt-in extras, not separate science projects.\n\n[PrismGuard](https://pypi.org/project/prismguard/) ([0.1.4](https://pypi.org/project/prismguard/0.1.4/)) is a **separate package** — not a `ChorusStack`\n\nport. Install it alongside ChorusGraph when you want an auditable prompt-injection check before retrieve / LLM hops:\n\n```\npip install chorusgraph \"prismguard[prism,guard-model]==0.1.4\"\nprismguard-model download   # ~705 MB ONNX — not in the wheel; from GitHub Release v0.1.2\nfrom prismguard.integrations.chorusgraph import (\n    create_checker_from_env,\n    make_guard_handler,\n    route_after_guard,\n)\n\nchecker = create_checker_from_env()  # once per process\nguard = make_guard_handler(checker)\n# START → guard → [blocked → END | retrieve → …]\n# Wire with Graph.add_node(\"guard\", dict_node_adapter(guard, hop=\"guard\"))\n# Place guard before cache-gated hops so blocked prompts never seed cache\n```\n\n| Link | URL |\n|---|---|\n| PyPI |\n|\n\n[https://github.com/insightitsGit/PrismGuard](https://github.com/insightitsGit/PrismGuard)[https://github.com/insightitsGit/PrismGuard/blob/main/docs/integration-guide.md](https://github.com/insightitsGit/PrismGuard/blob/main/docs/integration-guide.md)[https://github.com/insightitsGit/PrismGuard/releases/tag/v0.1.2](https://github.com/insightitsGit/PrismGuard/releases/tag/v0.1.2)See also [ docs/PLUGINS.md](/insightitsGit/ChorusGraph/blob/master/docs/PLUGINS.md#companion-prismguard).\n\nFair A/B vs competent LangGraph baselines — same model, tools, prompts, workload.\n\n| Tier | Run ID | Tasks/scenario | Role |\n|---|---|---|---|\nMid (canonical) |\n`mid_20260708_111539` |\n100 | Primary regression + outreach proof |\nHeavy (scale) |\n`heavy_20260708_140300` |\n300 | Scale validation + whitepaper / diligence |\n| Smoke | `light_20260708_101409` |\n40 | CI / quick regression |\n\n**Start here:** [ docs/BENCHMARK_RESULTS.md](/insightitsGit/ChorusGraph/blob/master/docs/BENCHMARK_RESULTS.md) · archive index:\n\n[· machine pointer:](/insightitsGit/ChorusGraph/blob/master/benchmark/results/mvp_scenarios/README.md)\n\n`benchmark/results/mvp_scenarios/README.md`\n\n`benchmark/results/mvp_scenarios/latest.json`\n\nMethodology: [ benchmark/FAIRNESS_H9.md](/insightitsGit/ChorusGraph/blob/master/benchmark/FAIRNESS_H9.md) · consolidated tables:\n\n`benchmark/results/BENCHMARK_LATENCY_LLM_SUMMARY.md`\n\nJuly 2026 methodology fixes (benchmark-only — **no library release**): FL2 researcher prompt uses `annual_rate_pct`\n\n(matches tool schema); comparison script counts agent/tool errors in LangGraph success denominators. Supersedes pre-fix runs that inflated FL2 vs FC2. **Do not cite** invalid quota run `heavy_20260708_124337`\n\n.\n\n| Scenario | LangGraph | ChorusGraph | Delta |\n|---|---|---|---|\n| Finance single (FL1→FC1) | 87.0% | 98.0% |\n+11.0 pp |\n| Finance multi (FL2→FC2) | 87.0% | 94.0% |\n+7.0 pp |\n| Healthcare single (HL1→HC1) | 74.0% | 79.0% |\n+5.0 pp |\n| Healthcare multi (HL2→HC2) | 59.0% | 85.0% |\n+26 pp |\n\n| Scenario | LangGraph | ChorusGraph | Delta |\n|---|---|---|---|\n| Finance single (FL1→FC1) | 90.0% | 96.7% |\n+6.7 pp |\n| Finance multi (FL2→FC2) | 89.0% | 93.0% |\n+4.0 pp |\n| Healthcare single (HL1→HC1) | 73.7% | 84.0% |\n+10.3 pp |\n| Healthcare multi (HL2→HC2) | 62.3% | 77.3% |\n+15 pp |\n\n| Scenario | LLM calls (L → C) | Mean latency ms (L → C) | Cache hit (C) |\n|---|---|---|---|\n| FL1 / FC1 | 3.24 → 0.77 (−76%) |\n4760 → 1348 (−72%) |\n52% |\n| FL2 / FC2 | 2.03 → 0.69 (−66%) |\n3269 → 1085 (−67%) |\n40% |\n| HL1 / HC1 | 3.00 → 1.56 (−48%) |\n7036 → 3990 (−43%) |\n60% |\n| HL2 / HC2 | 3.82 → 3.15 (−18%) |\n10296 → 10753 (tie) | 51% |\n\n| Scenario | LLM calls (L → C) | Mean latency ms (L → C) | Cache hit (C) |\n|---|---|---|---|\n| FL1 / FC1 | 3.33 → 0.80 (−76%) |\n4972 → 1318 (−73%) |\n49.7% |\n| FL2 / FC2 | 2.04 → 0.75 (−63%) |\n3081 → 1335 (−57%) |\n34.7% |\n| HL1 / HC1 | 2.94 → 1.33 (−55%) |\n7105 → 3812 (−46%) |\n72.7% |\n| HL2 / HC2 | 3.85 → 2.67 (−31%) |\n10354 → 9537 (−8%; p95 tie) |\n79.0% |\n\nHealthcare multi saves fewer LLM calls by design (facts-only cache, judgment hops re-run). Lead with accuracy (+26 pp mid / +15 pp heavy), not cost; disclose HC2 p95 wall-clock tie.\n\nEach run ships a human report, run metadata, and a tarball of per-task JSONL traces.\n\n| Tier | Comparison report | Raw results (`results.tar.gz` ) |\nRun metadata |\n|---|---|---|---|\n| Light (40) |\n`light_20260708_101409/COMPARISON_REPORT.md` |\n\n`results.tar.gz`\n\n`run_meta.json`\n\n**Mid (100)**`mid_20260708_111539/COMPARISON_REPORT.md`\n\n`results.tar.gz`\n\n`run_meta.json`\n\n**Heavy (300)**`heavy_20260708_140300/COMPARISON_REPORT.md`\n\n`results.tar.gz`\n\n`run_meta.json`\n\nExtract raw traces: `tar -xzf results.tar.gz`\n\n— contains per-scenario `*.jsonl`\n\nand `comparison.json`\n\n.\n\n```\npip install -e \".[benchmark,gemini]\"\npython -m benchmark.run_scenarios --tier light --scenarios all   # needs GEMINI_API_KEY\nchorusgraph-audit --log tests/fixtures/audit_cold_queries.jsonl  # no API key\n```\n\n| Capability | Status |\n|---|---|\n| Native engine (no LangGraph on product path) | ✅ |\n| CI — 329+ tests, deterministic tier (no live keys) | ✅ |\n| Resilience, security, observability | ✅ |\n| Docker / k8s deploy | ✅\n`docs/DEPLOY.md` |\n\n`docs/API_1_0.md`\n\nReadiness scorecard: [ docs/ENTERPRISE_READINESS.md](/insightitsGit/ChorusGraph/blob/master/docs/ENTERPRISE_READINESS.md) · threat model:\n\n`docs/THREAT_MODEL.md`\n\n| Doc | Description |\n|---|---|\n`docs/INSTALL.md` |\n\n`docs/DEVELOPER_GUIDE.md`\n\n`Graph`\n\n`docs/PLUGINS.md`\n\n`docs/COMPOSE.md`\n\n`ChorusStack`\n\ncomposition patterns`docs/WHITEPAPER.md`\n\n`docs/BENCHMARK.md`\n\n`docs/BENCHMARK_RESULTS.md`\n\n`docs/CACHE_PROFILES.md`\n\n`docs/STABILITY.md`\n\n`docs/TERMINOLOGY.md`\n\n`benchmark/SCENARIOS.md`\n\n`docs/AI_IDE_PROMPTS.md`\n\n[Local RAG with Chroma + ChorusGraph](/insightitsGit/ChorusGraph/blob/master/chorusgraph/examples/chroma_local_rag)-- offline vector RAG with native retrieve-to-answer graph\n\n```\ngit clone https://github.com/insightitsGit/ChorusGraph.git\ncd ChorusGraph\npip install -e \".[dev,benchmark,gemini,retrieval]\"\npytest                    # deterministic tier — no API keys\npytest -m live            # live Gemini (needs GEMINI_API_KEY)\nruff check tests .github\n```\n\nContributing: [ CONTRIBUTING.md](/insightitsGit/ChorusGraph/blob/master/CONTRIBUTING.md) · workflow:\n\n[· process:](/insightitsGit/ChorusGraph/blob/master/docs/WORKFLOW.md)\n\n`docs/WORKFLOW.md`\n\n`docs/PROCESS.md`\n\n| Extra | Purpose |\n|---|---|\n`retrieval` |\nChroma + `PrismRAGRetrievalBackend` |\n`gemini` |\nLive Gemini examples |\n`cortex` |\nPrismCortex L3 memory |\n`benchmark` |\nLangGraph baselines (FL/HL) + chromadb |\n`benchmark-healthcare` |\nHealthcare benchmark scenarios (HC1/HC2) |\n`postgres` |\nPostgres DSN paths in deploy docs |\n`postgres-checkpoint` |\nLangGraph Postgres checkpointer (optional) |\n`langgraph` |\nBaselines / compat tests — not required for core product |\n`dev` |\npytest, ruff, mypy, coverage |\n`enterprise-ci` |\nFull CI matrix locally |\n\nLockfile: `requirements-lock.txt`\n\n· release notes: [ CHANGELOG.md](/insightitsGit/ChorusGraph/blob/master/CHANGELOG.md) ·\n\n`docs/RELEASE.md`\n\n**Shipped in 1.0:** native engine, semantic cache, retrieval plug-in, Route Ledger, SQLite persistence, benchmarks, deploy packaging, frozen public API.\n\n**Phase 2 (documented, in progress):**\n\n| Item | Status |\n|---|---|\n| Postgres-native Cortex GraphStore | 🟡 SQLite ships today |\nLedger token fields for live dollar reporting in `chorusgraph-audit --ledger` |\n🟡 schema sign-off pending |\n| CHORUS cipher external audit | TLS default; cipher opt-in |\n| Production Azure soak SLO sign-off | harness shipped |\n| External penetration test certification | pre-regulated-customer |\n| Prebuilt agent nodes (ReAct / supervisor) | roadmap primitive |\n\nDetails: [ docs/WHITEPAPER.md](/insightitsGit/ChorusGraph/blob/master/docs/WHITEPAPER.md) §9 ·\n\n`docs/ENTERPRISE_READINESS.md`\n\nApache-2.0 — see [LICENSE](/insightitsGit/ChorusGraph/blob/master/LICENSE).\n\n|\n\n[Code of conduct](/insightitsGit/ChorusGraph/blob/master/CODE_OF_CONDUCT.md)[Security policy](/insightitsGit/ChorusGraph/blob/master/SECURITY.md)Built by [Insight IT Solutions](https://github.com/insightitsGit). Dogfooded in production agent hubs. Part of the Prism family (PrismLang, PrismCache, PrismCortex, PrismRAG, [PrismGuard 0.1.4](https://pypi.org/project/prismguard/0.1.4/)).\n\n**Questions / enterprise:** open a [GitHub issue](https://github.com/insightitsGit/ChorusGraph/issues) or see [ docs/WHITEPAPER.md](/insightitsGit/ChorusGraph/blob/master/docs/WHITEPAPER.md) for commercial framing.", "url": "https://wpnews.pro/news/same-agent-tasks-76-fewer-llm-calls-we-moved-semantic-cache-inside-the-graph", "canonical_source": "https://github.com/insightitsGit/ChorusGraph/", "published_at": "2026-07-13 05:14:13+00:00", "updated_at": "2026-07-13 05:35:17.377753+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["ChorusGraph", "PrismRAG", "LangGraph", "PrismCortex", "SQLite", "Postgres"], "alternates": {"html": "https://wpnews.pro/news/same-agent-tasks-76-fewer-llm-calls-we-moved-semantic-cache-inside-the-graph", "markdown": "https://wpnews.pro/news/same-agent-tasks-76-fewer-llm-calls-we-moved-semantic-cache-inside-the-graph.md", "text": "https://wpnews.pro/news/same-agent-tasks-76-fewer-llm-calls-we-moved-semantic-cache-inside-the-graph.txt", "jsonld": "https://wpnews.pro/news/same-agent-tasks-76-fewer-llm-calls-we-moved-semantic-cache-inside-the-graph.jsonld"}}