{"slug": "show-hn-spanda-sub-microsecond-llm-epistemic-uncertainty-in-rust", "title": "Show HN: Spanda – Sub-microsecond LLM epistemic uncertainty in Rust", "summary": "A Rust-based tool called Spanda computes LLM epistemic uncertainty in 652.1 nanoseconds (0.65 µs) per evaluation, roughly 90,000× faster than neural Semantic Entropy methods that add about 92.4 ms of GPU overhead per inference call, according to the project's Show HN release. Spanda's Exact-Match Normalized Entropy (R_sc) is a zero-parameter, zero-GPU metric that matches or exceeds DeBERTa-v3 NLI cross-encoder Semantic Entropy on structured reasoning across models from 1.5B to 120B parameters, the release states. The release also reports that at 120B scale on ungrounded factual recall (TriviaQA) the model exhibits \"Confident Mode Collapse,\" producing the same incorrect answer across samples for an inverted AUROC of 0.091, which the project says makes external grounding (RAG) mandatory in that regime.", "body_md": "*Detect LLM hallucinations and quantify uncertainty in microseconds without secondary NLI cross-encoders.*\n\nTraditional epistemic uncertainty estimation in LLMs relies on **Semantic Entropy (SE)** ([Kuhn et al., 2023](https://arxiv.org/abs/2302.09664); [Farquhar et al., Nature 2024](https://www.nature.com/articles/s41586-024-07421-0)). While effective, Semantic Entropy requires clustering \n\nThis introduces two severe production bottlenecks:\n\n1. \n**Quadratic Cost:**$\\binom{K}{2}$ forward passes per query (45 neural evaluations for$K=10$ ).\n2. \n**Serving Latency:** Adds $\\sim$90 ms of GPU overhead per inference call, making it unusable for high-throughput production serving.\n\n**Spanda** introduces **Exact-Match Normalized Entropy ( $R_{sc}$)**: a zero-parameter, zero-GPU metric that computes uncertainty directly over deterministic lexical clusters.\n\nAcross empirical evaluations spanning two orders of magnitude (**1.5B to 120B parameters**), Spanda matches or exceeds neural Semantic Entropy on structured reasoning while operating **~90,000$\\times$ faster** (\n\nAs model capacity increases from 1.5B to 27B parameters, internal reasoning coherence causes correct predictions to naturally converge to identical lexical sequences. On mathematical reasoning (**GSM8K**), exact-match AUROC scales monotonically:\n\nAt **7B+ parameters**, Spanda achieves the exact same discriminative power as heavy DeBERTa-v3 NLI cross-encoders, rendering the neural clustering step redundant for reasoning.\n\nAt the **120B frontier scale** on ungrounded factual recall (TriviaQA), the model exhibits **Confident Mode Collapse**: its parametric memory and RLHF tuning cause it to hallucinate the *exact same incorrect answer* identically across all **inverted AUROC of 0.091** (\n\n**Critical Safety Implication:** Any system using self-consistency or agreement as a proxy for truth will be systematically deceived by frontier models on ungrounded factual recall. External grounding (RAG) is mandatory in this regime.\n\n⚠️ \n\n| Model Scale | Benchmark | Accuracy | Spanda ( | Neural SE AUROC | Latency | GPU Req. | \n|---|---|---|---|---|---|---|\n| **Qwen-1.5B** | GSM8K | 11.4% | 0.577 | **0.584** |  | None | \n| **Qwen-1.5B** | TriviaQA | 32.0% | 0.797 | **0.801** |  | None | \n| **Mistral-7B** | GSM8K | 8.2% | **0.706** | 0.705 |  | None | \n| **Mistral-7B** | TriviaQA | 45.0% | 0.698 | **0.755** |  | None | \n| **Qwen-27B** | GSM8K | 61.2% | **0.889** | --- |  | None | \n| **DeBERTa Baseline** | *N/A* | --- | --- | --- | **$\\sim$92.4 ms** | Required | \n\nEmpirical audit conducted across 50,000 evaluation iterations and 300 concurrent live HTTP reverse-proxy round-trips:\n\n| Metric / Dimension | ⚡ Spanda Rust Gateway ( `spnda` ) | 🐢 LiteLLM Python ( `litellm` ) | Neural Semantic Entropy (DeBERTa) | \n|---|---|---|---|\n| **Mathematical Kernel Latency** | **652.1 nanoseconds (0.65 µs)** | ~15,000 µs (with neural judge) | 92,400 µs (92.4 ms) | \n| **Kernel Throughput (Single Core)** | **1,533,500 evals/sec** | ~200,000 evals/sec (no-op hook) | ~10 evals/sec | \n| **Cold Startup Time** | **3.69 ms** | **1,177.08 ms (1.17 s)** | N/A | \n| **Memory Footprint (Idle RSS)** | **2.98 MB** | **229.61 MB** | ~1.8 GB GPU VRAM | \n| **Proxy Net Latency Overhead** | **0.076 ms (76.3 µs)** | 12.0 – 28.0 ms (FastAPI/Uvicorn) | N/A | \n| **Hardware Requirement** | **Pure CPU (Zero GPU)** | Pure CPU (plumbing) / GPU (judge) | Dedicated Nvidia GPU | \n\nGiven \n\nThe **Normalized Shannon Entropy** is:\n$$H_{\\text{norm}} = \\begin{cases} 0 & \\text{if } n = 1 \\ \\displaystyle\\frac{-\\sum_{i=1}^n w_i \\ln w_i}{\\ln K} & \\text{if } n > 1 \\end{cases}$$\n\nThe combined **Spanda Risk Score ( $R_{sc}$)** balances entropy dispersion with modal dominance (\n\n- \n$R_{sc} = 0$ : Complete consensus (model is confident).\n- \n$R_{sc} \\to 1$ : Maximum epistemic divergence (model is guessing / hallucinating).\n\nSpanda is lightweight and requires **zero third-party dependencies** (pure Python standard library).\n\n```\npip install spnda\n```\n\n*(Package name on PyPI is `spnda`; module is imported in Python as `import spanda`)*\n\nOr install from source:\n\n```\ngit clone https://github.com/Adarshent/Spnda.git\ncd Spnda\npip install -e .\n```\n\nWrap any standard OpenAI, Groq, Ollama, or OpenAI-compatible client with transparent multi-path epistemic uncertainty quantification (\n\n``` python\nimport spanda\nfrom openai import OpenAI\n\n# 1-line drop-in wrapper (samples K=3 paths transparently)\nclient = spanda.wrap(OpenAI(), k=3, threshold=0.35, block=False)\n\nresponse = client.chat.completions.create(\n    model=\"gpt-4o-mini\",\n    messages=[{\"role\": \"user\", \"content\": \"What is 17 * 19?\"}]\n)\n\n# Under the hood: runs in < 1 microsecond via native compiled Rust engine\nprint(response.spanda.rsc)             # 0.0000 (Unanimous consensus)\nprint(response.spanda.is_safe)         # True\nprint(response.spanda.decision)        # 'FAST_PASS_CONSISTENT'\nprint(response.spanda.latency_us)      # 0.7 µs!\nprint(response.choices[0].message.content) # Dominant consensus answer\n```\n\nIf `block=True` is passed and the model hallucinates or diverges, `spanda.wrap` raises a `SpandaUncertaintyError` before invalid data reaches your users.\n\nFor production microservices and non-Python languages (TypeScript, Go, Rust, Ruby, curl), run the standalone compiled Rust gateway:\n\n```\n# Launch the 760-nanosecond Rust proxy forwarding to any upstream LLM\nspnda serve --upstream http://localhost:11434/v1 --port 8080 --block --k 3\n\n# Or benchmark the mathematical engine directly\nspnda bench --iterations 200000\n# ✓ Latency per Eval : 767.9 nanoseconds\n# ✓ Throughput       : 1,302,312 evaluations/sec on single core!\n\n# Test any candidate completions via CLI\nspnda eval \"42\" \"42.0\" \"42\"\n```\n\nAny application in any language can simply point `base_url=\"http://localhost:8080/v1\"` to receive automatic sub-microsecond epistemic verification, Prometheus `/metrics`, and headers:\n\n- `X-Spanda-Rsc: 0.0000`\n- `X-Spanda-State: CONSISTENT`\n- `X-Spanda-Decision: FAST_PASS_CONSISTENT`\n- `X-Spanda-Latency-Us: 0.7`\n- `X-Spanda-Attractor: false`\n\n``` python\nfrom spanda import compute_rsc, detect_hallucination, batch_compute_rsc\n\n# 1. Basic Uncertainty Quantification\nsamples = [\"Paris\", \"paris.\", \"Paris\", \"Paris\", \"Paris\"]\nres = compute_rsc(samples)\nprint(f\"R_sc Score: {res['rsc']}\")             # 0.0 (High confidence)\nprint(f\"Dominant Answer: {res['dominant_answer']}\") # 'Paris'\n\n# 2. Production Hallucination Guardrail\nguard = detect_hallucination([\"42\", \"42\", \"24\", \"17\", \"99\"], threshold=0.35)\nif guard[\"is_uncertain\"]:\n    print(f\"🚨 Hallucination Warning (R_sc = {guard['rsc']}). Routing to RAG / Review.\")\nelse:\n    print(f\"✅ Safe output: {guard['dominant_answer']}\")\n\n# 3. High-Throughput Batch Processing\nbatch = [\n    [\"Answer A\", \"Answer A\", \"Answer A\"],\n    [\"Choice 1\", \"Choice 2\", \"Choice 3\"]\n]\nfor r in batch_compute_rsc(batch):\n    print(r[\"rsc\"], r[\"dominant_answer\"])\n```\n\nFor mission-critical production pipelines, Spanda provides a **2-Tier Cascaded Guardrail** that combines sub-millisecond consensus filtering with context grounding and tool-call safety:\n\n``` python\nfrom spanda import CascadedGuardrail\n\nguard = CascadedGuardrail(\n    uncertainty_threshold=0.3,\n    grounding_threshold=0.15\n)\n\n# 1. RAG Query with Mode Collapse Protection\nrag_context = \"Documentation: The production cluster runs in us-east-1.\"\nunanimous_hallucination = [\"eu-west-3 Paris\", \"eu-west-3 Paris\", \"eu-west-3 Paris\"]\n\nreceipt = guard.evaluate(unanimous_hallucination, context=rag_context)\nprint(receipt.decision)       # 'MODE_COLLAPSE_RISK'\nprint(receipt.is_safe)        # False (Unanimous agreement, but 0% grounded in source!)\nprint(receipt.tier_executed)  # Tier 2\nprint(receipt.latency_ms)     # < 0.05 ms\n\n# 2. Agent Tool Call Argument Verification (e.g. preventing bad 'rm')\ntool_calls = [\n    {\"command\": \"rm -rf /var/cache\"},\n    {\"command\": \"rm -rf /var/log\"},  # Conflict detected across parallel paths!\n]\nagent_receipt = guard.evaluate_tool_calls(tool_calls)\nprint(agent_receipt.decision) # 'TOOL_ARG_MISMATCH' (Execution blocked!)\n\n# 3. Export SOC2 Audit Receipt\nimport json\nprint(json.dumps(receipt.to_dict(), indent=2))\n```\n\nSpanda connects into modern enterprise LLM pipelines with zero external dependencies:\n\n``` python\n# 1. LangChain String Evaluator\nfrom spanda.integrations.langchain import SpandaStringEvaluator\n\nevaluator = SpandaStringEvaluator(uncertainty_threshold=0.35)\nresult = evaluator.evaluate_strings(\n    prediction=[\"Paris\", \"Paris\", \"Paris\", \"Paris\"],\n    context=\"Paris is the capital of France.\"\n)\nprint(result[\"value\"])  # 'PASS' (Score: 0.0)\n\n# 2. LlamaIndex Response Guardrail\nfrom spanda.integrations.llamaindex import SpandaRAGGuardrail\n\nguard = SpandaRAGGuardrail()\nreceipt = guard.validate_response(\n    samples=[\"Result A\", \"Result A\", \"Result A\"],\n    context_str=\"Retrieved node knowledge...\"\n)\nprint(receipt.is_safe)  # True\n\n# 3. LiteLLM Proxy / SDK Callback Hook\nimport litellm\nfrom spanda.integrations.litellm import SpandaLiteLLMGuardrail\n\nlitellm.callbacks = [SpandaLiteLLMGuardrail(threshold=0.35, block_mode=False)]\n```\n\nRun the standalone compiled Rust gateway in Docker or Kubernetes:\n\n```\ndocker run -d -p 8080:8080 \\\n  -e SPANDA_UPSTREAM=https://api.openai.com/v1 \\\n  -e SPANDA_THRESHOLD=0.35 \\\n  brhmn/spnda-gateway\n```\n\n- `GET /metrics` : Standard Prometheus format for Grafana (`spanda_requests_total` ,`spanda_evaluations_total` ,`spanda_mode_collapses_total` ,`spanda_eval_latency_avg_us` ).\n- `GET /healthz` : Kubernetes liveness probe.\n- `GET /readyz` : Kubernetes readiness probe.\n- Structured JSON logging: Every transaction emits a machine-parseable log line to stdout for Datadog / CloudWatch / Splunk.\n\n**Operational Scope:** Spanda is engineered for **structured reasoning, math, code, agent tool-call arguments, SQL, and canonical factual RAG extraction** where 90ms GPU cross-encoders are an unacceptable bottleneck. It is **not** designed for open-ended, free-form creative prose (e.g., essays or poetry), where synonymous phrasing is naturally diverse and requires heavy neural NLI.\n\n⚠️ \n\n| Use Case / Architecture | Recommendation | Rationale | \n|---|---|---|\n| **Math, Code & Structured QA (7B–70B)** | ✅ **Recommended** | Coherence Scaling Law ensures exact-match matches neural SE at 0 cost. | \n| **High-Throughput Production APIs** | ✅ **Recommended** | 90,000x latency reduction without GPU requirements. | \n| **Free-form Paraphrase QA (<7B)** | **Use Neural SE** | Small models produce inconsistent surface phrasing. | \n| **Ungrounded Facts on Frontier Models (>100B)** | ❌ **Do Not Use Alone** | Subject to **Confident Mode Collapse** ; must combine with retrieval (RAG). | \n\nRun the test suite:\n\n```\npython3 -m unittest discover tests\n```\n\nIf you use Spanda in your research or production systems, please cite:\n\n```\n@article{nayak2026spanda,\n  title={Spanda: Zero-Cost Lexical Entropy Matches Neural Semantic Uncertainty---Until Frontier Models Break It},\n  author={Nayak, Bhupen},\n  journal={arXiv preprint},\n  year={2026},\n  doi={10.5281/zenodo.22233648},\n  url={https://doi.org/10.5281/zenodo.22233648}\n}\n```\n\nSpanda adopts a developer-friendly dual-licensing model:\n\n- **Python SDK & Integrations (`spanda`)** : Permissive**[MIT License](/Adarshent/Spnda/blob/main/LICENSE)** . Free for all developers, commercial and open-source applications, with zero dependency friction.\n- **Compiled Rust Core Engine & Gateway (`spnda`)** :**[Business Source License 1.1 (BSL 1.1)](/Adarshent/Spnda/blob/main/LICENSE)** . Free for developers, research, and internal production infrastructure. Prohibits offering Spanda as a competing commercial third-party managed service without an enterprise license from BRHMN Labs Private Limited. Automatically converts to Apache 2.0 on January 1, 2030.", "url": "https://wpnews.pro/news/show-hn-spanda-sub-microsecond-llm-epistemic-uncertainty-in-rust", "canonical_source": "https://github.com/Adarshent/Spnda", "published_at": "2026-09-11 21:48:56+00:00", "updated_at": "2026-09-11 22:23:29.718624+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-tools", "ai-research", "natural-language-processing"], "entities": ["Spanda", "Rust", "Semantic Entropy", "Exact-Match Normalized Entropy", "DeBERTa-v3", "Qwen-1.5B", "Mistral-7B", "GSM8K"], "alternates": {"html": "https://wpnews.pro/news/show-hn-spanda-sub-microsecond-llm-epistemic-uncertainty-in-rust", "markdown": "https://wpnews.pro/news/show-hn-spanda-sub-microsecond-llm-epistemic-uncertainty-in-rust.md", "text": "https://wpnews.pro/news/show-hn-spanda-sub-microsecond-llm-epistemic-uncertainty-in-rust.txt", "jsonld": "https://wpnews.pro/news/show-hn-spanda-sub-microsecond-llm-epistemic-uncertainty-in-rust.jsonld"}}