Show HN: Spanda – Sub-microsecond LLM epistemic uncertainty in Rust

A Rust-based tool called Spanda computes LLM epistemic uncertainty in 652.1 nanoseconds (0.65 µs) per evaluation, roughly 90,000× faster than neural Semantic Entropy methods that add about 92.4 ms of GPU overhead per inference call, according to the project's Show HN release. Spanda's Exact-Match Normalized Entropy (R_sc) is a zero-parameter, zero-GPU metric that matches or exceeds DeBERTa-v3 NLI cross-encoder Semantic Entropy on structured reasoning across models from 1.5B to 120B parameters, the release states. The release also reports that at 120B scale on ungrounded factual recall (TriviaQA) the model exhibits "Confident Mode Collapse," producing the same incorrect answer across samples for an inverted AUROC of 0.091, which the project says makes external grounding (RAG) mandatory in that regime.

Detect LLM hallucinations and quantify uncertainty in microseconds without secondary NLI cross-encoders. Traditional epistemic uncertainty estimation in LLMs relies on Semantic Entropy SE Kuhn et al., 2023 https://arxiv.org/abs/2302.09664 ; Farquhar et al., Nature 2024 https://www.nature.com/articles/s41586-024-07421-0 . While effective, Semantic Entropy requires clustering This introduces two severe production bottlenecks: 1. Quadratic Cost: $\binom{K}{2}$ forward passes per query 45 neural evaluations for$K=10$ . 2. Serving Latency: Adds $\sim$90 ms of GPU overhead per inference call, making it unusable for high-throughput production serving. Spanda introduces Exact-Match Normalized Entropy $R {sc}$ : a zero-parameter, zero-GPU metric that computes uncertainty directly over deterministic lexical clusters. Across empirical evaluations spanning two orders of magnitude 1.5B to 120B parameters , Spanda matches or exceeds neural Semantic Entropy on structured reasoning while operating ~90,000$\times$ faster As model capacity increases from 1.5B to 27B parameters, internal reasoning coherence causes correct predictions to naturally converge to identical lexical sequences. On mathematical reasoning GSM8K , exact-match AUROC scales monotonically: At 7B+ parameters , Spanda achieves the exact same discriminative power as heavy DeBERTa-v3 NLI cross-encoders, rendering the neural clustering step redundant for reasoning. At the 120B frontier scale on ungrounded factual recall TriviaQA , the model exhibits Confident Mode Collapse : its parametric memory and RLHF tuning cause it to hallucinate the exact same incorrect answer identically across all inverted AUROC of 0.091 Critical Safety Implication: Any system using self-consistency or agreement as a proxy for truth will be systematically deceived by frontier models on ungrounded factual recall. External grounding RAG is mandatory in this regime. ⚠️ | Model Scale | Benchmark | Accuracy | Spanda | Neural SE AUROC | Latency | GPU Req. | |---|---|---|---|---|---|---| | Qwen-1.5B | GSM8K | 11.4% | 0.577 | 0.584 | | None | | Qwen-1.5B | TriviaQA | 32.0% | 0.797 | 0.801 | | None | | Mistral-7B | GSM8K | 8.2% | 0.706 | 0.705 | | None | | Mistral-7B | TriviaQA | 45.0% | 0.698 | 0.755 | | None | | Qwen-27B | GSM8K | 61.2% | 0.889 | --- | | None | | DeBERTa Baseline | N/A | --- | --- | --- | $\sim$92.4 ms | Required | Empirical audit conducted across 50,000 evaluation iterations and 300 concurrent live HTTP reverse-proxy round-trips: | Metric / Dimension | ⚡ Spanda Rust Gateway spnda | 🐢 LiteLLM Python litellm | Neural Semantic Entropy DeBERTa | |---|---|---|---| | Mathematical Kernel Latency | 652.1 nanoseconds 0.65 µs | ~15,000 µs with neural judge | 92,400 µs 92.4 ms | | Kernel Throughput Single Core | 1,533,500 evals/sec | ~200,000 evals/sec no-op hook | ~10 evals/sec | | Cold Startup Time | 3.69 ms | 1,177.08 ms 1.17 s | N/A | | Memory Footprint Idle RSS | 2.98 MB | 229.61 MB | ~1.8 GB GPU VRAM | | Proxy Net Latency Overhead | 0.076 ms 76.3 µs | 12.0 – 28.0 ms FastAPI/Uvicorn | N/A | | Hardware Requirement | Pure CPU Zero GPU | Pure CPU plumbing / GPU judge | Dedicated Nvidia GPU | Given The Normalized Shannon Entropy is: $$H {\text{norm}} = \begin{cases} 0 & \text{if } n = 1 \ \displaystyle\frac{-\sum {i=1}^n w i \ln w i}{\ln K} & \text{if } n 1 \end{cases}$$ The combined Spanda Risk Score $R {sc}$ balances entropy dispersion with modal dominance - $R {sc} = 0$ : Complete consensus model is confident . - $R {sc} \to 1$ : Maximum epistemic divergence model is guessing / hallucinating . Spanda is lightweight and requires zero third-party dependencies pure Python standard library . pip install spnda Package name on PyPI is spnda ; module is imported in Python as import spanda Or install from source: git clone https://github.com/Adarshent/Spnda.git cd Spnda pip install -e . Wrap any standard OpenAI, Groq, Ollama, or OpenAI-compatible client with transparent multi-path epistemic uncertainty quantification python import spanda from openai import OpenAI 1-line drop-in wrapper samples K=3 paths transparently client = spanda.wrap OpenAI , k=3, threshold=0.35, block=False response = client.chat.completions.create model="gpt-4o-mini", messages= {"role": "user", "content": "What is 17 19?"} Under the hood: runs in < 1 microsecond via native compiled Rust engine print response.spanda.rsc 0.0000 Unanimous consensus print response.spanda.is safe True print response.spanda.decision 'FAST PASS CONSISTENT' print response.spanda.latency us 0.7 µs print response.choices 0 .message.content Dominant consensus answer If block=True is passed and the model hallucinates or diverges, spanda.wrap raises a SpandaUncertaintyError before invalid data reaches your users. For production microservices and non-Python languages TypeScript, Go, Rust, Ruby, curl , run the standalone compiled Rust gateway: Launch the 760-nanosecond Rust proxy forwarding to any upstream LLM spnda serve --upstream http://localhost:11434/v1 --port 8080 --block --k 3 Or benchmark the mathematical engine directly spnda bench --iterations 200000 ✓ Latency per Eval : 767.9 nanoseconds ✓ Throughput : 1,302,312 evaluations/sec on single core Test any candidate completions via CLI spnda eval "42" "42.0" "42" Any application in any language can simply point base url="http://localhost:8080/v1" to receive automatic sub-microsecond epistemic verification, Prometheus /metrics , and headers: - X-Spanda-Rsc: 0.0000 - X-Spanda-State: CONSISTENT - X-Spanda-Decision: FAST PASS CONSISTENT - X-Spanda-Latency-Us: 0.7 - X-Spanda-Attractor: false python from spanda import compute rsc, detect hallucination, batch compute rsc 1. Basic Uncertainty Quantification samples = "Paris", "paris.", "Paris", "Paris", "Paris" res = compute rsc samples print f"R sc Score: {res 'rsc' }" 0.0 High confidence print f"Dominant Answer: {res 'dominant answer' }" 'Paris' 2. Production Hallucination Guardrail guard = detect hallucination "42", "42", "24", "17", "99" , threshold=0.35 if guard "is uncertain" : print f"🚨 Hallucination Warning R sc = {guard 'rsc' } . Routing to RAG / Review." else: print f"✅ Safe output: {guard 'dominant answer' }" 3. High-Throughput Batch Processing batch = "Answer A", "Answer A", "Answer A" , "Choice 1", "Choice 2", "Choice 3" for r in batch compute rsc batch : print r "rsc" , r "dominant answer" For mission-critical production pipelines, Spanda provides a 2-Tier Cascaded Guardrail that combines sub-millisecond consensus filtering with context grounding and tool-call safety: python from spanda import CascadedGuardrail guard = CascadedGuardrail uncertainty threshold=0.3, grounding threshold=0.15 1. RAG Query with Mode Collapse Protection rag context = "Documentation: The production cluster runs in us-east-1." unanimous hallucination = "eu-west-3 Paris", "eu-west-3 Paris", "eu-west-3 Paris" receipt = guard.evaluate unanimous hallucination, context=rag context print receipt.decision 'MODE COLLAPSE RISK' print receipt.is safe False Unanimous agreement, but 0% grounded in source print receipt.tier executed Tier 2 print receipt.latency ms < 0.05 ms 2. Agent Tool Call Argument Verification e.g. preventing bad 'rm' tool calls = {"command": "rm -rf /var/cache"}, {"command": "rm -rf /var/log"}, Conflict detected across parallel paths agent receipt = guard.evaluate tool calls tool calls print agent receipt.decision 'TOOL ARG MISMATCH' Execution blocked 3. Export SOC2 Audit Receipt import json print json.dumps receipt.to dict , indent=2 Spanda connects into modern enterprise LLM pipelines with zero external dependencies: python 1. LangChain String Evaluator from spanda.integrations.langchain import SpandaStringEvaluator evaluator = SpandaStringEvaluator uncertainty threshold=0.35 result = evaluator.evaluate strings prediction= "Paris", "Paris", "Paris", "Paris" , context="Paris is the capital of France." print result "value" 'PASS' Score: 0.0 2. LlamaIndex Response Guardrail from spanda.integrations.llamaindex import SpandaRAGGuardrail guard = SpandaRAGGuardrail receipt = guard.validate response samples= "Result A", "Result A", "Result A" , context str="Retrieved node knowledge..." print receipt.is safe True 3. LiteLLM Proxy / SDK Callback Hook import litellm from spanda.integrations.litellm import SpandaLiteLLMGuardrail litellm.callbacks = SpandaLiteLLMGuardrail threshold=0.35, block mode=False Run the standalone compiled Rust gateway in Docker or Kubernetes: docker run -d -p 8080:8080 \ -e SPANDA UPSTREAM=https://api.openai.com/v1 \ -e SPANDA THRESHOLD=0.35 \ brhmn/spnda-gateway - GET /metrics : Standard Prometheus format for Grafana spanda requests total , spanda evaluations total , spanda mode collapses total , spanda eval latency avg us . - GET /healthz : Kubernetes liveness probe. - GET /readyz : Kubernetes readiness probe. - Structured JSON logging: Every transaction emits a machine-parseable log line to stdout for Datadog / CloudWatch / Splunk. Operational Scope: Spanda is engineered for structured reasoning, math, code, agent tool-call arguments, SQL, and canonical factual RAG extraction where 90ms GPU cross-encoders are an unacceptable bottleneck. It is not designed for open-ended, free-form creative prose e.g., essays or poetry , where synonymous phrasing is naturally diverse and requires heavy neural NLI. ⚠️ | Use Case / Architecture | Recommendation | Rationale | |---|---|---| | Math, Code & Structured QA 7B–70B | ✅ Recommended | Coherence Scaling Law ensures exact-match matches neural SE at 0 cost. | | High-Throughput Production APIs | ✅ Recommended | 90,000x latency reduction without GPU requirements. | | Free-form Paraphrase QA <7B | Use Neural SE | Small models produce inconsistent surface phrasing. | | Ungrounded Facts on Frontier Models 100B | ❌ Do Not Use Alone | Subject to Confident Mode Collapse ; must combine with retrieval RAG . | Run the test suite: python3 -m unittest discover tests If you use Spanda in your research or production systems, please cite: @article{nayak2026spanda, title={Spanda: Zero-Cost Lexical Entropy Matches Neural Semantic Uncertainty---Until Frontier Models Break It}, author={Nayak, Bhupen}, journal={arXiv preprint}, year={2026}, doi={10.5281/zenodo.22233648}, url={https://doi.org/10.5281/zenodo.22233648} } Spanda adopts a developer-friendly dual-licensing model: - Python SDK & Integrations spanda : Permissive MIT License /Adarshent/Spnda/blob/main/LICENSE . Free for all developers, commercial and open-source applications, with zero dependency friction. - Compiled Rust Core Engine & Gateway spnda : Business Source License 1.1 BSL 1.1 /Adarshent/Spnda/blob/main/LICENSE . Free for developers, research, and internal production infrastructure. Prohibits offering Spanda as a competing commercial third-party managed service without an enterprise license from BRHMN Labs Private Limited. Automatically converts to Apache 2.0 on January 1, 2030.