{"slug": "show-hn-strot-compile-an-llm-s-intent-once-run-decisions-locally-forever", "title": "Show HN: Strot – Compile an LLM's intent once, run decisions locally forever", "summary": "Strot released an open-source Python package, strot-ai, that uses an LLM once at compile time to build a frozen local decision artifact, then classifies, scores and routes records on CPU at roughly 60 ms per record with zero marginal cost and zero LLM calls at runtime. On a 144M-parameter SupersonicLabs/Julia-1 model, Strot reports 92.5% grounded routing accuracy across 398 records versus 74.8% for a naive approach (+18 points) and 86.7% versus 72.7% on AG News with unseen labels (+14 points), with low-confidence records abstaining via split conformal prediction. The package ships with published benchmarks and losses, and its typed decisions matched a leading hosted decision API's own example nearly exactly (sentiment 1.79 versus 1.8).", "body_md": "**Compile your intent once with an LLM. Then decide on every record forever** —\nclassification, scoring, routing — as type-safe JSON with probabilities and\nconfidence, on your own hardware, at **zero marginal cost**.\n\n[**Website**](https://strot.ai) · [** Docs**](https://github.com/strot-ai/Project-STROT/blob/main/DEVELOPERS.md) · [** Benchmarks**](https://github.com/strot-ai/Project-STROT/blob/main/benchmarks) · **PyPI**\n\n```\nAgents at compile time.  Never at runtime.\npip install strot-ai\npython\nimport strot\n\nresult = strot.run(\n    tickets,\n    query=\"Triage incoming support tickets into the correct workflow.\",\n    action_registry={\n        \"billing_dispute\":  \"Refunds, invoice corrections, double charges\",\n        \"technical_defect\": \"Crashes, bugs, broken behavior\",\n    },\n    llm=\"anthropic/claude-opus-4-8\",   # used once, on first compile only\n)\n\nfor o in result.outcomes:\n    print(o[\"action\"], round(o[\"confidence\"], 2))\n# billing_dispute 0.97\n# technical_defect 0.94\n```\n\nThe LLM runs **once**, at compile time. Production is a frozen, self-contained\nPython artifact: a small local decision model arbitrates every record, nothing\nis generated, every output is bounded, and low-confidence records abstain\ninstead of guessing.\n\n```\nSTROT = Agentic Schema Intent + Compiler & State + Code + Execution + NLU\n                      ⟹  JSON Decisions\n          (classification · scoring · routing · typed answers)\nCOMPILE  (once per intent + schema)\n  intent + sample ─▶ synthesize ─▶ golden gate ─▶ teacher labels ─▶ calibrate\n                     (labels,      (your known      (annotate         (fit the\n                      projection,   answers must     a sample,         confidence\n                      handlers)     pass)            once)             threshold)\n                                        │\n                                        ▼\n                             frozen artifact  (.py)\n\n            RUNTIME  (every record, forever)\n  record ─▶ projection ─▶ decision model ─▶ confidence gate ─▶ typed JSON\n            (compiled)     (144M, CPU,        (decide or         decision\n                            ~60 ms)            abstain)\n```\n\n**`run()`** — route each record to an action and execute the compiled handler.\n\n**`decide()`** — answer typed questions (choice / score / boolean) per record:\n\n```\nresult = strot.decide(records, questions={\n    \"is_bug_report\":     {\"type\": \"boolean\", \"instructions\": \"mentions crashes/errors?\"},\n    \"customer_sentiment\":{\"type\": \"score\", \"min\": 1, \"max\": 5, \"instructions\": \"1 angry, 5 happy\"},\n    \"routing_bucket\":    {\"type\": \"choice\", \"options\": [\"billing\", \"engineering\", \"spam\"]},\n})\n# → {\"booleans\": {...}, \"scores\": {...}, \"choices\": {...}}  with probabilities + confidence\n```\n\nReal model (SupersonicLabs/Julia-1, 144M, CPU), real datasets, clean Docker containers:\n\n|  | Result | \n|---|---|\n| Grounded routing accuracy (398 records) | **92.5%** compiled vs 74.8% naive — the compiler is worth +18pp on the same model | \n| Semantic text (AG News, unseen labels) | **86.7%** compiled vs 72.7% naive (+14pp) | \n| Typed decisions vs the leading hosted decision API's own example | near-identical (sentiment 1.79 vs 1.8) | \n| Latency / marginal cost / LLM calls per record | **~60 ms · $0 · 0** | \n\nReproduce everything: [`benchmarks/`](https://github.com/strot-ai/Project-STROT/blob/main/benchmarks) — with the losses published too.\n\n- **Compile once, decide forever** — content-addressed artifact cache; schema drift triggers exactly one recompile.\n- **Golden-record promotion** — artifacts must pass your known answers before they serve.\n- **Fitted guarantees** — temperature scaling + split conformal prediction, baked in: dispatched decisions are wrong ≤ α of all records (marginal); the rest abstain.\n- **Measured, not guessed** — the compiler scores labelings, and can train a distilled head, keeping only what beats the baseline on held-out data.\n- **0% structural hallucination** — output is selection + computation over your registry; nothing is generated at runtime.\n- **Model-independent** — the decision model is a slot (`nlu=` ): local Julia-1 by default, hosted APIs, or any drop-in specialist.\n\n```\npip install strot-ai            # core: pure stdlib\npip install strot-ai[local]     # + the local decision model (torch/transformers)\npip install strot-ai[hosted]    # + openai, for a hosted decision runtime\npip install strot-ai[anthropic] # + anthropic, for the synthesizer\n```\n\nBake the local model into your image for offline production:\n\n``` python\nRUN pip install strot-ai[local] \\\n && python -c \"import strot; strot.prefetch_nlu(local_dir='/opt/julia-1', install=True)\"\nENV STROT_NLU_CHECKPOINT=/opt/julia-1\n```\n\n(Never `pip install julia` — unrelated package.)\n\n[**DEVELOPERS.md**](https://github.com/strot-ai/Project-STROT/blob/main/DEVELOPERS.md) — architecture, compile lifecycle, calibration\nsemantics and caveats, deployment, model swapping, full API reference.\n\nMIT. Default decision model: Apache-2.0 ([SupersonicLabs/Julia-1](https://huggingface.co/SupersonicLabs/Julia-1)). Implements ideas from STROT (arXiv:2505.01636).", "url": "https://wpnews.pro/news/show-hn-strot-compile-an-llm-s-intent-once-run-decisions-locally-forever", "canonical_source": "https://github.com/strot-ai/Project-STROT", "published_at": "2026-10-19 04:32:07+00:00", "updated_at": "2026-10-19 04:48:41.316454+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "machine-learning", "developer-tools", "ai-agents"], "entities": ["Strot", "strot-ai", "SupersonicLabs", "Julia-1", "PyPI", "AG News"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-strot-compile-an-llm-s-intent-once-run-decisions-locally-forever", "markdown": "https://wpnews.pro/news/show-hn-strot-compile-an-llm-s-intent-once-run-decisions-locally-forever.md", "text": "https://wpnews.pro/news/show-hn-strot-compile-an-llm-s-intent-once-run-decisions-locally-forever.txt", "jsonld": "https://wpnews.pro/news/show-hn-strot-compile-an-llm-s-intent-once-run-decisions-locally-forever.jsonld"}}