{"slug": "stop-calling-an-llm-for-every-ticket-compile-the-decision-once-then-run-it", "title": "Stop Calling an LLM for Every Ticket: Compile the Decision Once, Then Run It Locally", "summary": "The open-source project STROT is promoting an architecture that uses a large language model once to compile a bounded decision task, then runs a smaller local decision model per record instead of calling a hosted LLM for every ticket. The runtime selects from a fixed label registry and abstains to a human when its confidence gate can't make a safe decision, with a compile phase that profiles a sample, synthesizes and validates a configuration, and tests golden records as a promotion gate. The project's performance figures are self-reported and the guide is based on its public README and developer documentation rather than independent testing.", "body_md": "A support queue gets a few thousand new tickets a day. Most need one of three destinations: billing, engineering, or account support. The usual AI implementation sends each ticket to a hosted model, asks for a category, parses the answer, then repeats the process for the next ticket.\n\nThat is a reasonable first prototype. It can become an expensive and difficult-to-audit production path when the decision space is small, the same instructions repeat, and every output must fit a strict schema.\n\nA different design is showing up in the open-source project STROT: use a larger language model once to compile the task, then use a smaller decision model locally for each record. The runtime selects from known labels instead of generating free-form text. If its confidence gate cannot make a safe decision, it abstains for a human or another system to handle.\n\nThis guide walks through that pattern with STROT's Python API. It is based on the project's public README and developer guide, not a claim of personal testing. STROT is new, and its performance figures are project-reported measurements. Treat the example as a starting point for an evaluation, not as a production endorsement.\n\nThis architecture is for repeated, bounded decisions. The system receives a record and must choose among known actions, answer a boolean question, or assign a score inside a defined range.\n\nGood candidates include:\n\nIt is a poor fit when the task needs fresh world knowledge, open-ended writing, complex multi-step reasoning, or an answer that is not represented by your labels. A small classifier cannot replace a generative model just because it is cheaper. The design works by making the decision space explicit and bounded.\n\nSTROT describes its architecture as a compile phase followed by a runtime phase. The compile phase profiles a sample, asks an LLM to synthesize a configuration, validates the result, tests known answers, and optionally fits calibration. The saved artifact then handles later records without another LLM call on a cache hit. The repository describes this lifecycle in its [developer guide](https://github.com/strot-ai/Project-STROT/blob/main/DEVELOPERS.md).\n\nThe project separates its base package from optional runtime integrations. Its README lists these extras:\n\n```\npip install 'strot-ai[local,anthropic]'\n```\n\nThe `local` extra supplies dependencies for the default local decision model. The `anthropic` extra enables the documented Anthropic synthesizer. If your environment uses a different supported synthesizer, install the matching integration and configure its credential using that provider's instructions. Do not store API keys in source code, fixtures, or a committed `.env` file.\n\nFor a small experiment, start with a representative fixture rather than a production export. Remove secrets and personal information first. The compile stage can inspect sample fields and values, so the data you provide is part of the prompt and the resulting artifact's design.\n\nThe registry is the boundary of what the runtime may choose. Keep the keys stable and descriptions distinct. Overlapping descriptions make a classification problem harder, not easier.\n\n```\nACTIONS = {\n    \"billing\": \"Invoices, duplicate charges, refunds, and payment failures\",\n    \"engineering\": \"Crashes, broken features, error messages, and technical defects\",\n    \"account_support\": \"Login trouble, profile changes, and account access\",\n}\n```\n\nThese are examples, not universal categories. If a ticket can reasonably belong to two actions, clarify your routing policy before asking a model to learn it. Decide whether the system should choose the first owning team, the team that can resolve it, or a queue for further triage.\n\nA golden record is an input paired with the answer your application expects. It acts as a promotion gate: an artifact that fails one of these known cases should not become the active artifact.\n\n```\nGOLDEN = [\n    {\n        \"record\": {\n            \"subject\": \"Charged twice for my monthly plan\",\n            \"body\": \"I see two payments on my bank statement.\",\n        },\n        \"expect\": \"billing\",\n    },\n    {\n        \"record\": {\n            \"subject\": \"The export button crashes\",\n            \"body\": \"Clicking export returns an error every time.\",\n        },\n        \"expect\": \"engineering\",\n    },\n    {\n        \"record\": {\n            \"subject\": \"I cannot sign in\",\n            \"body\": \"The password reset link says my account is unknown.\",\n        },\n        \"expect\": \"account_support\",\n    },\n]\n```\n\nThe examples should cover one clear case per action at minimum. Add edge cases that have caused real confusion, such as a refund request caused by a software defect. Do not build goldens out of only easy examples. The developer guide says goldens are checked before promotion and recommends three to five as a practical starting range, while emphasizing that compile results can vary and the gate is there to catch failures.\n\nGolden records are not a substitute for a held-out evaluation set. They are a compact regression check, not evidence that the task works on all future traffic.\n\nHere is the documented `run()` shape, with a small list of representative records:\n\n``` python\nimport strot\n\nsample = [\n    {\n        \"subject\": \"Refund for a duplicate payment\",\n        \"body\": \"My card was charged twice this morning.\",\n    },\n    {\n        \"subject\": \"Search returns an error\",\n        \"body\": \"The results page fails after I submit a query.\",\n    },\n]\n\nresult = strot.run(\n    sample,\n    query=\"Route each support ticket to the team that can resolve it.\",\n    action_registry=ACTIONS,\n    llm=\"anthropic/claude-opus-4-8\",\n    golden=GOLDEN,\n    state=\".strot\",\n)\n\nfor outcome in result.outcomes:\n    print(outcome[\"action\"], outcome[\"confidence\"])\n\nfor fallback in result.fallbacks:\n    print(\"Needs review:\", fallback[\"reason\"])\n```\n\nThe provider identifier above is the example documented by the project, and may change as provider models and integrations evolve. The code shows the expected shape, not a verified compatibility test for today's account or API.\n\nOn a cache miss, STROT's guide describes profiling the sample schema, synthesizing labels and routing logic, validating the candidate, checking goldens, and saving an artifact. Later calls with the same cache key load the saved artifact instead of involving the synthesizer again. Changing configuration, registry, schema fields or types, goldens, package version, or other keyed settings can trigger a new compile. Review and monitor that boundary instead of assuming every deployment is a cache hit.\n\nA classifier should not be forced to guess every time. STROT reports outcomes and fallbacks separately. That separation is operationally useful: the application can route confident decisions automatically while putting uncertain records into a human queue.\n\nDo not equate a confidence number with a guarantee. Calibration requires representative labeled examples, and its behavior depends on the assumptions being met. The STROT guide distinguishes a marginal error bound across all records from the error rate among records that were actually dispatched. Those are not interchangeable. It also warns that distribution shift and imperfect teacher labels weaken the interpretation.\n\nFor a consequential workflow, start by sending every record to a human while logging the model's proposed outcome. Compare those proposals against reviewed labels. Only then consider automating a narrow, high-confidence slice. Keep the fallback path real and staffed; an abstention that silently disappears into a backlog is not a safety mechanism.\n\nBefore comparing the result with a large model, compare it with the simplest reasonable alternative:\n\nReport more than overall accuracy. Look at per-class precision and recall, false routing cost, fallback rate, latency on your hardware, and performance on ambiguous or newly worded tickets. A 90% aggregate score can hide a failure on a rare but high-impact category.\n\nSTROT's repository reports 92.5% compiled routing accuracy versus 74.8% for a naive approach on a 398-record task, and 86.7% versus 72.7% on an AG News task. These are project-reported measurements, not independent results or a forecast for your data. Read the benchmark code and reproduce it before relying on the numbers. The project's own guide also explains that calibration samples need to match the runtime distribution and that small samples limit the granularity of statistical claims.\n\nA useful mental model is to treat the compiled artifact like generated code. Put it through review and deployment controls:\n\nThere is an important security caveat. STROT's developer guide says its security lint is validation, not a sandbox, and that candidate code is executed during the golden gate. Do not compile untrusted intent on a sensitive machine without external isolation, locked-down permissions, and resource limits. The project recommends CI compilation and shipping a frozen artifact as a deployment pattern.\n\nUse compile-once decision workflows when the task is repetitive, bounded, and measurable; labels are stable; local runtime is valuable; and an uncertain answer can safely be escalated. It may reduce runtime API calls, but there are still costs for compilation, model downloads, hardware, operations, evaluation, and human fallback.\n\nKeep a generative model in the loop when the output must be novel, the answer requires current knowledge, or the input may introduce categories that your registry cannot express. You can also use a hybrid: let the local decision layer handle routine cases and send abstentions or unusual records to a more capable model or a person. That fallback is often more sensible than asking a small model to solve every case.\n\nThe useful question is not whether a small local model beats a large hosted model in general. It is whether your particular workflow contains many repeated decisions whose answer space you can define, test, and safely constrain.\n\nThe project is interesting because it reframes where the LLM belongs. Instead of paying for a fresh generation on every record, it uses the model to help construct a bounded decision process, then keeps that process inspectable and local at runtime. That is promising for narrow jobs, but the tests, monitoring, and human fallback still belong to the team shipping it.\n\nI write about practical AI infrastructure and developer tools. Subscribe if you want more build-focused guides like this.\n\nHave you found a repeated classification or routing task that could be compiled into a bounded decision process? Where would you draw the line between automatic handling and human review?", "url": "https://wpnews.pro/news/stop-calling-an-llm-for-every-ticket-compile-the-decision-once-then-run-it", "canonical_source": "https://dev.to/jamilxt/stop-calling-an-llm-for-every-ticket-compile-the-decision-once-then-run-it-locally-49d7", "published_at": "2026-10-19 16:06:12+00:00", "updated_at": "2026-10-19 16:15:02.072867+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-tools", "mlops", "developer-tools"], "entities": ["STROT", "Anthropic", "Project-STROT"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/stop-calling-an-llm-for-every-ticket-compile-the-decision-once-then-run-it", "markdown": "https://wpnews.pro/news/stop-calling-an-llm-for-every-ticket-compile-the-decision-once-then-run-it.md", "text": "https://wpnews.pro/news/stop-calling-an-llm-for-every-ticket-compile-the-decision-once-then-run-it.txt", "jsonld": "https://wpnews.pro/news/stop-calling-an-llm-for-every-ticket-compile-the-decision-once-then-run-it.jsonld"}}