TLDR: Two Amazon engineers swapped Claude Sonnet 4.6 for Strands Decider, an open 4B decision model, in one of the company’s largest internal ticketing systems, and report the same labelling quality at 98 percent less cost. It enables a category of work that has been paying frontier-model prices for a job that never needed sentence generation. The difference between this and a cheaper language model is that a decision model cannot write anything at all: it returns a probability for each option in a fixed list, plus a confidence score.
What was actually running #
The system labelled millions of internal tickets into 20 category types: customer escalations, capacity requests, automated alarms, security findings. An agent sent every ticket through Sonnet 4.6 with a long system prompt and got back JSON with a label and reasoning.
That is a classification problem wearing a chat model’s clothes. The output was always one of twenty known labels, but the model was re-reading a prompt longer than the ticket itself and generating an explanation nobody stored.
The swap, and the numbers #
They started with the 2B Decider and moved to the E4B Gemma 4 build, which did better on a held-out set of tickets. Then they re-tuned the confidence threshold at which a label gets applied.
The arithmetic is worth sitting with. At $3 per million input tokens and $15 per million output, a 6,724-token ticket costs 2.2 cents. At $1.408 for an ml.g5.xlarge instance-hour, the same ticket costs about $0.0005. That is 44 times cheaper.
How they de-risked it #
They did not flip a switch. The labelling Lambda reads a three-way setting from AWS AppConfig on every message:
BEDROCK: Sonnet labels every ticket, Decider is never called. #
DUAL: both run in parallel, Sonnet’s answer is applied, and every disagreement is logged. #
DECIDER: Decider labels everything, Bedrock is not called.
Rolling back was a 45-second config change. Running both on live traffic, then reading every disagreement, told them more than any offline evaluation would have.
The deployment itself is ordinary: ticket events land in SQS, a Lambda reads the ticket and calls a SageMaker endpoint, CloudWatch holds the logs, the stack is defined in CDK with Beta, Gamma and Prod stages. The endpoint’s network is isolated with no outbound access, and weights are copied in from S3 at start-up.
The part that will bite anyone copying this #
Two constraints, both from the write-up.
Label descriptions have to fit a 4,096-token window. Production’s label descriptions alone were longer than that. The engineers rewrote all 20 labels as one- or two-sentence criteria, sent them as the options of a single choice question, and added a “none” option. On long tickets, the model drops the oldest comments to fit.
Key order changes the answer. “Sending the same content with keys in a different order moved the output probabilities by up to 0.19.” They now sort keys on every request. If you serve a decision model behind an API and compare runs without sorting, you are measuring your serializer.
When this does not work #
The engineers name the boundaries themselves, and they are narrow:
- The task must be a closed choice from a known catalogue .
- The volume must be high .
- The latency of the old path must matter .
- The outputs must not need explanation text.
- The catalogue must not change weekly.
- The useful context must fit into a few thousand tokens.
Break any of those and you are back to a language model, and the likely explanation is that you were paying for reasoning you were not using.
Why this matters beyond Amazon #
Decision models became a recognised class only this autumn, after TypeSafe AI’s Jev, and the pitch has been mostly about latency. This is the first production account with token counts, instance hours, a rollback plan and a failure mode that includes the words “keys in a different order”.
The lesson for anyone running a classification agent on a frontier API is not that a 4B model is magic. It is that the frontier model was doing two jobs, and only one of them was ever necessary. Strip out the generation, keep the decision, and the bill for a million tickets goes from a salary to a rounding error.
If you have a phone, a 24GB card, or a spare A10G, the experiment is a weekend: export a thousand labelled examples, build a choice question with the labels as options, and see whether a 4B decision model with a confidence threshold beats the prompt you are paying for.