Our AI pipeline runs three step kinds (ai.generate
, ai.extract
, ai.classify
), each independently resolving its own provider, model, and prompt revision at run time. The default model is Sonnet, not Opus — and for a while that felt like a compromise, because Opus was the expensive-but-reliable option and Sonnet needed babysitting to hit the same pass rate.
The fix that closed the gap wasn't a smarter prompt. It was tool_choice
plus a tightened output schema, forcing the model to commit to an answer shape instead of spending tokens hedging its way there. That alone got Sonnet to pass-parity with Opus's prior output at roughly a quarter of the cost, in our eval. A second, separate lever: Anthropic's own recommended effort: "xhigh"
for agentic tasks produced roughly twice the thinking tokens of `"high"`
for the same pass rate — thinking tokens bill like output tokens, so that's a straight 2x for zero accuracy gain, and it only matters when Opus is used via an explicit override (Sonnet doesn't get adaptive thinking at all). A third, independent lever: `max_tokens`
defaulted to 16000 out of caution; dropping it to 4000 (still comfortably above what any real step needed) cut effective output cost again, because Anthropic bills against the cap as an upper bound in some failure paths, not only against what the model actually emits.
None of these three touched the prompt content or the provider. All three came from looking at per-step token usage, which only exists because every step writes an immutable provenance record on completion: prompt revision, provider, model, token usage. Same idea as event sourcing, applied to LLM calls instead of domain writes. You don't trust "the pipeline probably used prompt v3, on whatever model was configured," you have a row that says so — and you can audit six months of "which prompt touched this tenant" without ever the actual LLM payloads (also the DSGVO-friendly shape: provenance rows carry no user content, only call metadata).
The other real trap, found the hard way: adaptive thinking and forced tool-use don't mix. Anthropic 400s with "Thinking may not be enabled when tool_choice forces tool use" the moment both are set, which sounds obvious in hindsight but not when you're setting both because both individually sound like "make the model try harder." The fix has to be automatic — detect a forced tool_choice
and disable adaptive thinking before the request goes out — because leaving it to every caller to remember means every eval run using toolChoice: { type: "any" }
against an Opus override breaks the same way, all at once.
Separately: prompt caching only pays off if the cache_control
breakpoint sits on the last static block, with tools and system prompt as one cacheable prefix. Past roughly 16k tokens without it, requests start hitting the SDK's HTTP timeout before the response streams back far enough to matter. Put the breakpoint on something that changes per request and you get a silent cache-miss that looks identical to a hit in the logs while you keep paying full price.
Full writeup with the pipeline architecture (with diagrams), the provider-resolution code, and the provenance shape: docs.kumiko.rocks/en/guides/ai-pipeline-provenance.