Spring AI Prompt Caching and Chat Memory: Where the Tokens Go — LLM Cost Control 2/4

Spring AI offers controls for output token generation and chat memory to manage LLM costs, as detailed in a developer's blog post. The post highlights that output tokens cost several times more than input tokens, with reasoning models adding hidden 'thinking' tokens, and recommends using ChatOptions.maxTokens() and provider-specific options like OpenAI's reasoningEffort to limit costs. It also warns that Ollama models like qwen3 and deepseek-r1 use reasoning by default, which can inflate costs when moved to billed providers.

Suppose the metrics are in place, each task has the model it actually needs, and every feature has its own client — that was Part 1 https://dev.to/julia denysova/spring-ai-token-usage-measure-cost-before-you-pick-a-model-llm-cost-control-14-41fo . The next thing to look at is what those clients send and receive. This part is about controls for output token generation, chat memory and static input tokens. Prices: list prices where a ratio matters, an example rate of $1 per million input tokens elsewhere. Full note in Part 1 . Every token the model generates costs several times more than a token you send it. Reasoning models also generate hidden "thinking" tokens, and these are billed at the same, higher output rate. The gap is on every price sheet. On OpenAI's list https://developers.openai.com/api/docs/pricing August 2026 , gpt-5 input costs $1.25 per million tokens and output costs $10 — an 8× difference. On Anthropic's price sheet https://claude.com/pricing api , Sonnet 5 is $2 in and $10 out — a 5× difference. That is introductory pricing to 31 August 2026; the standard $3/$15 keeps the same 5× ratio. The exact numbers change often, but the pattern does not: output tokens have cost several times more than input tokens for years. Reasoning makes this worse. A model may use 2,000 thinking tokens to produce a 200-token answer, so you pay for 2,200 output tokens in total — eleven times the text the user actually sees. A verbose model on a busy endpoint can end up costing more than all your input traffic combined. Spring AI https://docs.spring.io/spring-ai/reference/index.html gives you two controls here. The first works everywhere: ChatOptions.builder .maxTokens https://docs.spring.io/spring-ai/docs/current/api/org/springframework/ai/chat/prompt/ChatOptions.html provides a portable completion-length limit, when supported by the provider. Think of it as a safety net against runaway responses, not a tool for improving quality — a cut-off answer is still billed in full, so pair the limit with prompt instructions that ask for a short answer. The second control is provider-specific and kept isolated in that provider's options object: OpenAI has a // Provider-independent token limit ChatOptions.Builder capped = ChatOptions.builder .maxTokens 400 ; // Provider-specific reasoning control, isolated in one options object OpenAiChatOptions.Builder lowEffort = OpenAiChatOptions.builder .model "gpt-5-mini" .reasoningEffort "low" .maxCompletionTokens 400 ; // Ollama: disable reasoning for thinking-capable models OllamaOptions.Builder noThink = OllamaOptions.builder .model "qwen3" .think false ; One 2.0 upgrade note belongs here: the Anthropic module's maxTokens default rose from 500 to 4096 https://docs.spring.io/spring-ai/reference/upgrade-notes.html anthropic module . If you relied — even without noticing it — on the old 500-token cap to limit the cost of responses, they can now run up to eight times longer after the upgrade. Set the limit explicitly if you want to keep the old behaviour. One trap deserves its own warning. Several popular Ollama models https://ollama.com/search — including qwen3 and deepseek-r1 — use reasoning by default. During local development there is no per-token API bill, so it is easy to miss that a prompt pattern has become dependent on lengthy reasoning. Move the same prompts to a provider that bills reasoning tokens, and that hidden reasoning can become part of your completion cost. For simple workloads such as extraction, classification, or reformatting, disable or limit thinking during local development as well. This keeps token usage, latency, and production costs easier to predict. LLMs have no built-in memory, so "memory" in practice means sending the full conversation history with every request. Every past message is billed again, as if it were new input. The cost grows faster than you might expect. Assume a 500-token system prompt, roughly 50-token user messages, and roughly 200-token replies. Each finished turn adds about 250 tokens of history, and every later turn has to carry that history too. A message window bounds how much of that history each request includes. Two details must be noted here: the window counts the stored messages, so ten messages means the last five complete user–assistant exchanges, and the current user message always travels outside the window. From turn 6 onwards, every request is therefore the same size: 500 + 1,250 + 50 = 1,800 input tokens. | Turn | Input tokens, unbounded history | Input tokens, window = 10 messages | |---|---|---| | 1 | 550 | 550 | | 5 | 1,550 | 1,550 | | 6 | 1,800 | 1,800 | | 20 | 5,300 | 1,800 | | 50 | 12,800 | 1,800 | Whole 50-turn session | ≈ 333,750 | ≈ 86,250 | Without a limit, the cost of each turn grows in a straight line, but the cost of the whole session grows much faster than that: a 20-turn session sends about 58,500 input tokens in total, and a 50-turn support chat about 333,750 — for a single user. The same 50-turn chat with a 10-message window sends about 86,250, roughly a quarter of the unbounded total. One Spring AI 2.0 detail is worth knowing here: eviction now removes whole turns https://docs.spring.io/spring-ai/reference/api/chat-memory.html turn boundary eviction . A turn starts at a user message, so the kept window always begins with one, and maxMessages is an upper bound rather than a guarantee. For plain chat, choose an even window size so it maps cleanly onto complete exchanges. Spring AI's control here is MessageWindowChatMemory https://docs.spring.io/spring-ai/docs/current/api/org/springframework/ai/chat/memory/MessageWindowChatMemory.html , a sliding window of at most N messages, added through a memory advisor: @Bean ChatClient chatClient ChatClient.Builder builder, ChatMemoryRepository repository { ChatMemory memory = MessageWindowChatMemory.builder .chatMemoryRepository repository .maxMessages 10 // overrides the default 20-message window .build ; return builder .defaultAdvisors MessageChatMemoryAdvisor.builder memory .build .build ; } The service that injects this client selects the conversation on each call: chatClient.prompt .user message // mandatory in Spring AI 2.0 — there is no default conversation id .advisors a - a.param ChatMemory.CONVERSATION ID, sessionId .call .content ; ChatMemory https://docs.spring.io/spring-ai/docs/current/api/org/springframework/ai/chat/memory/ChatMemory.html and ChatMemoryRepository maxMessages as a cost-control decision, not just a convenience setting: a 20-message window from short Q&A may be cheap, while the same window from long conversations can add thousands of tokens to every request. MessageWindowChatMemory limits the number of messages, not tokens, so choose the window size based on the typical token size of your conversations and the context you actually need.Two things to keep in mind. First, a smaller window trades cost for what the model remembers. The model genuinely forgets removed turns, so choose the smallest window your use case can accept, not simply the smallest window possible. Second, the window affects Driver 5 too: a stable start to the prompt system prompt, then the oldest history is what makes provider-side caching work well. Removing messages aggressively, in a way that changes the start of the prompt on every turn, can cost you the cache discount. For very long sessions, VectorStoreChatMemoryAdvisor https://docs.spring.io/spring-ai/reference/api/chat-memory.html vectorstorechatmemoryadvisor is an alternative. It stores history in a vector store and adds back only the messages relevant to the current question, so the input size per turn stays flat no matter how long the session runs. The trade-off: every message has to be turned into an embedding and stored The system prompt, tool definitions including schemas , and few-shot examples are often the same across requests, and they are part of the model input context. Without caching, providers generally process and charge these repeated input tokens on every request. Prompt caching reduces this cost: supported providers store already-processed prompt prefixes and apply a lower input-token rate when the same content is reused. Every provider implements caching differently, with its own rules for eligibility, expiration, and configuration. Spring AI exposes named caching strategies — SYSTEM ONLY , TOOLS ONLY , SYSTEM AND TOOLS , and CONVERSATION HISTORY — through provider-specific enums for Anthropic https://docs.spring.io/spring-ai/reference/api/chat/anthropic-chat.html caching strategies and AWS Bedrock https://docs.spring.io/spring-ai/reference/api/chat/bedrock-converse.html cache strategies . These strategies define where Spring AI places cache breakpoints while respecting provider limitations, but cache lifetime and expiration remain managed by the underlying provider: AnthropicChatOptions.builder .cacheOptions AnthropicCacheOptions.builder .strategy AnthropicCacheStrategy.SYSTEM AND TOOLS .build Anthropic prompt caching charges https://platform.claude.com/docs/en/about-claude/pricing a higher price for cache writes and a lower price for cache reads, which makes the feature valuable for workloads with repeated, stable prompts. For example, at $1 per million input tokens, a 2,000-token system prompt sent 300,000 times per month costs $600 without caching; with a high cache hit rate and 0.1× cached reads, the cost can drop to roughly $60. The main things to watch are cache lifetime and traffic patterns: caches expire, and low-traffic endpoints may pay cache-write costs without enough cache hits to recover the overhead. For agent-style workloads, there is one more Anthropic-specific setting worth knowing: cacheToolResults https://docs.spring.io/spring-ai/reference/api/chat/anthropic-chat.html caching tool results . With CONVERSATION HISTORY caching, each tool-calling round adds tool results after the default cache point, so those tool outputs are billed as new, uncached input on later rounds. Enabling cacheToolResults moves the cache point to the last tool result, allowing the next round to read the previous round's often large tool output from the cache instead of processing it again. This pairs directly with tool schemas and agent loops Prompts of at least 1,024 tokens are cached automatically, with no code changes. Two things changed with the GPT-5.6 family, and both affect the bill https://developers.openai.com/api/docs/guides/prompt-caching . First, cache writes are billed at 1.25× the uncached input rate. On earlier models they were free. Second, the service caches exact prefixes at breakpoints. By default it places one implicit breakpoint at the latest user or tool message https://developers.openai.com/api/docs/guides/prompt-caching caching-behavior-changes-when-migrating-to-gpt-56 , and it no longer falls back to the longest matching prefix before that point. A request can therefore share thousands of identical tokens with the previous one, report zero cached tokens, and pay to write the changing prefix again. The lever Spring AI gives you is the cache key. Requests that share a prefix should carry the same one: OpenAiChatOptions.Builder shared = OpenAiChatOptions.builder .model "gpt-5.6-terra" .promptCacheKey "support-assistant-v1" ; or spring.ai.openai.chat.prompt-cache-key=support-assistant-v1 https://docs.spring.io/spring-ai/reference/api/chat/openai-chat.html configuration properties . On GPT-5.6 and later, this key is required for the more reliable matching. Keep traffic per key to roughly Spring AI 2.0 does not expose explicit cache breakpoints, so on OpenAI the cacheable prefix is whatever your prompt structure gives you — which makes the static-first rule below a cost control, not a style preference. It also reports cache reads but not cache writes for OpenAI https://docs.spring.io/spring-ai/reference/upgrade-notes.html unified cache usage metrics on usage , so the 1.25× write charge is invisible in your token metrics. Watch it on the provider's dashboard instead. Local models also use a cache, but it saves time, not money. Before a model can answer, it must first process every token of the prompt — an expensive "reading" step on the GPU. The engine keeps the result of this step in memory the KV cache . If the next request begins with exactly the same text, the engine skips that part and processes only what is new. The result: a faster first token and more requests per GPU. But nothing on a bill gets smaller, because locally there are no per-token charges. The cache also lives only while the model is loaded — Ollama unloads idle models after five minutes, though keep alive can hold them in memory for longer. Note that the cache only matches from the start of the prompt: at the first token that differs, everything after it is processed again. So the static-first rule applies here too. Provider prompt caching mechanisms depend on matching the beginning of the prompt. A cache hit requires an identical prefix, so fixed content — instructions, examples, and tool definitions — should be placed before changing content. A single timestamp placed at the top of the system prompt can invalidate the cached prefix that follows it. Spring AI 2.0 supports separating static and dynamic SystemMessage blocks for providers that support multi-block system caching, such as Anthropic https://docs.spring.io/spring-ai/reference/api/chat/anthropic-chat.html multi block system caching and AWS Bedrock Converse https://docs.spring.io/spring-ai/reference/api/chat/bedrock-converse.html multi block system message caching . With .multiBlockSystemCaching true , Spring AI can preserve cacheable system blocks while allowing later changing system content to remain outside the cached prefix. Structure your prompts this way even before you turn caching on — it is what makes every provider's cache, automatic or explicit, actually work for you. There is a limit to how far prompt structure can take you, though. Everything in this part assumed the content was yours: your system prompt, your conversation, your examples. The next part deals with the context your application adds automatically — document chunks from a vector store and tool definitions from every server you connect to. Both arrive as input tokens, both are sent whether the model uses them or not, and both scale with how much you have indexed rather than with how much you need. Part 3 is coming.