Spring AI Prompt Caching and Chat Memory: Where the Tokens Go — LLM Cost Control 2/4

Suppose the metrics are in place, each task has the model it actually needs, and every feature has its own client — that was Part 1. The next thing to look at is what those clients send and receive.

This part is about controls for output token generation, chat memory and static input tokens.

Prices: list prices where a ratio matters, an example rate of $1 per million

input tokens elsewhere.[Full note in Part 1].

Every token the model generates costs several times more than a token you send it. Reasoning models also generate hidden "thinking" tokens, and these are billed at the same, higher output rate.

The gap is on every price sheet. On OpenAI's list (August 2026), gpt-5

input costs $1.25 per million tokens and output costs $10 — an 8× difference. On Anthropic's price sheet, Sonnet 5

is $2 in and $10 out — a 5× difference. (That is introductory pricing to 31 August 2026; the standard $3/$15 keeps the same 5× ratio.)

The exact numbers change often, but the pattern does not: output tokens have cost several times more than input tokens for years. Reasoning makes this worse. A model may use 2,000 thinking tokens to produce a 200-token answer, so you pay for 2,200 output tokens in total — eleven times the text the user actually sees. A verbose model on a busy endpoint can end up costing more than all your input traffic combined.

Spring AI gives you two controls here. The first works everywhere: ChatOptions.builder().maxTokens() provides a portable completion-length limit, when supported by the provider. Think of it as a safety net against runaway responses, not a tool for improving quality — a cut-off answer is still billed in full, so pair the limit with prompt instructions that ask for a short answer. The second control is provider-specific and kept isolated in that provider's options object: OpenAI has a

// Provider-independent token limit 
ChatOptions.Builder capped = ChatOptions.builder()
    .maxTokens(400);

// Provider-specific reasoning control, isolated in one options object
OpenAiChatOptions.Builder lowEffort = OpenAiChatOptions.builder()
    .model("gpt-5-mini")
    .reasoningEffort("low")
    .maxCompletionTokens(400);

// Ollama: disable reasoning for thinking-capable models
OllamaOptions.Builder noThink = OllamaOptions.builder()
    .model("qwen3")
    .think(false);

One 2.0 upgrade note belongs here: the Anthropic module's maxTokens default rose from 500 to 4096. If you relied — even without noticing it — on the old 500-token cap to limit the cost of responses, they can now run up to eight times longer after the upgrade. Set the limit explicitly if you want to keep the old behaviour.

One trap deserves its own warning. Several popular Ollama models — including qwen3

and deepseek-r1

— use reasoning by default. During local development there is no per-token API bill, so it is easy to miss that a prompt pattern has become dependent on lengthy reasoning. Move the same prompts to a provider that bills reasoning tokens, and that hidden reasoning can become part of your completion cost. For simple workloads such as extraction, classification, or reformatting, disable or limit thinking during local development as well. This keeps token usage, latency, and production costs easier to predict.

LLMs have no built-in memory, so "memory" in practice means sending the full conversation history with every request. Every past message is billed again, as if it were new input.

The cost grows faster than you might expect. Assume a 500-token system prompt, roughly 50-token user messages, and roughly 200-token replies. Each finished turn adds about 250 tokens of history, and every later turn has to carry that history too. A message window bounds how much of that history each request includes. Two details must be noted here: the window counts the stored messages, so ten messages means the last five complete user–assistant exchanges, and the current user message always travels outside the window. From turn 6 onwards, every request is therefore the same size: 500 + 1,250 + 50 = 1,800 input tokens.

Without a limit, the cost of each turn grows in a straight line, but the cost of the whole session grows much faster than that: a 20-turn session sends about 58,500 input tokens in total, and a 50-turn support chat about 333,750 — for a single user. The same 50-turn chat with a 10-message window sends about 86,250, roughly a quarter of the unbounded total.

One Spring AI 2.0 detail is worth knowing here: eviction now removes whole turns. A turn starts at a user message, so the kept window always begins with one, and maxMessages is an upper bound rather than a guarantee. For plain chat, choose an even window size so it maps cleanly onto complete exchanges.

Spring AI's control here is MessageWindowChatMemory, a sliding window of at most N messages, added through a memory advisor:

@Bean
ChatClient chatClient(ChatClient.Builder builder,
                      ChatMemoryRepository repository) {

    ChatMemory memory = MessageWindowChatMemory.builder()
        .chatMemoryRepository(repository)
        .maxMessages(10)   // overrides the default 20-message window
        .build();
    return builder
        .defaultAdvisors(MessageChatMemoryAdvisor.builder(memory).build())
        .build();
}

The service that injects this client selects the conversation on each call:

chatClient.prompt()
    .user(message)
    // mandatory in Spring AI 2.0 — there is no default conversation id
    .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, sessionId))
    .call()
    .content();

ChatMemory and

ChatMemoryRepository

maxMessages

as a cost-control decision, not just a convenience setting: a 20-message window from short Q&A may be cheap, while the same window from long conversations can add thousands of tokens to every request. MessageWindowChatMemory

limits the number of messages, not tokens, so choose the window size based on the typical token size of your conversations and the context you actually need.Two things to keep in mind. First, a smaller window trades cost for what the model remembers. The model genuinely forgets removed turns, so choose the smallest window your use case can accept, not simply the smallest window possible. Second, the window affects Driver #5 too: a stable start to the prompt (system prompt, then the oldest history) is what makes provider-side caching work well. Removing messages aggressively, in a way that changes the start of the prompt on every turn, can cost you the cache discount.

For very long sessions, VectorStoreChatMemoryAdvisor is an alternative. It stores history in a vector store and adds back only the messages relevant to the current question, so the input size per turn stays flat no matter how long the session runs. The trade-off: every message has to be turned into an embedding and stored (

The system prompt, tool definitions (including schemas), and few-shot examples are often the same across requests, and they are part of the model input context. Without caching, providers generally process and charge these repeated input tokens on every request. Prompt caching reduces this cost: supported providers store already-processed prompt prefixes and apply a lower input-token rate when the same content is reused. Every provider implements caching differently, with its own rules for eligibility, expiration, and configuration.

Spring AI exposes named caching strategies — SYSTEM_ONLY

, TOOLS_ONLY

, SYSTEM_AND_TOOLS

, and CONVERSATION_HISTORY

— through provider-specific enums for Anthropic and AWS Bedrock. These strategies define where Spring AI places cache breakpoints while respecting provider limitations, but cache lifetime and expiration remain managed by the underlying provider:

AnthropicChatOptions.builder()
    .cacheOptions(AnthropicCacheOptions.builder()
        .strategy(AnthropicCacheStrategy.SYSTEM_AND_TOOLS)
        .build())

Anthropic prompt caching charges a higher price for cache writes and a lower price for cache reads, which makes the feature valuable for workloads with repeated, stable prompts. For example, at $1 per million input tokens, a 2,000-token system prompt sent 300,000 times per month costs $600 without caching; with a high cache hit rate and 0.1× cached reads, the cost can drop to roughly $60. The main things to watch are cache lifetime and traffic patterns: caches expire, and low-traffic endpoints may pay cache-write costs without enough cache hits to recover the overhead.

For agent-style workloads, there is one more Anthropic-specific setting worth knowing: cacheToolResults. With

CONVERSATION_HISTORY

caching, each tool-calling round adds tool results after the default cache point, so those tool outputs are billed as new, uncached input on later rounds. Enabling cacheToolResults

moves the cache point to the last tool result, allowing the next round to read the previous round's (often large) tool output from the cache instead of processing it again. This pairs directly with tool schemas and agent loops (Prompts of at least 1,024 tokens are cached automatically, with no code changes. Two things changed with the GPT-5.6 family, and both affect the bill.

First, cache writes are billed at 1.25× the uncached input rate. On earlier models they were free.

Second, the service caches exact prefixes at breakpoints. By default it places one implicit breakpoint at the latest user or tool message, and it no longer falls back to the longest matching prefix before that point. A request can therefore share thousands of identical tokens with the previous one, report zero cached tokens, and pay to write the changing prefix again.

The lever Spring AI gives you is the cache key. Requests that share a prefix should carry the same one:

OpenAiChatOptions.Builder shared = OpenAiChatOptions.builder()
    .model("gpt-5.6-terra")
    .promptCacheKey("support-assistant-v1");

or spring.ai.openai.chat.prompt-cache-key=support-assistant-v1. On GPT-5.6 and later, this key is required for the more reliable matching. Keep traffic per key to roughly

Spring AI 2.0 does not expose explicit cache breakpoints, so on OpenAI the cacheable prefix is whatever your prompt structure gives you — which makes the static-first rule below a cost control, not a style preference. It also reports cache reads but not cache writes for OpenAI, so the 1.25× write charge is invisible in your token metrics. Watch it on the provider's dashboard instead.

Local models also use a cache, but it saves time, not money. Before a model can answer, it must first process every token of the prompt — an expensive "reading" step on the GPU. The engine keeps the result of this step in memory (the KV cache). If the next request begins with exactly the same text, the engine skips that part and processes only what is new. The result: a faster first token and more requests per GPU. But nothing on a bill gets smaller, because locally there are no per-token charges. The cache also lives only while the model is loaded — Ollama unloads idle models after five minutes, though keep_alive

can hold them in memory for longer.

Note that the cache only matches from the start of the prompt: at the first token that differs, everything after it is processed again. So the static-first rule applies here too.

Provider prompt caching mechanisms depend on matching the beginning of the prompt. A cache hit requires an identical prefix, so fixed content — instructions, examples, and tool definitions — should be placed before changing content. A single timestamp placed at the top of the system prompt can invalidate the cached prefix that follows it. Spring AI 2.0 supports separating static and dynamic SystemMessage

blocks for providers that support multi-block system caching, such as Anthropic and AWS Bedrock Converse. With .multiBlockSystemCaching(true)

, Spring AI can preserve cacheable system blocks while allowing later changing system content to remain outside the cached prefix.

Structure your prompts this way even before you turn caching on — it is what makes every provider's cache, automatic or explicit, actually work for you.

There is a limit to how far prompt structure can take you, though. Everything in this part assumed the content was yours: your system prompt, your conversation, your examples. The next part deals with the context your application adds automatically — document chunks from a vector store and tool definitions from every server you connect to. Both arrive as input tokens, both are sent whether the model uses them or not, and both scale with how much you have indexed rather than with how much you need.

Part 3 is coming.