In this article, you will learn how prompt caching and fine-tuning differ as strategies for reducing cost and latency in agentic AI systems, and how to choose between them.
Topics we will cover include:
- What prompt caching is, how it works, and when it reduces costs and latency most effectively.
- What fine-tuning is, why parameter-efficient methods like LoRA keep compute costs manageable, and when it is the right tool for the job.
- A practical decision framework for applying prompt caching, fine-tuning, or a hybrid of both to your agentic architecture.
Introduction #
Agentic AI systems have long been limited to prototypes, but recent parallel advances in trends like large language models (LLMs) have fostered significant progress and a dramatic push of these systems to production. Two bottlenecks unavoidably arise as a consequence of this shift: rising API costs and increasing —sometimes unacceptable— latency. Simply put, modern autonomous agents rely on iterative LLM calls to plan, execute actions, and reflect on them. Thus, optimizing the underlying infrastructure that makes this possible becomes imperative to also make it sustainable.
This article provides a breakdown of two concepts or strategies that are closely related to mitigating the two aforesaid issues, highlighting how they differ: prompt caching and fine-tuning. Likewise, we present a decision framework for combining them to construct applications that are both high-performing and cost-effective.
Understanding Prompt Caching and Fine-Tuning in LLMs and Agentic AI #
Let’s first demystify the two core concepts underlying the subsequent decision framework for cost and latency optimization.
1. Prompt Caching
Prompt caching involves safeguarding information from previous model interactions — from now on, by model we refer to the LLM. This can be done either by storing the raw outputs of previously sent prompts or the model’s internal attention states (also known as KV caching). Accordingly, if an agent (or user) sends the model a prompt that closely resembles a cached one, a data retrieval mechanism is leveraged rather than recomputing everything from scratch before generating the response.
The direct advantages of prompt caching include a significant reduction in Time to First Token (TTFT) —the time elapsed until the response starts being generated as a result of prior computation— and a reduction in compute costs to near zero for largely repeated requests.
Let this simplified Python implementation using diskcache
serve to illustrate the purpose and rationale behind prompt caching in practice:
import diskcache
import hashlib
cache = diskcache.Cache('./llm_cache')
def get_cached_llm_response(prompt, mock_api_call):
prompt_hash = hashlib.md5(prompt.encode()).hexdigest()
if prompt_hash in cache:
return cache[prompt_hash], "Cache Hit - 0ms latency, $0 cost"
response = mock_api_call(prompt)
cache.set(prompt_hash, response, expire=3600) # Cache for 1 hour
return response, "Cache Miss - Standard latency and cost applied"
print(get_cached_llm_response("Translate 'Hello' to Spanish", lambda x: "Hola"))
1234567891011121314151617181920
import diskcacheimport hashlib # Initializing a free, local persistent cachecache = diskcache.Cache('./llm_cache') def get_cached_llm_response(prompt, mock_api_call): # Hashing the prompt to create a unique identifier prompt_hash = hashlib.md5(prompt.encode()).hexdigest() if prompt_hash in cache: return cache[prompt_hash], "Cache Hit - 0ms latency, $0 cost" # If not in cache, call the LLM and store the result: the model is mocked for simplicity response = mock_api_call(prompt) cache.set(prompt_hash, response, expire=3600) # Cache for 1 hour return response, "Cache Miss - Standard latency and cost applied" # Example of useprint(get_cached_llm_response("Translate 'Hello' to Spanish", lambda x: "Hola"))
The first time you execute the code, there won’t be any cached information, so standard latency and costs will apply. From the second execution onwards, however, you will hit the cache and save those costs. No actual model or agent is used here, but the key ideas behind prompt caching are reflected in the example above.
In sum, caching is an effective approach to making agent and LLM-based architectures more budget-friendly and efficient.
2. Fine-Tuning
Fine-tuning consists of having the model learn specific agent or user behaviors, formatting rules, and new domain knowledge, so that instead of repeatedly sending massive instruction sets and context as part of a prompt, the knowledge is used to directly update the model’s weights. To avoid the high costs of a full-parameter model retraining, there exist specific techniques like Parameter-Efficient Fine-Tuning (PEFT), among which LoRA (Low-Rank Adaptation) has gained special popularity.
The following code illustrates the use of LoRA on a transformers
model from Hugging Face and shows the percentage of actual parameters being retrained. Make sure you run pip install --upgrade torchao
first to ensure a smooth run:
from transformers import AutoModelForCausalLM
from peft import get_peft_model, LoraConfig
model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
lora_config = LoraConfig(
r=8,
lora_alpha=32,
target_modules=["q_proj", "v_proj"],
bias="none",
task_type="CAUSAL_LM"
)
efficient_model = get_peft_model(model, lora_config)
efficient_model.print_trainable_parameters()
1234567891011121314151617181920
from transformers import AutoModelForCausalLMfrom peft import get_peft_model, LoraConfig # a fully open, ungated base modelmodel = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") # Configuring LoRA to train only a tiny fraction of parameterslora_config = LoraConfig( r=8, lora_alpha=32, target_modules=["q_proj", "v_proj"], bias="none", task_type="CAUSAL_LM") # Applying the adapter to the modelefficient_model = get_peft_model(model, lora_config) # Notice how few parameters actually need training, keeping compute costs lowefficient_model.print_trainable_parameters()
Output:
trainable params: 1,126,400 || all params: 1,101,174,784 || trainable%: 0.1023
1
trainable params: 1,126,400 || all params: 1,101,174,784 || trainable%: 0.1023
Cost-Latency Decision Framework #
How do you find the right balance between these two strategies to optimize cost and latency, or how do you combine them? Ultimately, it depends on the nature of your data and the intended behavior of your agent-based system.
Focus on prompt caching when:
- You have massive system prompts, a static document base for RAG, or standard operating procedures repeatedly required by the agent. Caching them all as a prompt prefix saves significant token costs.
- You are working on applications like customer support where nearly identical questions are routinely encountered.
- You seek a drastic reduction in latency (TTFT) and direct token billing costs.
Focus on fine-tuning when:
- The agent must ensure consistent output formatting, e.g. strict JSON, SQL, or other specialized code. Fine-tuning eliminates the need to supply extensive few-shot examples for this purpose.
- You want your model to “sound” a certain way (persona customization) without being constantly reminded through added prompt instructions.
- You seek a drastic reduction in the required context window per request, making repeated LLM calls cheaper and faster.
Adopt a balanced, hybrid approach when:
- You want a resilient agentic architecture overall, built on state-of-the-art standards.
- You can achieve this by first fine-tuning a smaller, open-source model (see the second example above), then implementing prompt caching to handle the agent’s system instructions and scratchpad, so that as it loops through actions and thoughts, it only needs to compute the newest tokens.
Closing Remarks #
As we have seen, prompt caching primarily scales down the costs associated with redundant contexts, while fine-tuning solidly tackles the challenge of adopting repetitive behavior. The best and most scalable approach when it comes to these two strategies boils down to mastering the interplay between them.