# Agent-Cache: Multi-Tier LLM Caching for Valkey and Redis

> Source: <https://dev.to/mech_app_ai/agent-cache-multi-tier-llm-caching-for-valkey-and-redis-10kf>
> Published: 2026-09-14 00:11:50+00:00

Agent loops burn tokens and clock cycles on repeated work. The same LLM prompt fires twice, the same tool call fetches identical data, and session state gets reconstructed from scratch on every request. Agent-cache solves this with a three-tier caching architecture backed by Valkey or Redis, putting LLM responses, tool results, and session snapshots behind a single connection.

The project shipped v0.1.0 with Valkey 7+ and Redis 6.2+ support, then v0.2.0 with cluster mode the next day. It includes framework adapters for LangChain, LangGraph, and Vercel AI SDK, plus OpenTelemetry and Prometheus instrumentation at the cache layer.

Most agent caching solutions pick one layer and stop. LangChain caches LLM completions. LangGraph persists checkpoints. Agent-cache handles all three:

**Tier 1: LLM Response Cache**

Exact-match cache keyed on prompt text and model parameters. If your agent calls `gpt-4o` with identical input twice, the second call returns from Valkey in under 1ms instead of hitting the API. This is the biggest token saver when agents retry or loop over similar prompts.

**Tier 2: Tool Output Cache**

Caches function call results keyed on tool name and arguments. If `get_weather("Sofia")` runs twice with the same parameters, the cached result comes back instantly. Useful when agents re-invoke tools during backtracking or multi-step reasoning.

**Tier 3: Session State Cache**

Stores agent checkpoints, user intent, and execution state with per-field TTL. This is where LangGraph checkpoints live, along with any custom state your orchestrator needs to resume mid-flow.

Each tier uses a different TTL strategy. LLM responses might cache for hours if the model and prompt are stable. Tool outputs expire faster when external data changes frequently. Session state persists only as long as the user session is active.

The cache key for LLM responses combines prompt hash, model name, temperature, and top-p. This means changing any parameter busts the cache. If you tweak the system prompt or adjust temperature, the agent hits the API again.

Tool output keys include the function name and a hash of the arguments object. This works for deterministic tools (database lookups, API calls with stable responses) but breaks down when tools mutate external state. If your agent calls `create_ticket(title, description)`, caching the result means subsequent calls with the same arguments return the old ticket ID instead of creating a new one.

The invalidation strategy is manual. Agent-cache does not track dependencies between cache entries. If a tool mutates state that affects future LLM calls, you must explicitly invalidate the relevant keys. The library exposes a `cache.invalidate(pattern)` method that accepts Redis glob patterns, but you need to know which keys to target.

When Valkey or Redis becomes unavailable, the agent has three options:

Agent-cache defaults to option 2 (graceful degradation) but lets you configure the behavior per tier. You can fail fast on session state loss while degrading gracefully for LLM and tool caches.

The observability layer helps here. OpenTelemetry spans track cache hits, misses, and errors. Prometheus metrics expose hit rate, latency, and connection pool health. If your cache hit rate drops suddenly, you know to check Redis availability or inspect your invalidation logic.

The library ships adapters for three frameworks:

**LangChain**: Wraps the `BaseLLMCache` interface. Drop it into your chain config and LLM calls automatically cache.

**LangGraph**: Implements the `BaseCheckpointSaver` interface. Checkpoints persist to Valkey instead of requiring Redis 8 with modules.

**Vercel AI SDK**: Hooks into the `streamText` and `generateText` APIs. Streaming support is on the roadmap but not yet shipped.

Each adapter handles serialization differently. LangChain uses JSON for LLM responses. LangGraph uses MessagePack for checkpoints to save space. Tool outputs serialize as JSON by default but accept custom serializers if you need binary formats.

Agent-cache assumes you already run Valkey or Redis in production. It does not bundle a server or manage deployment. You point it at an existing instance (standalone, sentinel, or cluster) and it handles connection pooling.

Cluster mode support landed in v0.2.0. The library uses hash tags to ensure related keys (LLM response + tool outputs for the same agent run) land on the same shard. This avoids cross-shard transactions and keeps latency predictable.

For high-availability setups, use Redis Sentinel or Valkey's built-in replication. The client automatically fails over to a replica if the primary goes down. Session state might lag by a few seconds during failover, but LLM and tool caches remain available.

| Concern | Risk | Mitigation | 
|---|---|---|
| Stale tool outputs | Cached results don't reflect external state changes | Use short TTLs or invalidate on mutation | 
| Cache key collisions | Different prompts hash to the same key | Include model params and full prompt in key | 
| Memory pressure | Large LLM responses fill Redis memory | Set max memory policy to `allkeys-lru` | 
| Session state loss | Redis restart wipes active sessions | Persist RDB snapshots or use AOF | 
| Observability overhead | Tracing every cache hit adds latency | Sample traces at 1% in production | 

The biggest risk is treating the cache as a source of truth. If your agent relies on cached tool outputs to make decisions, and those outputs are stale, the agent acts on outdated information. This is fine for read-only tools (weather lookups, documentation search) but dangerous for tools that mutate state (database writes, API calls with side effects).

``` js
import { AgentCache } from '@betterdb/agent-cache';
import { ChatOpenAI } from 'langchain/chat_models/openai';
import { initializeAgentExecutorWithOptions } from 'langchain/agents';
import { Calculator } from 'langchain/tools/calculator';

const cache = new AgentCache({
  redis: { host: 'localhost', port: 6379 },
  ttl: {
    llm: 3600,        // 1 hour for LLM responses
    tool: 300,        // 5 minutes for tool outputs
    session: 1800     // 30 minutes for session state
  },
  telemetry: {
    otel: true,
    prometheus: { port: 9090 }
  }
});

const model = new ChatOpenAI({
  modelName: 'gpt-4o',
  cache: cache.llmCache()  // Plug into LangChain's cache interface
});

const tools = [new Calculator()];

const executor = await initializeAgentExecutorWithOptions(
  tools,
  model,
  { agentType: 'openai-functions' }
);

// First call hits OpenAI API
const result1 = await executor.call({
  input: 'What is the weather in Sofia?'
});

// Second identical call returns from Valkey in <1ms
const result2 = await executor.call({
  input: 'What is the weather in Sofia?'
});
```

The `cache.llmCache()` method returns an object that implements LangChain's `BaseLLMCache` interface. LangChain automatically checks the cache before calling the model and stores responses after successful completions.

**Use agent-cache when:**

**Avoid it when:**

The library fills a gap between framework-specific caching (LangChain's LLM cache, LangGraph's checkpoint store) and general-purpose Redis usage. It works best when you control the agent loop and can reason about which operations are safe to cache. If your agent is a black box or you don't understand when tools get called, start with observability and measure cache hit rate before committing to this architecture.