Originally published on tamiz.pro.
When a Retrieval-Augmented Generation (RAG) agent or any LLM-backed service produces an answer, the real value rarely lives in the response alone. It lives in the complete context: the original user query, the retrieval results, the system prompt template, the temperature and top-p values, token counts, latency, the model version, and every intermediate tool call or function invocation. Without that record, you cannot reproduce, debug, evaluate, or improve your system.
Prompt archival is the practice of persisting the full execution trace of every AI interaction. It is not merely a logging exercise — it is the foundation of reproducible AI engineering. In this deep-dive, we explore why it matters, what to capture, how to structure storage, and what production patterns actually work at scale.
Traditional software is deterministic by default. Given the same inputs and code, the output is identical. LLM-powered systems break this assumption fundamentally. The same query can produce different outputs across temperature 0 settings if the underlying model weights shift, if the prompt template changes, if the retrieval vector database returns different chunks, or if a rate limiter delays a call just enough to change the context window's contents.
Reproducibility in AI systems means something slightly different than in conventional engineering. It does not guarantee bit-identical outputs across runs. It means:
Without archival, none of this is possible. You are flying blind every time an agent fails, every time a stakeholder asks "why did the model say that," and every time you want to run a proper A/B evaluation.
The first design decision is scope. Archiving everything is expensive and noisy; archiving too little makes the system useless. The industry-standard granularity is the execution trace, which consists of several layers.
Every trace should contain at minimum:
Beyond the trace object, you typically want:
Do not archive raw embeddings unless you have a specific research need. Do not archive PII beyond what is required for your use case, and ensure encryption at rest. Do not store full image payloads unless the vision component is central to your product — store the URL or a content hash instead.
The choice of storage is the single most consequential technical decision in prompt archival. Your system needs to support three access patterns simultaneously: point-in-time reconstruction for debugging, bulk scan for evaluation, and aggregation for cost and quality dashboards.
The most effective production architecture separates concerns across three storage layers:
1. Object store for raw traces (S3, GCS, or equivalent)
Each trace becomes a JSON document stored under a predictable key pattern. Object stores give you near-infinite durability, low cost, and simple consistency. A typical key layout looks like:
traces/<year>/<month>/<day>/<trace-id>.jsonl
sessions/<session-id>/<trace-id>.jsonl
Storing one trace per line in JSONL format means you can stream-read entire days of data without gigabytes into memory. It also means every append is atomic and idempotent.
2. Columnar or wide-column database for query and analytics
PostgreSQL, BigQuery, Snowflake, or ClickHouse give you fast filtering across metadata, cost rollups, and time-range queries. A representative schema might include:
| Column | Type | Purpose |
|---|---|---|
| trace_id | UUID | Primary key |
| session_id | UUID | Grouping key |
| user_id | VARCHAR(128) | Tenant or customer |
| created_at | TIMESTAMPTZ | Time index |
| model | VARCHAR(256) | Model identifier |
| input_tokens | BIGINT | Usage metric |
| output_tokens | BIGINT | Usage metric |
| latency_ms | INTEGER | Performance metric |
| status | VARCHAR(32) | success, error, timeout |
| cost_usd | DECIMAL(10,4) | Billing metric |
| prompt_version | VARCHAR(128) | Template version |
| feedback_score | SMALLINT | Human rating |
You keep the JSON payload as a JSONB
column or as a foreign reference to the object store. This avoids duplication while preserving query performance.
3. Vector store for semantic search over traces
When you need to find past interactions similar to a current bug, you embed the trace's input and output and store them alongside the trace ID. This is what lets you do queries like "show me all cases where the agent confused billing policy with shipping policy."
Your trace schema will change. New fields will be added, old ones deprecated. Design for this from day one:
How traces reach storage is as important as where they land. The ingestion path must be reliable, non-blocking for your application, and resistant to data loss.
Never block the request path on archival. Use one of these patterns:
Fire-and-forget with retries
import asyncio
import httpx
from ulid import ULID
async def enqueue_trace(trace: dict, archive_client: httpx.AsyncClient):
task = asyncio.create_task(_retryable_write(trace, archive_client))
task.add_done_callback(_log_failure)
async def _retryable_write(trace: dict, client: httpx.AsyncClient):
ulid = ULID.from_str(trace["trace_id"])
path = f"traces/{ulid.timestamp().ts.year}/{ulid.timestamp().ts.month:02d}/..."
retries = 3
for attempt in range(retries):
try:
await client.put(
f"https://archive.example.com/{path}.jsonl",
json=trace,
headers={"Content-Type": "application/jsonl"},
timeout=10,
)
return
except httpx.TimeoutException:
if attempt == retries - 1:
await _dead_letter(trace)
await asyncio.sleep(0.5 * (2 ** attempt))
Buffered batching
For high-throughput systems, batch traces into 500-1000 row chunks before writing. This reduces object store operations and cuts cost. Flush on a timer (every 30 seconds) or on buffer threshold, whichever comes first.
If your archival service lags, you risk losing data or corrupting ordering. Implement:
Duplicate writes are inevitable in distributed systems. Make your archival layer idempotent:
If you are building agents with LangChain, LangGraph, CrewAI, or custom frameworks, integration points vary but the principle is the same: instrument at the boundary between your code and the model.
Lowest level: wrap the LLM client
Intercept calls at the provider interface. This captures everything uniformly regardless of which framework orchestrates the agent. For OpenAI-compatible clients:
import { OpenAI } from "openai";
class TracedOpenAI extends OpenAI {
async chat completions.create(
params: Parameters<OpenAI.Chat.Completions>["create"],
options?: Parameters<OpenAI.Chat.Completions>["create"][1]
) {
const traceId = generateULID();
const start = Date.now();
try {
const response = await super.chat.completions.create(params, options);
await archiveTrace({
trace_id: traceId,
model: params.model,
input: params.messages,
output: response.choices[0].message,
usage: response.usage,
latency_ms: Date.now() - start,
config: { temperature: params.temperature, ... },
});
return response;
} catch (error) {
await archiveTrace({
trace_id: traceId,
model: params.model,
input: params.messages,
error: { message: error.message, type: error.type },
latency_ms: Date.now() - start,
});
throw error;
}
}
}
Mid level: framework callbacks
LangChain's Tracer
interface, LangGraph's built-in callbacks, and CrewAI's observability hooks let you capture tool calls, retrieval steps, and multi-agent handoffs automatically. Use these when available — they reduce the chance of missing intermediate states.
Highest level: session-level composition
Build a decorator or context manager that wraps entire agent runs, composing the trace from fragments emitted by the lower layers. This is where you add business context: which user asked, which feature flag was active, which prompt version was loaded.
Streaming responses complicate archival because the trace is not complete until the stream closes. Handle this by buffering chunks in memory and assembling the final message when the stream ends or when an error occurs. For multi-turn sessions, maintain a session-level trace accumulator that attaches each turn's sub-trace to the parent session ID.
from contextlib import contextmanager
import uuid
@contextmanager
def traced_session(session_id: str):
trace = {
"session_id": session_id,
"turns": [],
"started_at": datetime.now(timezone.utc),
}
try:
yield trace
finally:
trace["ended_at"] = datetime.now(timezone.utc)
archive_session(trace)
RAG adds a critical dimension: the retrieval step must be archived alongside the generation step. Two agents answering the same question with different retrieved documents are fundamentally different executions, even if their prompts look identical.
For each retrieval call, store:
The generation trace must reference the retrieval trace. Use a parent-child relationship via trace IDs:
{
"trace_id": "01HVXK...",
"type": "generation",
"parent_trace_id": "01HVXJ...",
"retrieval_context": {
"num_chunks": 5,
"chunk_ids": ["c1", "c2", ...],
"query_embedding_model": "text-embedding-3-small"
}
}
This link is essential for debugging. When an agent hallucinates, you need to know whether the source material was missing, irrelevant, or misinterpreted.
Archival without prompt versioning is nearly useless. If you change your system prompt and then see degraded outputs, you cannot tell whether the degradation came from the new prompt or from a model update unless you captured the prompt version with every trace.
Treat prompts as code. Assign each prompt template a semantic version. When you resolve template variables, store the resolved string alongside the template version and a content hash:
from hashlib import sha256
def resolve_and_record(template_version: str, variables: dict, raw_template: str) -> dict:
resolved = render_template(raw_template, variables)
return {
"template_version": template_version,
"template_hash": sha256(raw_template.encode()).hexdigest(),
"resolved_hash": sha256(resolved.encode()).hexdigest(),
"resolved_prompt": resolved,
}
With versioned prompts in your traces, you can build automated drift detection:
Archived traces are the single source of truth for AI spend. Every dollar should be traceable to a user, a feature, a model, and a time window.
Calculate cost at ingestion time using the provider's published pricing. Store both the raw token counts and the computed cost. This way, if pricing changes retroactively, you can recalculate without re-ingesting traces.
PRICING = {
"gpt-4o": {"input": 0.0000025, "output": 0.00001},
"gpt-4o-mini": {"input": 0.00000015, "output": 0.0000006},
"claude-3-5-sonnet": {"input": 0.000003, "output": 0.000015},
}
def compute_cost(model: str, usage: dict) -> float:
rates = PRICING.get(model)
if not rates:
return None # unknown model, flag for manual review
input_cost = (usage["input_tokens"] or 0) * rates["input"]
output_cost = (usage["output_tokens"] or 0) * rates["output"]
return round(input_cost + output_cost, 6)
Archival introduces compliance risk. Implement:
The ultimate payoff of prompt archival is evaluation. With complete traces, you can build evaluation loops that would otherwise be impossible.
Take a set of historical traces, extract the inputs, and replay them through a new prompt or a new model. Compare outputs against the original outputs and against human labels. This is the backbone of production prompt engineering.
Integrate trace replay into your CI pipeline. Before deploying a new prompt version, run it against a gold standard set of historical inputs and fail the pipeline if quality degrades below a threshold.
Build a tool that accepts a trace ID and reconstructs the full execution: the prompt, the retrieved chunks, the model response, and the tool calls. This turns a "why did the agent do that" question into a five-minute investigation instead of a five-day one.
Use ULIDs for trace IDs. They are lexicographically sortable, embed a timestamp, and avoid the collision worries of UUIDv4. Use them for both traces and sessions.
Define retention tiers:
Review your legal requirements before setting these. Some industries mandate longer retention.
Monitor the ingestion pipeline the same way you monitor your production services:
For teams building their first archival system, here is a minimal but production-ready stack:
pgvector
for metadata and semantic search, and a separate vector index if you need large-scale similarity search.The Tamiz's Insights series on production AI engineering covers several of these patterns in depth, particularly around evaluation pipelines and prompt versioning.
Q: Do I really need to archive tool calls and retrieval results, or is the prompt and response enough?
No, those pieces are not optional if you want true reproducibility. Two traces with identical user inputs but different retrieved documents represent different executions. If you cannot reconstruct what the model saw, you cannot reproduce or debug the output.
Q: How much storage will this actually consume?
A typical trace for a GPT-4o agent run is 2–10 KB depending on retrieval richness. At 100,000 traces per day, you are looking at roughly 500 MB to 5 GB per day. Compressed and tiered, the long-term cost is manageable, but set your retention policy before you start archiving or storage bills will surprise you.
Q: Can I use existing observability tools instead of building this?
Tools like LangSmith, Phoenix, and Langfuse cover many of these needs out of the box. They are excellent choices for teams that want to move fast. However, they impose vendor lock-in and may not support your cost, retention, or compliance requirements. Evaluate them honestly, but do not assume they replace the architectural decisions discussed here — they implement them.
Q: How do I handle streaming responses in the archive?
Buffer the stream in memory until completion, then write the assembled response. If the stream errors out, archive the partial response with an error flag so you can still investigate. Never archive streamed chunks one-by-one — that creates thousands of incomplete trace objects and defeats the purpose.