Imagine a production AI assistant that retrieves the correct policy, caches an older version, compresses away an important exception, and then delivers the wrong answer confidently in less than a second.
Every part of that failure is a context problem.
Modern LLM applications build their context from many sources: system prompts, conversation history, retrieved documents, tool outputs, schemas, user memories, cached prefixes and agent state.
At inference time, all of this becomes working state. It has to be stored, transferred, reused, updated, and eventually discarded. But managing context is not only an infrastructure problem. The system must also decide whether the information it has assembled is current, authoritative, consistent, relevant, and safe to use.
This creates two connected layers:
KV-cache infrastructure represents the runtime plane. Contradiction-aware RAG represents the evidence plane.
Together, they point to a broader shift: context is becoming infrastructure.
A production system may reuse context in several different ways.
These mechanisms may sound similar, but they solve different problems.
A KV cache is numerical runtime state. It is not semantic memory.
A retrieval cache stores possible evidence for an answer. A semantic cache reuses earlier answers, but it does not eliminate the need for embedding, matching, policy checks, freshness checks, or verification.
Treating these mechanisms as interchangeable makes systems harder to reason about and easier to break.
Transformer inference has two main phases.
Prefill processes the input prompt and creates the initial key-value state. This phase is usually compute-intensive.
Decode generates the response one token at a time while repeatedly reading from the state. This phase is often limited by memory capacity and bandwidth.
A simplified estimate of KV-cache memory is :
Here, memory use grows with batch size, sequence length, layer count, the number of KV heads, head dimension, and numeric precision.
In practical terms, longer uncached prompts generally increase time to the first token, all else being equal. Long conversations also keep more KV state alive, leaving less memory available for other requests and reducing concurrency.
Serving systems address this with techniques such as paging, prefix reuse, quantisation, pruning, off, and the separation of prefill and decode across different resources.
PagedAttention introduced block-based KV-memory management, while modern serving platforms have expanded the idea into broader scheduling and execution systems.
Recent research adds several important qualifications.
First, compression ratio is not enough.
A 2026 benchmark found that KV-cache optimisations vary by model, task, and context length. A memory-saving method may reduce answer quality or introduce sufficient processing overhead to increase latency. Quality, throughput, memory consumption, and response time therefore need to be evaluated together.
Second, compression is becoming more flexible.
Emerging approaches such as MosaicKV adapt compression across both tokens and feature channels rather than applying a single fixed policy everywhere.
Third, a system does not know in advance how long a model’s response will be.
Reserving too much memory wastes capacity. Reserving too little can cause preemption, recomputation, and poor tail latency.
KV management is therefore becoming a scheduling and service-level-objective problem, not simply a compression problem.
Infrastructure claims all need to be tied to specific versions. Current TensorRT-LLM documentation, for example, covers KV connectors, external storage, cache salting, and disaggregated serving. Hugging Face’s Text Generation Inference, by contrast, is now in maintenance mode.
RAG gives models access to external knowledge, but retrieving information is not the same as retrieving trustworthy evidence.
A retrieved passage may be:
Retrieving more documents does not necessarily solve this.
Increasing top-k can add prefill time, KV pressure, duplicated information, and more opportunities for the source to disagree.
The question is no longer simply, “Did we retrieve something relevant?”
It is also, “Can these pieces of evidence be combined into a reliable answer?”
**Contradiction-aware RAG **should therefore be understood as an emerging family of architectures, not a settled standard.
A practical system might:
The important point is that contradictions should not simply be smoothed away during retrieval, ranking, or summarisation.
A robust system needs to distinguish between different kinds of disagreement.
GroupQA, published in Findings of ACL 2026, showed that models can be influenced by the order, repetition, and paraphrasing of evidence. A claim may appear more persuasive simply because it is repeated, even when each repetition comes from the same underlying score.
ConfRAG, published at ALC 2026, similarly shows that conflicting evidence is not merely an occasional retrieval failure. It is a distinct reasoning challenge that systems need to address deliberately.
A policy-assistant example: Consider an internal assistant asked:
Can this customer receive a refund after 45 days?
The retrieval system returns four documents:
A naive system may count three documents in favour of the 60-day window and only one in favour of the 30-day limit.
But those three documents are not independent evidence. They all descend from the same outdated policy.
A contradiction-aware system would identify the shared lineage, recognise that the old policy has been superseded and treat the current policy as authoritative. It would also preserve any exception or approval language instead of removing it during compression.
The resulting answer might say:
The current policy sets a 30-day return period, so a standard refund after 45 days would not qualify. Some older internal documents refer to a superseded 60-day policy. Any exception would require separate approval.
The improvement does not come from retrieving more text. It comes from understanding provenance, time, authority, and conflict.
Context compression is often treated as a simple optimisation: remove enough text to fit within a token or memory budget.
But careless compression can change the meaning of the evidence. It may remove exceptions, flatten disagreements, combine claims from different scopes, or make uncertain conclusions sound definitive.
A compressed evidence unit should preserve at least:
The goal is not merely to produce a shorter context. It is to produce a shorter context without manufacturing agreement.
Deletion has similar implications.
When a source is corrected, withdrawn, deleted, or no longer authorised, removing the original document may be enough. Teams may also need to invalidate:
A cache hit is not a success when the cache context is stale, incorrect, or no longer permitted.
The cost of an LLM request extends beyond the final generation call.
A useful high-level decomposition is: This is not intended to be a complete hardware or queueing model. Its purpose is to make visible the costs that are often hidden behind a single model invocation.
A system may reduce generation cost while increasing verification cost. It may improve cache-hit rates while creating more stale-answer incidents. It may retrieve fewer documents but spend more on conflict analysis.
For many production systems, the more meaningful business metric is therefore not cost per response, but cost per sufficiently grounded answer. That distinction matters.
A cheap answer that must be manually checked, corrected, or regenerated may not be cheap at all.
Runtime observability and semantic observability should appear in the same trace.
On the evidence side, teams may need to monitor:
On the runtime side, they may need to monitor:
These metrics should also be segmented by workload.
A compression policy that performs well for summarisation may fail in multi-document question answering. A retrieval strategy that works for technical support may be inappropriate for legal, medical, or policy-related queries.
OpenTelemetry’s generative-AI semantic conventions may help teams structure these traces, although the conventions remain under active development rather than representing a settled standard.
Retrieved content should be treated as untrusted input. It may contain outdated claims, poisoned information, prompt-injection instructions, or data the user is not authorised to access.
Production systems therefore need controls such as :
The security boundary is also becoming more agentic.
In multi-agent systems, shared memory, tool outputs, and inter-agent messages can carry mistakes or malicious instructions beyond a single prompt. A compromised piece of context may be reused across agents, sessions, or workflows.
A trustworthy system must also be able to say that the available evidence is insufficient, contradictory, or unresolved.
Abstention is not a failure when the context does not justify a confident answer. In many cases, it is the correct system behaviour.
The core mistake is to treat context as though it were only text. Context is simultaneously:
On the runtime plane, the key question is:
Can the system retain and reuse enough context while staying within its memory, latency, and cost constraints?
On the evidence plan, the question is:
Is the assembled context relevant, current, attributable, and coherent enough to justify the answer?
Both questions matter.
KV-cache infrastructure without evidence discipline simply makes unreliable answers faster. Evidence verification without efficient serving may make reliable answers too slow or expensive to deploy.
The next generation of LLM platforms will therefore be differentiated not only by model quality, but by the quality, efficiency, and governance of their context stack.
Context is no longer simply what we place inside a prompt. It is the operational substrate on which the system thinks.
Context is Becoming Infrastructure: From KV Cache to Contradiction-Aware RAG was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.