Upstream will add a field, rename a field, change a type from string to object, and start sending null where it never did. None of that is avoidable. What is avoidable is finding out about it from a crash in a consumer at three in the morning.
A document pipeline has three separate schemas and they evolve independently. Conflating them is the root of most of the pain.
The mapping between source and internal is the shock absorber. Put the tolerance there — one adapter per source, each responsible for producing a valid internal document or failing loudly — and the rest of the pipeline never sees an upstream change at all.
The internal schema deserves one more constraint that is easy to state and unpopular to enforce: it should contain nothing that is specific to one source. The moment a field exists because a particular vendor sends it, every other adapter has to decide what to put there, and the “internal” schema has become a union of external ones. Put source-specific data in a namespaced extras map instead, where it is available to anyone who wants it and mandatory for nobody.
Streaming systems formalised this years ago and the vocabulary transfers directly. Avro’s schema resolution rules and the compatibility modes used by schema registries give you four named answers to “can I deploy this?”:
| Mode | Description |
|---|---|
| BACKWARD | New readers can read old data. Permits deleting a field, and adding an optional field with a default. This is the mode you want when consumers upgrade first — the usual case for a pipeline you own. |
| FORWARD | Old readers can read new data. Permits adding a field, and deleting an optional one. This is the mode you want when producers upgrade first — the usual case for an upstream you do not control. |
| FULL | Both. In practice: add and remove optional fields with defaults, and nothing else. Deployment order stops mattering, which is worth a great deal in a system with many consumers. |
| TRANSITIVE variants | The same guarantee against every previous version rather than only the last. Without it, a sequence of individually compatible changes can leave version 5 unable to read version 1 — which is exactly what a reprocessing job over historical data needs to do. |
The transitive distinction is the one that catches people running AI pipelines specifically, because reprocessing the archive is a normal operation here rather than an exotic one. If you will ever re-embed three-year-old documents from their stored artefacts, you need FULL_TRANSITIVE
behaviour, not FULL
.
The rule underneath all four is the same and it is worth internalising rather than memorising the table: a field added without a default is a breaking change, because a reader encountering old data has nothing to put there. A default turns an incompatible change into a compatible one, which is why every one of the safe operations above has “with a default” attached.
Codify the tolerance at the boundary. Unknown fields are captured, not dropped and not fatal; missing optional fields take defaults; a missing required field fails that document and only that document.
from pydantic import BaseModel, ConfigDict, Field, ValidationError
class SourceDoc(BaseModel):
model_config = ConfigDict(extra="allow")
id: str
body: str
title: str | None = None
published_at: str | None = None
tags: list[str] = Field(default_factory=list)
def adapt(raw: dict) -> InternalDoc | None:
try:
s = SourceDoc.model_validate(raw)
except ValidationError as e:
quarantine(raw, reason=e.json()) # one document, not the batch
return None
extras = set(s.model_extra or ()) - KNOWN_EXTRAS
if extras:
metrics.increment("source.unknown_field", tags=sorted(extras))
return InternalDoc(id=s.id, text=s.body, title=s.title or "",
published=parse_date(s.published_at), tags=s.tags)
The unknown_field
counter is the cheapest early warning in this whole cluster. Upstream adds a field weeks before anyone tells you, the counter moves, and you find out from a metric instead of from a consumer that started failing when the field became mandatory.
Note the granularity of the failure: one document is quarantined and the batch continues. That choice is worth making deliberately, because the alternative — fail the batch on any invalid record — sounds rigorous and behaves badly. A single malformed document from a source of two hundred thousand stops the whole pipeline, and the pressure to get data flowing again produces a bypass flag that then stays on forever. Per-document quarantine with a visible count gives you the same information without the outage.
There is one exception where failing the batch is correct: when the validation failure rate crosses a threshold. One bad document is a bad document; twenty per cent of them failing the same way is an upstream change, and continuing means publishing a corpus that is missing a fifth of itself. That is a corpus-level check rather than a per-record one, which is exactly why both layers exist.
The common request is to add something to every chunk — a language tag, a security label, a section path — over an index that is serving traffic. Doing it in one migration means either a long write lock or a rebuild. The four-step version has neither:
1. ADD the column/field as nullable, no default backfill.
Postgres 11+ makes ADD COLUMN with a constant default O(1) — it stores
the default in the catalogue rather than rewriting every row. Adding a
nullable column has always been cheap. A volatile default is not.
2. WRITE it from the pipeline for every new and updated document.
Now new data has it and old data does not. Readers must tolerate both,
which is the FORWARD-compatible position from the table above.
3. BACKFILL in batches, keyed by content hash so it is resumable:
UPDATE chunks SET lang = $1 WHERE id = ANY($2) -- 5k ids at a time
Track progress in a table, not in the shell history.
4. TIGHTEN the constraint only once step 3 reports zero remaining rows.
This is the step people skip, and skipping it means the field is
permanently optional and every consumer keeps its null branch forever.
Do not filter on the new field between steps 1 and 4. A query with WHERE lang = 'en'
during the backfill silently returns the subset that happens to be done, which is a correctness bug that looks like a relevance problem.
Some changes cannot be made compatible: a type change from scalar to object, a unit change (dollars to cents), a semantic change to a field that keeps its name. For these the only safe move is a new field alongside the old one, both populated during a transition window, and the old one removed when nothing reads it.
Semantic changes deserve special fear because they are invisible to every automated compatibility check. If score
was 0–1 and is now 0–100, every schema validator passes and every threshold in your system is wrong. Version the name when the meaning changes — score_pct
next to score
— and let the compiler and the null checks find the call sites for you. It is uglier and it is the only version of this that has ever worked.
Plan the transition window explicitly rather than letting it run indefinitely. Write down when dual-writing starts, what has to be true before the old field can be dropped — usually a read counter at zero for a full cycle, including whatever monthly job nobody remembers — and who is responsible for the removal. Without that, the transition window never closes: both fields are populated forever, every new consumer picks whichever one it finds first, and the ambiguity you were trying to remove is now permanent and doubled.