Incremental RAG indexing without losing the old index

Project Pavilion, a .NET 9 project that indexes its records into Qdrant, has adopted an incremental RAG indexing design that separates preparation from publication so retrieval never sees a half-updated collection. Each source record is stored as a single Qdrant point keyed by the first 16 bytes of a SHA-256 hash of its type and id, so re-indexing overwrites the point atomically and retrieval sees either the old or new vector, never a mix. The project also defines explicit degraded-mode behavior: failed embeddings skip the write and keep the old vector, and queries during embedding outages return no context rather than an error, with the chat's search tool falling back to SQL.

Re-indexing is easy when the collection is small and nobody is using it. The harder problem is changing a live index while documents, parsers, embedding services, and vector storage can all fail at different points. The examples come from Project Pavilion, which indexes its project records into Qdrant on .NET 9. An indexing job is not just a loop that reads files and writes vectors. It changes the version of knowledge that retrieval is allowed to use. That makes the publish boundary more important than the loop itself. If new chunks replace old chunks one at a time, users can query a half-updated collection. A document may have new text with old metadata, or only some of its chunks may exist. The system is technically online, but the answers are difficult to trust. A safer model separates preparation from publication. Build the next document version away from the active one, validate it, then make one small state change that tells retrieval which version is current. File names are weak identifiers. They change, collide, and say nothing about whether the content is actually different. Keep a small indexing record for each source so the job can make a deliberate decision before it calls an expensive parser or embedding model. In Project Pavilion each source is a record, such as a requirement or a task, and its key is the record’s type and id. The Qdrant point id comes from the first 16 bytes of a SHA-256 of those two, and each record is a single point, so indexing it again overwrites that point. Retrieval sees the old vector or the new one, never a mix and never two copies. Minimal indexing record source key content hash pipeline version active version staged version status last error updated at The exact storage model can vary. The useful property is the same: retrieval sees either the old complete document or the new complete document, never a mixture. Degraded mode should be decided before an outage. Otherwise every dependency failure turns into an improvised choice between returning an error and returning an answer based on incomplete data. The safest fallback is often the last known-good index. It may be stale, but it has a clear boundary and can be labelled as stale. A partially written new index has neither property. Project Pavilion makes two of these decisions in code. If the embedding fails while a record is being indexed, the write is skipped and the old vector stays. If a question arrives while embeddings are down, retrieval returns no context instead of an error, the answer is built without it, and the chat’s search tool falls back to a SQL query when it knows which kind of record it wants. Examples of explicit failure behaviour | Failure | Ingestion behaviour | Retrieval behaviour | |---|---|---| | Embedding service unavailable | Keep the source pending and retry with limits. | Serve the active index. Do not publish empty vectors. | | Parser rejects one document | Quarantine that source and retain its old active version. | Continue serving other documents and expose the failed source to operators. | | Vector store is slow | Pause new batches when a bounded timeout is reached. | Return a clear temporary failure, or use a tested secondary search path if one exists. | | Metadata store unavailable | Do not start jobs that cannot record their state. | Keep reading the active index only if its ownership and version remain unambiguous. | Retries are useful for temporary network and service failures. They are dangerous when the write may already have succeeded and the operation is not idempotent. Give each job, document version, and chunk a deterministic key so the same work can be attempted again without producing duplicates. A malformed file is not a temporary failure. After a small number of attempts, move it out of the normal queue and record enough context to repair it. One bad source should not block every healthy source behind it. An ingestion dashboard does not need dozens of charts. It needs to answer a few operational questions quickly: what changed, what is active, what failed, and whether retrieval is using stale data. Originally published at anasamin.dev https://anasamin.dev/notes/incremental-rag-indexing/ . I'm Anas Amin, a software engineer in .NET and the founder of Arkedia https://arkedia.dev .