Postmortem: Silent Data Loss in an In-Process Vector Store, and the Case for Durable Execution

A developer discovered that calling optimize() on a ZVec collection recovered from an unclean process termination silently discards documents restored by WAL replay, reporting index_completeness: 1.0 while doc_count regresses to 0. The bug is specific to WAL-replayed data and highlights the need for durable execution guarantees in in-process vector stores used with workflow orchestrators.

optimize on a ZVec Collection that had recovered from an unclean process termination silently discarded the documents restored by WAL replay. The call raised no exception, and stats on the same handle reported index completeness: 1.0 — a false signal of success. The loss only became visible when the collection was reopened from a separate process. doc count regressed from 150 to 0 after optimize on a crash-recovered collection with no prior flush calls. alibaba/zvec optimize treated the writing segment as having no WAL records and skipped persisting the recovered data.ZVec is an in-process vector database: no server process, no connection string, storage backed by write-ahead logging WAL on the local filesystem. That simplicity is attractive for embedding retrieval directly inside a workflow orchestrator, where the vector store is expected to behave like any other durable resource an orchestrator's Activities touch. For that use case, "crash-durable" is not a nice-to-have — it is a precondition for adoption. If a storage engine can silently lose state on process failure, it cannot be trusted as the backing store for anything a workflow engine expects to survive a crash and resume correctly. Before this bug surfaces, the base case is worth establishing, since it is where trust in the system starts. A hard-killed process os. exit 1 , equivalent to kill -9 with unflushed inserts fully recovers on zvec.open : crashed = zvec.open "./zvec crash test" print crashed.stats doc count=150 index completeness={'embedding': 0.0} doc count is correct immediately after a crash, purely from automatic WAL replay — no manual recovery call required. index completeness: 0.0 at this point is not an anomaly; it is the default state of any collection, crashed or not, until optimize has been called on it, and query still returns correct results while it holds. Calling optimize on a crash-recovered collection does not raise. Worse, stats on the same handle immediately afterward reports the pre-existing doc count unchanged, alongside index completeness: 1.0 — every observable signal on that handle says the optimize succeeded cleanly: optimize before: doc count=150 index completeness={'embedding': 0.0} optimize same-handle after optimize : doc count=150 index completeness={'embedding': 1.0} Neither the return value nor an exception nor the immediately-following stats call exposes the defect. This is the core hazard: a call that completes cleanly and reports a plausible, successful-looking state can still have discarded data. The regression is invisible on the handle that called optimize because that handle still holds the in-memory view of the data. It becomes visible only when a separate process reopens the same collection from disk: after-optimize doc count=0 index completeness={'embedding': 0.0} doc count regresses to whatever was last durably segmented to disk before the crash — in this case, 0, since no flush was ever called. The gap between the in-memory state a handle believes is true and the state actually persisted to disk is exactly what optimize 's bug exploited. A plausible but wrong hypothesis at this point is " optimize loses unflushed data." A control experiment rules that out: a collection with unflushed data but no crash history — created, populated, and shut down normally — retains 100% of its documents after optimize , unflushed data included. The loss is therefore not a property of "unflushed data" in general. It is specific to documents that were recovered via WAL replay from a genuine unclean termination. optimize 's index-rebuild path excluded that WAL-replayed data from the rebuilt segment rather than merging it in. A second experiment narrowed the loss boundary precisely. With a periodic flush cadence — every 100 documents, 551 total inserted — optimize on the crash-recovered collection left exactly 501 documents; IDs 501–550, everything inserted after the last flush at ID 500, were missing. The loss boundary is exactly the last durably-flushed segment, not "everything since the crash" and not "everything unflushed." A single experiment result is not treated as proof of a causal claim here — the crash-history-vs-no-crash-history comparison is what isolates the trigger. Root cause, in one sentence: segment recovery replayed WAL records into a local WAL handle, then closed it without restoring the member WAL handle on the segment — so optimize 's subsequent flush treated the writing segment as having no WAL records and skipped persisting them. The patch to src/db/index/segment/segment.cc : @@ -4212,8 +4212,6 @@ Status SegmentImpl::recover { wal file path.c str ; return Status::OK ; } - AILEGO DEFER & { recover wal file- close ; } ; - std::array<uint64 t, static cast<size t Operator::DELETE + 1 recovered doc count{}; uint64 t total recovered doc count{0}; @@ -4297,7 +4295,17 @@ Status SegmentImpl::recover { size t recovered doc count 3 // DELETE ; - return Status::OK ; + if recover wal file- close = 0 { + return Status::InternalError "Failed to close recovered wal file: ", + wal file path ; + } + recover wal file.reset ; + + // Keep the recovered WAL attached to the segment. Operations such as + // optimize flush the writing segment before sealing it; without an open + // member WAL, flush treats the recovered memory components as empty and + // returns without persisting them. + return open wal file ; } Instead of discarding the recovery WAL handle once replay finishes, recover now reopens the segment's member WAL handle open wal file before returning. optimize 's subsequent flush step then sees a live WAL attached to the segment and persists the recovered records instead of treating the segment as empty. This is a handle-lifecycle bug, not a logic bug in the index-rebuild algorithm itself — which is also why the patch is narrow: three lines removed, roughly a dozen added, confined to the recovery path. optimize " Isn't a Strategy PR 600 closes the specific intersection of crash recovery and optimize . It does not change the underlying design: an in-process storage engine that owns its own, implicit durability contract, with no external signal exposed at the time of a call like whether a collection currently carries unreplayed crash-recovery state. The next unknown intersection between recovery and some other operation is a design property, not a one-off accident, and a single patch cannot retire that property. ZVec's WAL-based recovery model has a boundary : the last segment durably flushed to disk. Everything after that boundary depends on the WAL being correctly replayed and correctly reattached on every code path that might act on it — which is exactly the class of invariant that PR 600 had to repair by hand. Durable execution frameworks built on event sourcing, such as Temporal, do not have an equivalent boundary. A Workflow's state is not a periodically-flushed snapshot; it is the deterministic replay of its complete event history — every signal received, every activity result recorded — reconstructed from that history on every recovery. There is no concept of "the last flush point" for the replay to fall behind, because the event history is the durable state, not a cache of it. A companion Temporal-based agent workflow used elsewhere demonstrates the pattern concretely. A long-running Workflow exposes signal handlers that mutate its in-memory state: php @workflow.signal def add task self, task: str - None: self. task queue.append task @workflow.signal def update task priority self, priority: int - None: self. priority = max 1, min 10, priority @workflow.signal def inject human feedback self, feedback: str - None: self. human feedback = feedback Every signal is recorded into Temporal's event history as it is received. This is not a proposal — it has an automated resilience test that actually exercises the failure mode: start the Workflow, send signals inject feedback, change priority , forcibly kill the worker process , restart it, and query the Workflow's state to confirm the signal-driven changes survived the crash intact. The recovery path here is full deterministic replay of the event history against a fresh worker, not a WAL segment reattachment. This does not make the underlying storage problem disappear entirely. ZVec still exposes no full-collection enumeration API, so any migration-based recovery workaround depends on the caller already knowing every document ID. That is precisely why, in an architecture where ZVec sits inside a Temporal-orchestrated pipeline, the orchestrator's own durable state — not the storage engine's internal bookkeeping — is the right place to keep the ID ledger of record. As of this verification pass, pip index versions zvec still reports 0.5.1 as latest — the same version the bug was originally found on. The merge commit containing the fix e89be98b is not included in any published PyPI release. A v0.6.0 git tag exists in the repository, but comparing it against the fix commit shows it is behind the fix — it was cut before the patch landed. Re-running the original repro script against a released package is therefore not currently possible, and that limitation is stated here rather than worked around. The change described above was confirmed directly against the PR's diff, not inferred from the PR description: the root-cause explanation "the recovered WAL was not reattached to the segment's member handle" matches the actual patch recover now calls open wal file before returning, instead of leaving the segment without a live WAL handle . The PR adds a regression test that mirrors the originally reported reproduction almost exactly: TEST F CrashRecoveryTest, OptimizeAfterCrashRecoveryPersistsReplayedData { // ... create collection, crash during insertion ... auto result = Collection::Open dir path , options ; auto collection = result.value ; recovered doc count = collection- Stats .value .doc count; ASSERT GT recovered doc count, 0 << "No documents were recovered from the WAL"; auto status = collection- Optimize ; ASSERT TRUE status.ok << status.message ; ASSERT EQ collection- Stats .value .doc count, recovered doc count ; // Reopen from a fresh handle and assert nothing was lost. auto reopened = Collection::Open dir path , options ; ASSERT EQ reopened.value - Stats .value .doc count, recovered doc count << "Optimize discarded documents restored from the WAL"; } Crash, reopen, optimize , reopen again from a fresh handle, assert doc count is unchanged — the same crash-then-verify-from-a-fresh-process structure used to originally isolate the bug. The PR description states that "the full CI matrix passed on the same commit: Linux, macOS, Windows, Android, iOS, lint, and clang-tidy." Pulling the actual check-runs from the GitHub Actions API tells a more precise story. On the PR's own head commit aaa4aab4 , the Build & Test jobs for every platform, along with lint and clang-tidy , show as skipped — consistent with a permission gate that withholds full CI from external-contributor pull requests until a maintainer authorizes it. "The same commit" did not, in fact, run the full matrix. What actually ran, and passed, was the merge commit onto main e89be98b . Its CI log shows: 66/152 Test 67: optimize recovery test ............................ Passed 354.84 sec 109/152 Test 68: write recovery test ............................... Passed 1687.70 sec write recovery test is the binary that contains the new OptimizeAfterCrashRecoveryPersistsReplayedData case; it passed on the merge commit, along with the C++ test suite, the Python test suite, lint , clang-tidy , and a linux-x64 build. That is solid first-party evidence that the fix behaves as intended. It is also worth being precise about what did not run: the Windows, macOS multiple variants , iOS, and Android Build & Test jobs remained skipped even after the merge to main . "Full CI matrix" is therefore accurate for the core Linux test suite and static analysis, but not for the full platform matrix implied by the PR description. This verification did not include building ZVec from source and re-running the original Python repro script against the patched code — there is no PyPI release yet to install, and a from-source build was out of scope for this pass. The natural follow-up, once a release beyond 0.5.1 ships, is to re-run repro zvec optimize dataloss.py against the released package and confirm doc count no longer regresses to 0.