{"slug": "postmortem-silent-data-loss-in-an-in-process-vector-store-and-the-case-for", "title": "Postmortem: Silent Data Loss in an In-Process Vector Store, and the Case for Durable Execution", "summary": "A developer discovered that calling optimize() on a ZVec collection recovered from an unclean process termination silently discards documents restored by WAL replay, reporting index_completeness: 1.0 while doc_count regresses to 0. The bug is specific to WAL-replayed data and highlights the need for durable execution guarantees in in-process vector stores used with workflow orchestrators.", "body_md": "`optimize()`\n\non a ZVec `Collection`\n\nthat had recovered from an unclean process termination silently discarded the documents restored by WAL replay. The call raised no exception, and `stats()`\n\non the same handle reported `index_completeness: 1.0`\n\n— a false signal of success. The loss only became visible when the collection was reopened from a separate process.`doc_count`\n\nregressed from 150 to 0 after `optimize()`\n\non a crash-recovered collection with no prior `flush()`\n\ncalls.`alibaba/zvec`\n\n`optimize()`\n\ntreated the writing segment as having no WAL records and skipped persisting the recovered data.ZVec is an in-process vector database: no server process, no connection string, storage backed by write-ahead logging (WAL) on the local filesystem. That simplicity is attractive for embedding retrieval directly inside a workflow orchestrator, where the vector store is expected to behave like any other durable resource an orchestrator's Activities touch.\n\nFor that use case, \"crash-durable\" is not a nice-to-have — it is a precondition for adoption. If a storage engine can silently lose state on process failure, it cannot be trusted as the backing store for anything a workflow engine expects to survive a crash and resume correctly.\n\nBefore this bug surfaces, the base case is worth establishing, since it is where trust in the system starts. A hard-killed process (`os._exit(1)`\n\n, equivalent to `kill -9`\n\n) with unflushed inserts fully recovers on `zvec.open()`\n\n:\n\n```\ncrashed = zvec.open(\"./zvec_crash_test\")\nprint(crashed.stats)\n# doc_count=150 index_completeness={'embedding': 0.0}\n```\n\n`doc_count`\n\nis correct immediately after a crash, purely from automatic WAL replay — no manual recovery call required. `index_completeness: 0.0`\n\nat this point is not an anomaly; it is the default state of any collection, crashed or not, until `optimize()`\n\nhas been called on it, and `query()`\n\nstill returns correct results while it holds.\n\nCalling `optimize()`\n\non a crash-recovered collection does not raise. Worse, `stats()`\n\non the *same* handle immediately afterward reports the pre-existing `doc_count`\n\nunchanged, alongside `index_completeness: 1.0`\n\n— every observable signal on that handle says the optimize succeeded cleanly:\n\n```\n[optimize] before: doc_count=150 index_completeness={'embedding': 0.0}\n[optimize] same-handle after optimize(): doc_count=150 index_completeness={'embedding': 1.0}\n```\n\nNeither the return value nor an exception nor the immediately-following `stats()`\n\ncall exposes the defect. This is the core hazard: a call that completes cleanly and reports a plausible, successful-looking state can still have discarded data.\n\nThe regression is invisible on the handle that called `optimize()`\n\nbecause that handle still holds the in-memory view of the data. It becomes visible only when a **separate process** reopens the same collection from disk:\n\n```\n[after-optimize] doc_count=0 index_completeness={'embedding': 0.0}\n```\n\n`doc_count`\n\nregresses to whatever was last durably segmented to disk before the crash — in this case, 0, since no `flush()`\n\nwas ever called. The gap between the in-memory state a handle believes is true and the state actually persisted to disk is exactly what `optimize()`\n\n's bug exploited.\n\nA plausible but wrong hypothesis at this point is \"`optimize()`\n\nloses unflushed data.\" A control experiment rules that out: a collection with unflushed data but **no crash history** — created, populated, and shut down normally — retains 100% of its documents after `optimize()`\n\n, unflushed data included.\n\nThe loss is therefore not a property of \"unflushed data\" in general. It is specific to documents that were recovered via WAL replay from a genuine unclean termination. `optimize()`\n\n's index-rebuild path excluded that WAL-replayed data from the rebuilt segment rather than merging it in.\n\nA second experiment narrowed the loss boundary precisely. With a periodic `flush()`\n\ncadence — every 100 documents, 551 total inserted — `optimize()`\n\non the crash-recovered collection left exactly 501 documents; IDs 501–550, everything inserted after the last `flush()`\n\nat ID 500, were missing. The loss boundary is exactly the last durably-flushed segment, not \"everything since the crash\" and not \"everything unflushed.\" A single experiment result is not treated as proof of a causal claim here — the crash-history-vs-no-crash-history comparison is what isolates the trigger.\n\n**Root cause, in one sentence:** segment recovery replayed WAL records into a *local* WAL handle, then closed it without restoring the *member* WAL handle on the segment — so `optimize()`\n\n's subsequent flush treated the writing segment as having no WAL records and skipped persisting them.\n\nThe patch to `src/db/index/segment/segment.cc`\n\n:\n\n```\n@@ -4212,8 +4212,6 @@ Status SegmentImpl::recover() {\n               wal_file_path.c_str());\n     return Status::OK();\n   }\n-  AILEGO_DEFER([&]() { recover_wal_file->close(); });\n-\n   std::array<uint64_t, static_cast<size_t>(Operator::DELETE) + 1>\n       recovered_doc_count{};\n   uint64_t total_recovered_doc_count{0};\n@@ -4297,7 +4295,17 @@ Status SegmentImpl::recover() {\n       (size_t)recovered_doc_count[3]   // DELETE\n   );\n\n-  return Status::OK();\n+  if (recover_wal_file->close() != 0) {\n+    return Status::InternalError(\"Failed to close recovered wal file: \",\n+                                 wal_file_path);\n+  }\n+  recover_wal_file.reset();\n+\n+  // Keep the recovered WAL attached to the segment. Operations such as\n+  // optimize() flush the writing segment before sealing it; without an open\n+  // member WAL, flush() treats the recovered memory components as empty and\n+  // returns without persisting them.\n+  return open_wal_file();\n }\n```\n\nInstead of discarding the recovery WAL handle once replay finishes, `recover()`\n\nnow reopens the segment's member WAL handle (`open_wal_file()`\n\n) before returning. `optimize()`\n\n's subsequent flush step then sees a live WAL attached to the segment and persists the recovered records instead of treating the segment as empty.\n\nThis is a handle-lifecycle bug, not a logic bug in the index-rebuild algorithm itself — which is also why the patch is narrow: three lines removed, roughly a dozen added, confined to the recovery path.\n\n`optimize()`\n\n\" Isn't a Strategy\nPR #600 closes the specific intersection of crash recovery and `optimize()`\n\n. It does not change the underlying design: an in-process storage engine that owns its own, implicit durability contract, with no external signal exposed at the time of a call like whether a collection currently carries unreplayed crash-recovery state. The next unknown intersection between recovery and some other operation is a design property, not a one-off accident, and a single patch cannot retire that property.\n\nZVec's WAL-based recovery model has a *boundary*: the last segment durably flushed to disk. Everything after that boundary depends on the WAL being correctly replayed and correctly reattached on every code path that might act on it — which is exactly the class of invariant that PR #600 had to repair by hand.\n\nDurable execution frameworks built on event sourcing, such as Temporal, do not have an equivalent boundary. A Workflow's state is not a periodically-flushed snapshot; it is the deterministic replay of its complete event history — every signal received, every activity result recorded — reconstructed from that history on every recovery. There is no concept of \"the last flush point\" for the replay to fall behind, because the event history *is* the durable state, not a cache of it.\n\nA companion Temporal-based agent workflow used elsewhere demonstrates the pattern concretely. A long-running Workflow exposes signal handlers that mutate its in-memory state:\n\n``` php\n@workflow.signal\ndef add_task(self, task: str) -> None:\n    self._task_queue.append(task)\n\n@workflow.signal\ndef update_task_priority(self, priority: int) -> None:\n    self._priority = max(1, min(10, priority))\n\n@workflow.signal\ndef inject_human_feedback(self, feedback: str) -> None:\n    self._human_feedback = feedback\n```\n\nEvery signal is recorded into Temporal's event history as it is received. This is not a proposal — it has an automated resilience test that actually exercises the failure mode: start the Workflow, send signals (inject feedback, change priority), **forcibly kill the worker process**, restart it, and query the Workflow's state to confirm the signal-driven changes survived the crash intact. The recovery path here is full deterministic replay of the event history against a fresh worker, not a WAL segment reattachment.\n\nThis does not make the underlying storage problem disappear entirely. ZVec still exposes no full-collection enumeration API, so any migration-based recovery workaround depends on the caller already knowing every document ID. That is precisely why, in an architecture where ZVec sits inside a Temporal-orchestrated pipeline, the orchestrator's own durable state — not the storage engine's internal bookkeeping — is the right place to keep the ID ledger of record.\n\nAs of this verification pass, `pip index versions zvec`\n\nstill reports `0.5.1`\n\nas latest — the same version the bug was originally found on. The merge commit containing the fix (`e89be98b`\n\n) is not included in any published PyPI release. A `v0.6.0`\n\ngit tag exists in the repository, but comparing it against the fix commit shows it is *behind* the fix — it was cut before the patch landed. Re-running the original repro script against a released package is therefore not currently possible, and that limitation is stated here rather than worked around.\n\nThe change described above was confirmed directly against the PR's diff, not inferred from the PR description: the root-cause explanation (\"the recovered WAL was not reattached to the segment's member handle\") matches the actual patch (`recover()`\n\nnow calls `open_wal_file()`\n\nbefore returning, instead of leaving the segment without a live WAL handle).\n\nThe PR adds a regression test that mirrors the originally reported reproduction almost exactly:\n\n```\nTEST_F(CrashRecoveryTest, OptimizeAfterCrashRecoveryPersistsReplayedData) {\n  // ... create collection, crash during insertion ...\n\n  auto result = Collection::Open(dir_path_, options_);\n  auto collection = result.value();\n  recovered_doc_count = collection->Stats().value().doc_count;\n  ASSERT_GT(recovered_doc_count, 0)\n      << \"No documents were recovered from the WAL\";\n\n  auto status = collection->Optimize();\n  ASSERT_TRUE(status.ok()) << status.message();\n  ASSERT_EQ(collection->Stats().value().doc_count, recovered_doc_count);\n\n  // Reopen from a fresh handle and assert nothing was lost.\n  auto reopened = Collection::Open(dir_path_, options_);\n  ASSERT_EQ(reopened.value()->Stats().value().doc_count, recovered_doc_count)\n      << \"Optimize discarded documents restored from the WAL\";\n}\n```\n\nCrash, reopen, `optimize()`\n\n, reopen again from a fresh handle, assert `doc_count`\n\nis unchanged — the same crash-then-verify-from-a-fresh-process structure used to originally isolate the bug.\n\nThe PR description states that \"the full CI matrix passed on the same commit: Linux, macOS, Windows, Android, iOS, lint, and clang-tidy.\" Pulling the actual check-runs from the GitHub Actions API tells a more precise story.\n\nOn the PR's own head commit (`aaa4aab4`\n\n), the Build & Test jobs for every platform, along with `lint`\n\nand `clang-tidy`\n\n, show as **skipped** — consistent with a permission gate that withholds full CI from external-contributor pull requests until a maintainer authorizes it. \"The same commit\" did not, in fact, run the full matrix.\n\nWhat actually ran, and passed, was the **merge commit onto main** (\n\n`e89be98b`\n\n). Its CI log shows:\n\n```\n66/152 Test  #67: optimize_recovery_test ............................   Passed  354.84 sec\n109/152 Test  #68: write_recovery_test ...............................   Passed  1687.70 sec\n```\n\n`write_recovery_test`\n\nis the binary that contains the new `OptimizeAfterCrashRecoveryPersistsReplayedData`\n\ncase; it passed on the merge commit, along with the C++ test suite, the Python test suite, `lint`\n\n, `clang-tidy`\n\n, and a `linux-x64`\n\nbuild. That is solid first-party evidence that the fix behaves as intended.\n\nIt is also worth being precise about what did *not* run: the Windows, macOS (multiple variants), iOS, and Android Build & Test jobs remained **skipped** even after the merge to `main`\n\n. \"Full CI matrix\" is therefore accurate for the core Linux test suite and static analysis, but not for the full platform matrix implied by the PR description.\n\nThis verification did not include building ZVec from source and re-running the original Python repro script against the patched code — there is no PyPI release yet to install, and a from-source build was out of scope for this pass. The natural follow-up, once a release beyond `0.5.1`\n\nships, is to re-run `repro_zvec_optimize_dataloss.py`\n\nagainst the released package and confirm `doc_count`\n\nno longer regresses to 0.", "url": "https://wpnews.pro/news/postmortem-silent-data-loss-in-an-in-process-vector-store-and-the-case-for", "canonical_source": "https://dev.to/obataka/postmortem-silent-data-loss-in-an-in-process-vector-store-and-the-case-for-durable-execution-16i2", "published_at": "2026-07-21 01:47:00+00:00", "updated_at": "2026-07-21 02:29:16.960297+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure", "machine-learning"], "entities": ["ZVec", "alibaba/zvec"], "alternates": {"html": "https://wpnews.pro/news/postmortem-silent-data-loss-in-an-in-process-vector-store-and-the-case-for", "markdown": "https://wpnews.pro/news/postmortem-silent-data-loss-in-an-in-process-vector-store-and-the-case-for.md", "text": "https://wpnews.pro/news/postmortem-silent-data-loss-in-an-in-process-vector-store-and-the-case-for.txt", "jsonld": "https://wpnews.pro/news/postmortem-silent-data-loss-in-an-in-process-vector-store-and-the-case-for.jsonld"}}