optimize()
on a ZVec Collection
that had recovered from an unclean process termination silently discarded the documents restored by WAL replay. The call raised no exception, and stats()
on the same handle reported index_completeness: 1.0
— a false signal of success. The loss only became visible when the collection was reopened from a separate process.doc_count
regressed from 150 to 0 after optimize()
on a crash-recovered collection with no prior flush()
calls.alibaba/zvec
optimize()
treated the writing segment as having no WAL records and skipped persisting the recovered data.ZVec is an in-process vector database: no server process, no connection string, storage backed by write-ahead logging (WAL) on the local filesystem. That simplicity is attractive for embedding retrieval directly inside a workflow orchestrator, where the vector store is expected to behave like any other durable resource an orchestrator's Activities touch.
For that use case, "crash-durable" is not a nice-to-have — it is a precondition for adoption. If a storage engine can silently lose state on process failure, it cannot be trusted as the backing store for anything a workflow engine expects to survive a crash and resume correctly.
Before this bug surfaces, the base case is worth establishing, since it is where trust in the system starts. A hard-killed process (os._exit(1)
, equivalent to kill -9
) with unflushed inserts fully recovers on zvec.open()
:
crashed = zvec.open("./zvec_crash_test")
print(crashed.stats)
doc_count
is correct immediately after a crash, purely from automatic WAL replay — no manual recovery call required. index_completeness: 0.0
at this point is not an anomaly; it is the default state of any collection, crashed or not, until optimize()
has been called on it, and query()
still returns correct results while it holds.
Calling optimize()
on a crash-recovered collection does not raise. Worse, stats()
on the same handle immediately afterward reports the pre-existing doc_count
unchanged, alongside index_completeness: 1.0
— every observable signal on that handle says the optimize succeeded cleanly:
[optimize] before: doc_count=150 index_completeness={'embedding': 0.0}
[optimize] same-handle after optimize(): doc_count=150 index_completeness={'embedding': 1.0}
Neither the return value nor an exception nor the immediately-following stats()
call exposes the defect. This is the core hazard: a call that completes cleanly and reports a plausible, successful-looking state can still have discarded data.
The regression is invisible on the handle that called optimize()
because that handle still holds the in-memory view of the data. It becomes visible only when a separate process reopens the same collection from disk:
[after-optimize] doc_count=0 index_completeness={'embedding': 0.0}
doc_count
regresses to whatever was last durably segmented to disk before the crash — in this case, 0, since no flush()
was ever called. The gap between the in-memory state a handle believes is true and the state actually persisted to disk is exactly what optimize()
's bug exploited.
A plausible but wrong hypothesis at this point is "optimize()
loses unflushed data." A control experiment rules that out: a collection with unflushed data but no crash history — created, populated, and shut down normally — retains 100% of its documents after optimize()
, unflushed data included.
The loss is therefore not a property of "unflushed data" in general. It is specific to documents that were recovered via WAL replay from a genuine unclean termination. optimize()
's index-rebuild path excluded that WAL-replayed data from the rebuilt segment rather than merging it in.
A second experiment narrowed the loss boundary precisely. With a periodic flush()
cadence — every 100 documents, 551 total inserted — optimize()
on the crash-recovered collection left exactly 501 documents; IDs 501–550, everything inserted after the last flush()
at ID 500, were missing. The loss boundary is exactly the last durably-flushed segment, not "everything since the crash" and not "everything unflushed." A single experiment result is not treated as proof of a causal claim here — the crash-history-vs-no-crash-history comparison is what isolates the trigger.
Root cause, in one sentence: segment recovery replayed WAL records into a local WAL handle, then closed it without restoring the member WAL handle on the segment — so optimize()
's subsequent flush treated the writing segment as having no WAL records and skipped persisting them.
The patch to src/db/index/segment/segment.cc
:
@@ -4212,8 +4212,6 @@ Status SegmentImpl::recover() {
wal_file_path.c_str());
return Status::OK();
}
- AILEGO_DEFER([&]() { recover_wal_file->close(); });
-
std::array<uint64_t, static_cast<size_t>(Operator::DELETE) + 1>
recovered_doc_count{};
uint64_t total_recovered_doc_count{0};
@@ -4297,7 +4295,17 @@ Status SegmentImpl::recover() {
(size_t)recovered_doc_count[3] // DELETE
);
- return Status::OK();
+ if (recover_wal_file->close() != 0) {
+ return Status::InternalError("Failed to close recovered wal file: ",
+ wal_file_path);
+ }
+ recover_wal_file.reset();
+
+ // Keep the recovered WAL attached to the segment. Operations such as
+ // optimize() flush the writing segment before sealing it; without an open
+ // member WAL, flush() treats the recovered memory components as empty and
+ // returns without persisting them.
+ return open_wal_file();
}
Instead of discarding the recovery WAL handle once replay finishes, recover()
now reopens the segment's member WAL handle (open_wal_file()
) before returning. optimize()
's subsequent flush step then sees a live WAL attached to the segment and persists the recovered records instead of treating the segment as empty.
This is a handle-lifecycle bug, not a logic bug in the index-rebuild algorithm itself — which is also why the patch is narrow: three lines removed, roughly a dozen added, confined to the recovery path.
optimize()
" Isn't a Strategy
PR #600 closes the specific intersection of crash recovery and optimize()
. It does not change the underlying design: an in-process storage engine that owns its own, implicit durability contract, with no external signal exposed at the time of a call like whether a collection currently carries unreplayed crash-recovery state. The next unknown intersection between recovery and some other operation is a design property, not a one-off accident, and a single patch cannot retire that property.
ZVec's WAL-based recovery model has a boundary: the last segment durably flushed to disk. Everything after that boundary depends on the WAL being correctly replayed and correctly reattached on every code path that might act on it — which is exactly the class of invariant that PR #600 had to repair by hand.
Durable execution frameworks built on event sourcing, such as Temporal, do not have an equivalent boundary. A Workflow's state is not a periodically-flushed snapshot; it is the deterministic replay of its complete event history — every signal received, every activity result recorded — reconstructed from that history on every recovery. There is no concept of "the last flush point" for the replay to fall behind, because the event history is the durable state, not a cache of it.
A companion Temporal-based agent workflow used elsewhere demonstrates the pattern concretely. A long-running Workflow exposes signal handlers that mutate its in-memory state:
@workflow.signal
def add_task(self, task: str) -> None:
self._task_queue.append(task)
@workflow.signal
def update_task_priority(self, priority: int) -> None:
self._priority = max(1, min(10, priority))
@workflow.signal
def inject_human_feedback(self, feedback: str) -> None:
self._human_feedback = feedback
Every signal is recorded into Temporal's event history as it is received. This is not a proposal — it has an automated resilience test that actually exercises the failure mode: start the Workflow, send signals (inject feedback, change priority), forcibly kill the worker process, restart it, and query the Workflow's state to confirm the signal-driven changes survived the crash intact. The recovery path here is full deterministic replay of the event history against a fresh worker, not a WAL segment reattachment.
This does not make the underlying storage problem disappear entirely. ZVec still exposes no full-collection enumeration API, so any migration-based recovery workaround depends on the caller already knowing every document ID. That is precisely why, in an architecture where ZVec sits inside a Temporal-orchestrated pipeline, the orchestrator's own durable state — not the storage engine's internal bookkeeping — is the right place to keep the ID ledger of record.
As of this verification pass, pip index versions zvec
still reports 0.5.1
as latest — the same version the bug was originally found on. The merge commit containing the fix (e89be98b
) is not included in any published PyPI release. A v0.6.0
git tag exists in the repository, but comparing it against the fix commit shows it is behind the fix — it was cut before the patch landed. Re-running the original repro script against a released package is therefore not currently possible, and that limitation is stated here rather than worked around.
The change described above was confirmed directly against the PR's diff, not inferred from the PR description: the root-cause explanation ("the recovered WAL was not reattached to the segment's member handle") matches the actual patch (recover()
now calls open_wal_file()
before returning, instead of leaving the segment without a live WAL handle).
The PR adds a regression test that mirrors the originally reported reproduction almost exactly:
TEST_F(CrashRecoveryTest, OptimizeAfterCrashRecoveryPersistsReplayedData) {
// ... create collection, crash during insertion ...
auto result = Collection::Open(dir_path_, options_);
auto collection = result.value();
recovered_doc_count = collection->Stats().value().doc_count;
ASSERT_GT(recovered_doc_count, 0)
<< "No documents were recovered from the WAL";
auto status = collection->Optimize();
ASSERT_TRUE(status.ok()) << status.message();
ASSERT_EQ(collection->Stats().value().doc_count, recovered_doc_count);
// Reopen from a fresh handle and assert nothing was lost.
auto reopened = Collection::Open(dir_path_, options_);
ASSERT_EQ(reopened.value()->Stats().value().doc_count, recovered_doc_count)
<< "Optimize discarded documents restored from the WAL";
}
Crash, reopen, optimize()
, reopen again from a fresh handle, assert doc_count
is unchanged — the same crash-then-verify-from-a-fresh-process structure used to originally isolate the bug.
The PR description states that "the full CI matrix passed on the same commit: Linux, macOS, Windows, Android, iOS, lint, and clang-tidy." Pulling the actual check-runs from the GitHub Actions API tells a more precise story.
On the PR's own head commit (aaa4aab4
), the Build & Test jobs for every platform, along with lint
and clang-tidy
, show as skipped — consistent with a permission gate that withholds full CI from external-contributor pull requests until a maintainer authorizes it. "The same commit" did not, in fact, run the full matrix.
What actually ran, and passed, was the merge commit onto main (
e89be98b
). Its CI log shows:
66/152 Test #67: optimize_recovery_test ............................ Passed 354.84 sec
109/152 Test #68: write_recovery_test ............................... Passed 1687.70 sec
write_recovery_test
is the binary that contains the new OptimizeAfterCrashRecoveryPersistsReplayedData
case; it passed on the merge commit, along with the C++ test suite, the Python test suite, lint
, clang-tidy
, and a linux-x64
build. That is solid first-party evidence that the fix behaves as intended.
It is also worth being precise about what did not run: the Windows, macOS (multiple variants), iOS, and Android Build & Test jobs remained skipped even after the merge to main
. "Full CI matrix" is therefore accurate for the core Linux test suite and static analysis, but not for the full platform matrix implied by the PR description.
This verification did not include building ZVec from source and re-running the original Python repro script against the patched code — there is no PyPI release yet to install, and a from-source build was out of scope for this pass. The natural follow-up, once a release beyond 0.5.1
ships, is to re-run repro_zvec_optimize_dataloss.py
against the released package and confirm doc_count
no longer regresses to 0.