Zenoh's put is fire-and-forget, get isn't — a read-after-write race in Elixir

A developer experimenting with Zenoh's Elixir bindings, Zenohex, discovered a read-after-write race condition where a put followed immediately by a get occasionally returns stale data. The issue stems from Zenoh's put being fire-and-forget, only waiting for the local publish to be queued, while get waits for a remote reply. The developer proposes a wrapper that retries the get until the written payload is confirmed, and notes an open Zenoh issue tracking the same gap.

This English version is an AI translation of my original article on Qiita in Japanese https://qiita.com/kikuyuta/items/578bceafe60b4bb53d31 . I've been experimenting with Zenoh https://zenoh.io via its Elixir bindings, Zenohex https://hex.pm/packages/zenohex , not for its usual pub/sub use case but for its put / get storage feature. It mostly worked, except every so the state I picked back up was one step behind. Digging into why turned into a fun rabbit hole, so here's the writeup. To keep things simple, strip out the GenServer part entirely and just loop put immediately followed by get on the same key: php {:ok, session id} = Zenohex.Session.open config Enum.each 1..2000, fn i - payload = Integer.to string i :ok = Zenohex.Session.put session id, key, payload {:ok, replies} = Zenohex.Session.get session id, key, 3 000, consolidation: :latest case Enum.find replies, &match? %Zenohex.Sample{}, &1 do %Zenohex.Sample{payload: ^payload} - :ok %Zenohex.Sample{payload: other} - IO.puts "stale put {payload} but got {other}" nil - IO.puts "no reply at all" end end Out of 2000 iterations, a small fraction print stale — about 78 3.9% in one run. The interesting part: querying again immediately afterward almost always returns the correct value the fastest I measured was a single extra get about 1ms later . So it's not that the value disappears — there's just a small window of lag before the write is actually visible. Zenohex.Session.put/4 is a thin Rustler wrapper around zenoh-rust's put . Looking at the NIF implementation https://github.com/biyooon-ex/zenohex/blob/main/native/zenohex nif/src/session.rs : php fn session put ... - rustler::NifResult<rustler::Atom { ... publication builder .apply opts opts ? .wait // <- only waits for the local publish to be queued ... Ok rustler::types::atom::ok } .wait only waits for the local session to finish handing the message off — not for the remote side the zenohd router backing the storage to actually receive and apply it. session get , on the other hand, is registered as a DirtyIo NIF and genuinely waits for a reply from the remote side within a timeout — a real request/response. If you think of it in Elixir/GenServer terms, put behaves like cast and get behaves like call . Firing a cast and immediately assuming the effect is visible, then doing a call that depends on it, is exactly the kind of race this pattern invites. This isn't just me — zenoh itself has an open issue tracking the same gap: eclipse-zenoh/zenoh 2511 https://github.com/eclipse-zenoh/zenoh/issues/2511 " Design Acknowledged put: confirmed storage writes via query path vs protocol extension" , still open as of this writing. One line from it sums up the whole thing: Zenoh's pub/sub path is fire-and-forget — session.put returns when the message is sent, not when it's stored. A small wrapper: put , then get the same key right after, and only return once the written payload can actually be read back. Retry at a short interval until a timeout is reached. defmodule ZenohAckPut do @default confirm timeout ms 3 000 @default confirm interval ms 1 @default query timeout ms 3 000 def put session id, key expr, payload, put opts \\ , confirm opts \\ do confirm timeout ms = Keyword.get confirm opts, :confirm timeout ms, @default confirm timeout ms confirm interval ms = Keyword.get confirm opts, :confirm interval ms, @default confirm interval ms query timeout ms = Keyword.get confirm opts, :query timeout ms, @default query timeout ms with :ok <- Zenohex.Session.put session id, key expr, payload, put opts do deadline = System.monotonic time :millisecond + confirm timeout ms confirm session id, key expr, payload, query timeout ms, confirm interval ms, deadline end end defp confirm session id, key expr, payload, query timeout ms, confirm interval ms, deadline do if fetch session id, key expr, query timeout ms == payload do :ok else if System.monotonic time :millisecond = deadline do {:error, :not confirmed} else Process.sleep confirm interval ms confirm session id, key expr, payload, query timeout ms, confirm interval ms, deadline end end end defp fetch session id, key expr, query timeout ms do case Zenohex.Session.get session id, key expr, query timeout ms, consolidation: :latest do {:ok, replies} - case Enum.find replies, &match? %Zenohex.Sample{}, &1 do %Zenohex.Sample{payload: found payload} - found payload nil - nil end {:error, reason} - nil end end end Usage: iex ZenohAckPut.put session id, "key/expr", "payload" :ok Three possible return values: :ok — the put succeeded and the read-after-write confirmation also succeeded {:error, :not confirmed} — the put itself succeeded, but confirmation didn't land within the timeout this does {:error, reason} — the underlying put itself failedRunning the same 2000-iteration loop through ZenohAckPut.put instead: zero stale reads, zero unconfirmed timeouts. It's published as a standalone module, along with the reproduction scripts used above and a script that verifies the fix: Not on Hex yet, so pull it in as a git dependency for now: defp deps do {:zenohackput, git: "https://github.com/kikuyuta/zenohackput.git"} end Zenoh's put is fire-and-forget while get is a real request/response, and a get right after a put can occasionally return a stale value — a few percent of the time in my measurements. It's a known, currently-unresolved gap upstream. An application-level "put, then confirm with a get" wrapper is enough to close it in practice for use cases like state handoff that need read-your-own-writes. If you're using Zenoh's put / get for anything where you expect a write to be immediately visible — not just eventually — keep this asymmetry in mind.