Build a Reproducible Multi-Agent Pipeline on a Versioned Filesystem

Tensorlake's Cloud Volumes enable a reproducible five-agent pipeline by versioning the filesystem instead of teaching agents Git, allowing every run to be snapshotted, diffed, and restored on a fresh machine. The tutorial, based on nine years of experience with LangGraph, CrewAI, and AutoGen, outlines seven steps to partition worker outputs into subtrees and autosave writes, with the config baked into the message for diffing run #17 against run #18.

Here’s the part most multi-agent tutorials skip: making the run reproducible. Fan five agents out, and by the time they finish, their outputs are scattered across sandboxes that vanish on exit. Nothing versioned. Nothing to diff. This tutorial fixes that — without teaching a single agent Git. I’ve built fan-out pipelines for about nine years across LangGraph, CrewAI, and AutoGen, and the reproducibility gap has bitten me on every one. By the end of this tutorial you’ll have a five-agent pipeline whose every run is snapshot-able, diff-able, and restorable on a fresh machine — and you’ll know the exact flags and ordering that make it work the first time instead of the fifth. TL;DR:Version thefilesystem, not the agents. Give each worker its own subtree on a shared Tensorlake Cloud Volume, let writes autosave, snapshot at the end with the config baked into the message, and you can diff run 17 against run 18 or fork either onto a clean machine. No commits, no locks, no per-agent Git — just seven steps. Agents are unusually exposed to version drift: the same logical step depends on model version, prompt text, tool schema, and retrieval index, so if any of those shift before you re-run, replay diverges 2 . Left unversioned, agent output is a pile of mutable files you can save but can’t reason about 3 . The instinct is to teach every agent Git. It fights the grain — agents spray dozens of small, disjoint files as they go, not tidy commits. The cleaner move is to keep the version control and drop the per-agent ceremony 4 : capture the working state by versioning the filesystem itself, then restore it later to reproduce exact conditions 5 . Tensorlake’s Cloud Volumes give you a shared, durable, versioned POSIX directory that does exactly this 1 . Let’s build on it. Use the mount-free client for anything that just needs to persist output. It talks straight over HTTP through a native Rust core — no CLI, no FUSE, no root: python from tensorlake.filesystem import FilesystemClientclient = FilesystemClient reads TENSORLAKE API KEY / ORGANIZATION ID / PROJECT IDfs = client.create "pipeline-runs" fs.write file ".init", b"", message="initial commit" see the note below That empty write is just insurance, not a requirement. Send gives you one worker per item, then a single supervisor fan-in: python def fan out state : return Send "worker", { "agent index": i, "agent id": f"agent-{i}", "item": item, "run id": state "run id" , "filesystem": state "filesystem" , "sandbox id": state "sandbox id" , "token": state "token" , "temperature": state "temperature" , "prompt variant": state "prompt variant" , } for i, item in enumerate state "items" def build graph : graph = StateGraph PipelineState graph.add node "setup", setup graph.add node "worker", worker graph.add node "supervisor", supervisor graph.add edge START, "setup" graph.add conditional edges "setup", fan out graph.add edge "worker", "supervisor" runs once, after every worker finishes graph.add edge "supervisor", END return graph.compile Partition by subtree — runs/<run id /agents/<agent id — so no two workers ever touch the same path. Then mount, and get three details right that the docs gloss over: mount path = f"/home/tl-user/mnt-{agent id}"subtree = f"runs/{run id}/agents/{agent id}" 1 inside the user's home - no root needed run "tl", "sbx", "exec", sandbox id, "--detach", "--name", f"fsmount-{agent id}", "--user", "root", "-e", f"TENSORLAKE GIT TOKEN={token}", "--", "tl", "fs", "mount", filesystem, mount path deadline = time.monotonic + 20 2 poll before writingwhile time.monotonic < deadline: if subprocess.run ..., "tl", "fs", "status", mount path .returncode == 0: break time.sleep 1 else: raise RuntimeError f"{agent id}: mount never became ready in 20s" time.sleep stagger seconds agent index 3 stagger session startsworker dir = f"{mount path}/{subtree}" 5 subtree is a path inside the mount, not a mount argos.makedirs worker dir, exist ok=True 2 Poll tl fs status. --detach only confirms the launch command was accepted , not that the mount is live. Skip this and a failed mount silently writes to a plain local directory — you'll find out three steps downstream with a baffling head is unborn error. 3 Stagger the starts. Under heavy fan-out, space fresh session starts a couple of seconds apart. Partitioned writers coordinate cleanly with no locks once each has settled — a beat of spacing makes that rock-solid on every run. The supervisor is a single writer with nothing to race, so it skips the mount entirely and reads across every subtree through the fast mount-free API: python def run supervisor , filesystem, run id, worker results : fs = FilesystemClient .get filesystem sections = f" {r 'agent id' }\n\n Item: {r 'item' }\n\n" f"{ read with retry fs, r 'subtree' + '/output.md' .strip }\n" for r in sorted worker results, key=lambda r: r "agent id" summary = f" Run Summary — {run id}\n\n" + "\n".join sections fs.write file f"runs/{run id}/summary.md", summary.encode , message=f"supervisor summary for run {run id}" Use a small retry on the read: a worker’s write returning doesn’t guarantee it’s published yet, and a naive read right after fan-in can race it. This is what makes a run reproducible instead of merely saved. Bake the parameters into the snapshot message: fs = FilesystemClient .get filesystem snapshot = fs.snapshot f"run {run id}: temperature={temperature}, prompt variant={prompt variant}" snapshot id = snapshot.i Now every permanent checkpoint carries the exact config that produced it. Run the pipeline twice — once at temperature 0.0, once at temperature 0.9, prompt variant v2. Because both runs write to the same shared timeline, they already coexist and the diff needs no restore: fs = FilesystemClient .get filesystem diff = difflib.unified diff read fs, f"runs/{run a}/summary.md" .splitlines keepends=True , read fs, f"runs/{run b}/summary.md" .splitlines keepends=True , --- run-a/agents/agent-0/summary.md+++ run-b/agents/agent-0/summary.md@@ -1,3 +1,3 @@-A versioned filesystem automatically saves snapshots of your files...+A versioned filesystem is a storage system that keeps a history... That’s the whole payoff in one view: precisely what one config change did to five agents’ output. To reproduce an earlier state on a clean machine, fork a new filesystem pinned to an old snapshot. It never mutates the live timeline, which makes it safer than an in-place restore: restored = client.fork "restored-to-run1", base filesystem="pipeline-runs", snapshot=snap a.id restored.read file f"runs/{run a}/summary.md" presentrestored.read file f"runs/{run b}/summary.md" absent — run 2 didn't exist yet Fork forward from run 2’s snapshot and both appear. Real backward-and-forward time travel over real runs. A handful of things will trip you up if you follow the documentation literally. These are the ones I hit, with the fix baked in: Two scope boundaries so you point this at the right problem. This is working-state reproducibility — replaying the exact filesystem state of a run — not semantic memory; pair it with a vector store when you need recall-by-meaning. And it’s a single linear timeline , not a branching workflow; for agents exploring divergent branches you merge later, Tensorlake’s Git Repositories is the better-fit primitive. Inside its lane — reproduce a run, diff two runs, restore on a fresh box — this is the cleanest approach I’ve used. Start tiny: pip install tensorlake, then run Step 1 and Step 5 alone — create a filesystem, write a file, snapshot it, read it back. Under five minutes, no mount. Then wire Steps 2–4 with two dummy items and a single cheap LLM call per worker, and watch two subtrees land in one coherent tree. Once that works, run it twice at different temperatures and diff the summaries. The first time Step 6 shows you exactly what one prompt change did across five agents at once, reproducibility stops being the chore you keep meaning to add — and becomes something your pipeline just has. 1 Tensorlake — Sandboxes for AI Agents https://www.tensorlake.ai/?utm source=medium&utm medium=sponsored content&utm campaign=darshan aug2026 https://www.tensorlake.ai/?utm source=medium&utm medium=sponsored content&utm campaign=darshan aug2026 2 Zylos Research, “Durable Execution for AI Agent Runtimes: Checkpointing, Replay, and Recovery,” April 2026. https://zylos.ai/research/2026-04-24-durable-execution-agent-runtimes/ https://zylos.ai/research/2026-04-24-durable-execution-agent-runtimes/ 3 Freestyle, “Version Control for AI Agents,” May 2026. https://www.freestyle.sh/blog/engineering/version-control-for-ai-agents https://www.freestyle.sh/blog/engineering/version-control-for-ai-agents 4 “From Prompt–Response to Goal-Directed Systems: The Evolution of Agentic AI Software Architecture,” arXiv, Feb 2026. https://arxiv.org/html/2602.10479v1 https://arxiv.org/html/2602.10479v1 5 P. Enberg, “Towards a Disaggregated Agent Filesystem on Object Storage.” https://penberg.org/blog/disaggregated-agentfs.html https://penberg.org/blog/disaggregated-agentfs.html 6 Tensorlake Documentation — Sandboxes https://docs.tensorlake.ai/sandboxes/introduction?utm source=medium&utm medium=sponsored content&utm campaign=darshan aug2026 https://docs.tensorlake.ai/sandboxes/introduction?utm source=medium&utm medium=sponsored content&utm campaign=darshan aug2026 Build a Reproducible Multi-Agent Pipeline on a Versioned Filesystem https://pub.towardsai.net/build-a-reproducible-multi-agent-pipeline-on-a-versioned-filesystem-9d8c15dcf093 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.