{"slug": "build-a-reproducible-multi-agent-pipeline-on-a-versioned-filesystem", "title": "Build a Reproducible Multi-Agent Pipeline on a Versioned Filesystem", "summary": "Tensorlake's Cloud Volumes enable a reproducible five-agent pipeline by versioning the filesystem instead of teaching agents Git, allowing every run to be snapshotted, diffed, and restored on a fresh machine. The tutorial, based on nine years of experience with LangGraph, CrewAI, and AutoGen, outlines seven steps to partition worker outputs into subtrees and autosave writes, with the config baked into the message for diffing run #17 against run #18.", "body_md": "Here’s the part most multi-agent tutorials skip: making the run reproducible. Fan five agents out, and by the time they finish, their outputs are scattered across sandboxes that vanish on exit.\n\nNothing versioned. Nothing to diff. This tutorial fixes that — without teaching a single agent Git.\n\nI’ve built fan-out pipelines for about nine years across LangGraph, CrewAI, and AutoGen, and the reproducibility gap has bitten me on every one. By the end of this tutorial you’ll have a five-agent pipeline whose every run is snapshot-able, diff-able, and restorable on a fresh machine — and you’ll know the exact flags and ordering that make it work the first time instead of the fifth.\n\nTL;DR:Version thefilesystem, not the agents. Give each worker its own subtree on a shared Tensorlake Cloud Volume, let writes autosave, snapshot at the end with the config baked into the message, and you can diff run #17 against run #18 or fork either onto a clean machine. No commits, no locks, no per-agent Git — just seven steps.\n\nAgents are unusually exposed to version drift: the same logical step depends on model version, prompt text, tool schema, and retrieval index, so if any of those shift before you re-run, replay diverges [2]. Left unversioned, agent output is a pile of mutable files you can save but can’t reason about [3].\n\nThe instinct is to teach every agent Git. It fights the grain — agents spray dozens of small, disjoint files as they go, not tidy commits. The cleaner move is to keep the version control and drop the per-agent ceremony [4]: capture the working state by versioning the filesystem itself, then restore it later to reproduce exact conditions [5]. Tensorlake’s Cloud Volumes give you a shared, durable, versioned POSIX directory that does exactly this [1]. Let’s build on it.\n\nUse the mount-free client for anything that just needs to persist output. It talks straight over HTTP through a native Rust core — no CLI, no FUSE, no root:\n\n``` python\nfrom tensorlake.filesystem import FilesystemClientclient = FilesystemClient()          # reads TENSORLAKE_API_KEY / _ORGANIZATION_ID / _PROJECT_IDfs = client.create(\"pipeline-runs\")fs.write_file(\".init\", b\"\", message=\"initial commit\")   # see the note below\n```\n\nThat empty write is just insurance, not a requirement.\n\nSend gives you one worker per item, then a single supervisor fan-in:\n\n``` python\ndef fan_out(state):    return [        Send(\"worker\", {            \"agent_index\": i, \"agent_id\": f\"agent-{i}\", \"item\": item,            \"run_id\": state[\"run_id\"], \"filesystem\": state[\"filesystem\"],            \"sandbox_id\": state[\"sandbox_id\"], \"token\": state[\"token\"],            \"temperature\": state[\"temperature\"], \"prompt_variant\": state[\"prompt_variant\"],        })        for i, item in enumerate(state[\"items\"])    ]def build_graph():    graph = StateGraph(PipelineState)    graph.add_node(\"setup\", setup)    graph.add_node(\"worker\", worker)    graph.add_node(\"supervisor\", supervisor)    graph.add_edge(START, \"setup\")    graph.add_conditional_edges(\"setup\", fan_out)    graph.add_edge(\"worker\", \"supervisor\")   # runs once, after every worker finishes    graph.add_edge(\"supervisor\", END)    return graph.compile()\n```\n\nPartition by subtree — runs/<run_id>/agents/<agent_id> — so no two workers ever touch the same path. Then mount, and get three details right that the docs gloss over:\n\n```\nmount_path = f\"/home/tl-user/mnt-{agent_id}\"subtree    = f\"runs/{run_id}/agents/{agent_id}\"        # (1) inside the user's home - no root needed_run(\"tl\", \"sbx\", \"exec\", sandbox_id, \"--detach\", \"--name\", f\"fsmount-{agent_id}\",     \"--user\", \"root\",                                       \"-e\", f\"TENSORLAKE_GIT_TOKEN={token}\",     \"--\", \"tl\", \"fs\", \"mount\", filesystem, mount_path)deadline = time.monotonic() + 20                        # (2) poll before writingwhile time.monotonic() < deadline:    if subprocess.run([..., \"tl\", \"fs\", \"status\", mount_path]).returncode == 0:        break    time.sleep(1)else:    raise RuntimeError(f\"{agent_id}: mount never became ready in 20s\")time.sleep(stagger_seconds * agent_index)               # (3) stagger session startsworker_dir = f\"{mount_path}/{subtree}\"                     # (5) subtree is a path *inside* the mount, not a mount argos.makedirs(worker_dir, exist_ok=True)\n```\n\n**(2) Poll ****tl fs status.** --detach only confirms the launch command was *accepted*, not that the mount is live. Skip this and a failed mount silently writes to a plain local directory — you'll find out three steps downstream with a baffling head is unborn error.\n\n**(3) Stagger the starts.** Under heavy fan-out, space fresh session starts a couple of seconds apart. Partitioned writers coordinate cleanly with no locks once each has settled — a beat of spacing makes that rock-solid on every run.\n\nThe supervisor is a single writer with nothing to race, so it skips the mount entirely and reads across every subtree through the fast mount-free API:\n\n``` python\ndef run_supervisor(*, filesystem, run_id, worker_results):    fs = FilesystemClient().get(filesystem)    sections = [        f\"## {r['agent_id']}\\n\\n**Item:** {r['item']}\\n\\n\"        f\"{_read_with_retry(fs, r['subtree'] + '/output.md').strip()}\\n\"        for r in sorted(worker_results, key=lambda r: r[\"agent_id\"])    ]    summary = f\"# Run Summary — {run_id}\\n\\n\" + \"\\n\".join(sections)    fs.write_file(f\"runs/{run_id}/summary.md\", summary.encode(),                  message=f\"supervisor summary for run {run_id}\")\n```\n\nUse a small retry on the read: a worker’s write returning doesn’t guarantee it’s published yet, and a naive read right after fan-in can race it.\n\nThis is what makes a run reproducible instead of merely saved. Bake the parameters into the snapshot message:\n\n```\nfs = FilesystemClient().get(filesystem)snapshot = fs.snapshot(    f\"run {run_id}: temperature={temperature}, prompt_variant={prompt_variant}\")snapshot_id = snapshot.i\n```\n\nNow every permanent checkpoint carries the exact config that produced it.\n\nRun the pipeline twice — once at temperature 0.0, once at temperature 0.9, prompt_variant v2. Because both runs write to the *same* shared timeline, they already coexist and the diff needs no restore:\n\n```\nfs = FilesystemClient().get(filesystem)diff = difflib.unified_diff(    read(fs, f\"runs/{run_a}/summary.md\").splitlines(keepends=True),    read(fs, f\"runs/{run_b}/summary.md\").splitlines(keepends=True),)\n--- run-a/agents/agent-0/summary.md+++ run-b/agents/agent-0/summary.md@@ -1,3 +1,3 @@-A versioned filesystem automatically saves snapshots of your files...+A versioned filesystem is a storage system that keeps a history...\n```\n\nThat’s the whole payoff in one view: precisely what one config change did to five agents’ output.\n\nTo reproduce an earlier state on a clean machine, fork a *new* filesystem pinned to an old snapshot. It never mutates the live timeline, which makes it safer than an in-place restore:\n\n```\nrestored = client.fork(\"restored-to-run1\",                       base_filesystem=\"pipeline-runs\", snapshot=snap_a.id)restored.read_file(f\"runs/{run_a}/summary.md\")   # presentrestored.read_file(f\"runs/{run_b}/summary.md\")   # absent — run 2 didn't exist yet\n```\n\nFork forward from run 2’s snapshot and both appear. Real backward-and-forward time travel over real runs.\n\nA handful of things will trip you up if you follow the documentation literally. These are the ones I hit, with the fix baked in:\n\nTwo scope boundaries so you point this at the right problem. This is **working-state reproducibility** — replaying the exact filesystem state of a run — not semantic memory; pair it with a vector store when you need recall-by-meaning. And it’s a **single linear timeline**, not a branching workflow; for agents exploring divergent branches you merge later, Tensorlake’s Git Repositories is the better-fit primitive. Inside its lane — reproduce a run, diff two runs, restore on a fresh box — this is the cleanest approach I’ve used.\n\nStart tiny: pip install tensorlake, then run Step 1 and Step 5 alone — create a filesystem, write a file, snapshot it, read it back. Under five minutes, no mount.\n\nThen wire Steps 2–4 with two dummy items and a single cheap LLM call per worker, and watch two subtrees land in one coherent tree. Once that works, run it twice at different temperatures and diff the summaries. The first time Step 6 shows you exactly what one prompt change did across five agents at once, reproducibility stops being the chore you keep meaning to add — and becomes something your pipeline just has.\n\n[1] Tensorlake — Sandboxes for AI Agents] ( [https://www.tensorlake.ai/?utm_source=medium&utm_medium=sponsored_content&utm_campaign=darshan_aug2026](https://www.tensorlake.ai/?utm_source=medium&utm_medium=sponsored_content&utm_campaign=darshan_aug2026) )\n\n[2] Zylos Research, “Durable Execution for AI Agent Runtimes: Checkpointing, Replay, and Recovery,” April 2026. [https://zylos.ai/research/2026-04-24-durable-execution-agent-runtimes/](https://zylos.ai/research/2026-04-24-durable-execution-agent-runtimes/)\n\n[3] Freestyle, “Version Control for AI Agents,” May 2026. [https://www.freestyle.sh/blog/engineering/version-control-for-ai-agents](https://www.freestyle.sh/blog/engineering/version-control-for-ai-agents)\n\n[4] “From Prompt–Response to Goal-Directed Systems: The Evolution of Agentic AI Software Architecture,” arXiv, Feb 2026. [https://arxiv.org/html/2602.10479v1](https://arxiv.org/html/2602.10479v1)\n\n[5] P. Enberg, “Towards a Disaggregated Agent Filesystem on Object Storage.” [https://penberg.org/blog/disaggregated-agentfs.html](https://penberg.org/blog/disaggregated-agentfs.html)\n\n[6] Tensorlake Documentation — Sandboxes ([https://docs.tensorlake.ai/sandboxes/introduction?utm_source=medium&utm_medium=sponsored_content&utm_campaign=darshan_aug2026](https://docs.tensorlake.ai/sandboxes/introduction?utm_source=medium&utm_medium=sponsored_content&utm_campaign=darshan_aug2026))\n\n[Build a Reproducible Multi-Agent Pipeline on a Versioned Filesystem](https://pub.towardsai.net/build-a-reproducible-multi-agent-pipeline-on-a-versioned-filesystem-9d8c15dcf093) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/build-a-reproducible-multi-agent-pipeline-on-a-versioned-filesystem", "canonical_source": "https://pub.towardsai.net/build-a-reproducible-multi-agent-pipeline-on-a-versioned-filesystem-9d8c15dcf093?source=rss----98111c9905da---4", "published_at": "2026-08-26 05:05:31+00:00", "updated_at": "2026-08-26 05:12:37.440024+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools"], "entities": ["Tensorlake", "LangGraph", "CrewAI", "AutoGen", "Cloud Volumes", "FilesystemClient"], "alternates": {"html": "https://wpnews.pro/news/build-a-reproducible-multi-agent-pipeline-on-a-versioned-filesystem", "markdown": "https://wpnews.pro/news/build-a-reproducible-multi-agent-pipeline-on-a-versioned-filesystem.md", "text": "https://wpnews.pro/news/build-a-reproducible-multi-agent-pipeline-on-a-versioned-filesystem.txt", "jsonld": "https://wpnews.pro/news/build-a-reproducible-multi-agent-pipeline-on-a-versioned-filesystem.jsonld"}}