{"slug": "i-ran-5-ai-agents-in-parallel-on-tensorlake-the-isolation-held-here-is-how-i-it", "title": "I Ran 5 AI Agents in Parallel on Tensorlake. The Isolation Held. Here Is How I Built It.", "summary": "A developer built a five-agent data analysis pipeline on Tensorlake and confirmed that the platform's MicroVM-based sandbox isolation prevents agent crashes, filesystem conflicts, and memory contention from propagating between agents, even under deliberate crash injection. The pipeline uses separate sandboxed environments for each agent, making multi-agent parallelism structurally safe without requiring defensive coding.", "body_md": "In shared-runtime multi-agent setups, three failure modes make defensive coding necessary: a crashed agent process takes down the whole executor, filesystem writes from one agent bleed into another’s working directory, a memory-heavy agent degrades every other process on the same host. You end up writing code against each of these, one by one.\n\nTensorlake contains all three at the infrastructure level. Each agent gets its own MicroVM with its own process, filesystem, and memory. A crash stays in its sandbox and cannot propagate. I built a five-agent data analysis pipeline to test whether that isolation held under real conditions — including a deliberate crash injection. It did. This article is a full account of how the pipeline was built and what the results showed.\n\nWhen you run multiple AI agents in a single pipeline — statistical analysis, forecasting, anomaly detection — you typically run them in a shared runtime: the same Python process, the same server, the same filesystem. That produces three failure modes. One unhandled exception or OOM error kills the process running all agents — every agent’s work, gone. Agents share a filesystem, so if Agent A writes to /workspace/data/output.csv and Agent B reads the same path expecting its own output, it gets Agent A's data. And one agent loading a large model consumes host RAM that every other agent depends on.\n\nNone of these are prevented structurally, so you write compensation code — try/except blocks, output validation, quarantine logic — and still discover the failure modes you didn’t anticipate in production.\n\nIn a conventional multi-agent setup, isolation is something you engineer. You catch exceptions, you validate outputs, you design handoffs defensively. You write a lot of code to compensate for the fact that your agents share a runtime.\n\nTensorlake flips this. Isolation is structural — it comes from the infrastructure, not from your code. Each Sandbox.create() call gives you a fresh, fully provisioned Linux environment that runs independently of every other sandbox on your account. The agents don't share memory. They don't share a filesystem. They don't share a process. If one of them crashes, the others don't know it happened.\n\nThis is what I wanted to test with a real pipeline: five agents with distinct analytical responsibilities, running in parallel, with a deliberate crash injection on the fourth to verify that the other three completed cleanly.\n\nTL;DR:Tensorlake’s sandbox isolation makes multi-agent parallelism structurally safe — not just defensively coded. This article shows how to build a 5-agent pipeline from scratch, including the implementation details that make it production-honest.\n\nThe pipeline has five agents:\n\nEach analyst agent has its own Python module, its own system prompt, its own computation code, and its own sandbox. The Aggregator has none of those — its job is reasoning over text, not running code. Mixing sandbox agents with no-sandbox agents is a natural pattern: use isolated compute where you need it, skip it where you don’t.\n\nBefore any agent can run, you need a registered image. Understanding what this actually is — and why it matters for a multi-agent pipeline — is worth a moment.\n\nA registered image is a named snapshot: define it once, and every Sandbox.create(image=\"analyst-agent-image\") boots a fresh MicroVM from that snapshot with packages already installed and no runtime setup. The SDK uses a fluent builder:\n\n```\nimage = (    Image(base_image=\"python:3.11-slim\")    .run(\"useradd -m tl-user\")    .run(\"pip install numpy pandas scikit-learn scipy statsmodels anthropic requests\"))image.build(registered_name=\"analyst-agent-image\", cpus=1.0, memory_mb=1024)\n```\n\nWhen you call image.build(), Tensorlake executes your build steps and materialises the result as a snapshot (~241 MB for this pipeline) stored in the registry. Every Sandbox.create() after that boots a fresh MicroVM from that snapshot — packages already installed, user already created, filesystem exactly as it was at build time.\n\n**Use a multi-platform base image.** If you’ve built a Docker image locally on Apple Silicon (ARM) and pushed it to Docker Hub, it will have only an ARM manifest — no linux/amd64 entry. Tensorlake’s builder sandbox runs on Linux x86_64, so it cannot pull the image and will fail at the first step. Use python:3.11-slim from Docker Hub, which ships as a proper multi-platform image with both ARM and AMD64 manifests. Any official Python image works; custom images pushed from a Mac do not unless you explicitly build with --platform linux/amd64.\n\n**Create tl-user as your first build step.** Tensorlake’s sandbox runtime expects a system user named tl-user to exist inside every image. This is the user under which your commands and processes run inside the sandbox. The python:3.11-slim base creates no such user. Add .run(\"useradd -m tl-user\") as the very first step — before any pip installs — and the sandbox boots cleanly. Skip it, and you'll see a 500 error on the first command you try to run inside a sandbox booted from that image. It's one line, and it has to be first.\n\nThe build completed in 67.5 seconds. That’s a one-time cost — every agent boot after this is free of it.\n\nThe payoff shows up immediately in the boot benchmark:\n\n```\nBoot method                          | Time-------------------------------------|--------Cold boot + pip install at runtime   | 19.49sBoot from analyst-agent-image        | 4.51sSpeedup                              | 4.32x\n```\n\nThe cold path boots a blank sandbox and runs pip at runtime on every single launch. For a single agent running once, that 19.5 seconds is annoying. For a pipeline launching four agents, the math changes: even in parallel, each of those four agents spends 19.5 seconds installing packages before it touches the dataset. With the registered image, they boot in 4.5 seconds each and go straight to computation.\n\nThe 67.5-second build cost pays for itself after four agent runs. Every run after that is pure gain. For a production pipeline that runs dozens of times a day, or a development loop where you’re iterating on agent logic and re-running frequently, the registered image is not optional — it’s the decision that makes the architecture practical.\n\nEach analyst agent follows the same structure: boot a sandbox, write the dataset into it, run the computation code, call the Claude API with the raw numbers, and return a structured JSON result.\n\n``` php\nclass StatisticianAgent:    def call(self, user_message: User) -> Assistant:        sb = boot_sandbox()        try:            write_data_to_sandbox(sb, dataset_csv)            result = run_python_in_sandbox(sb, STATS_CODE)            response = llm_call(system=SYSTEM_PROMPT, user=result)            return Assistant(content=response)        finally:            sb.terminate()\n```\n\nA few implementation notes:\n\nWith four working agents, the orchestrator fans them out using ThreadPoolExecutor:\n\n```\nwith ThreadPoolExecutor(max_workers=4) as executor:    futures = {executor.submit(agent.call, user_msg): name               for name, agent in agents.items()}    results = {name: future.result() for future, name in futures.items()}\n```\n\nOn a paid account with quota >= 4, all four sandboxes boot simultaneously. The total parallel phase takes approximately as long as the slowest single agent (~20 seconds). On the free tier, quota is capped at one concurrent sandbox — agents queue. The fix is a retry loop in boot_sandbox():\n\n``` php\ndef boot_sandbox(max_retries=12, retry_delay=8.0) -> Sandbox:    for attempt in range(max_retries):        try:            return Sandbox.create(image=AGENT_IMAGE, timeout_secs=300)        except Exception as e:            err = str(e).lower()            if (\"quota\" in err or \"limit\" in err or \"capacity\" in err) and attempt < max_retries - 1:                time.sleep(retry_delay)            else:                raise\n```\n\nThe broadened error check — catching “quota”, “limit”, and “capacity” — makes this resilient to minor SDK wording changes between versions.\n\nOn the free tier, the four-agent parallel phase ran in 99.02 seconds total. On a paid account, the same code delivers true parallelism: all four agents boot, compute, and report simultaneously, collapsing the parallel phase from ~99 seconds to ~20 seconds.\n\nThe dataset was 90 days of synthetic daily sales revenue with an $800 spike injected on day 45.\n\n**Statistician:** Near-symmetric distribution (skewness +0.17), mean $1,467, median $1,492. The less-than-2% gap between mean and median confirmed no significant outlier distortion. Negative kurtosis (-0.70) means the distribution is broadly spread with lighter tails than normal. The day-45 maximum of $2,312.86 sits 2.87 standard deviations above the mean — notable but within expected range for a trending dataset.\n\n**Trend Analyst:** Statistically definitive upward trajectory (p < 0.0001), slope +$10.18/day, R-squared 81.6%. Week-over-week growth decelerating sharply: W1 to W2 +10%, W2 to W3 +5.9%, W3 to W4 +3.2% — momentum down 68% from initial pace. The final 7-day rolling average of $1,851.69 sat $632 above Week 4’s weekly average of $1,220. That discrepancy became the central finding of the synthesis.\n\n**Anomaly Detector:** Day 45 flagged by Z-score, cleared by IQR. The spike at $2,312.86 sits $96 below the IQR upper fence ($2,409) and passes that test. Z-score flagged it because the rising trend component lifts the mean, making the spike statistically unusual relative to it. IQR and Z-score disagreement is a useful teaching moment: the methods ask different questions, and neither result is wrong.\n\n**Forecaster:** Linear regression trained on days 1–72, evaluated on days 73–90. R-squared 0.76, holdout MAE $96.92. Day 92 forecast: $2,001 — the $2,000/day threshold crossed. Day 100 forecast: $2,090, with a realistic range of $1,993–$2,187 given the +/-$97/day uncertainty. Directional signal is credible; precise daily figures are not.\n\n**Aggregator:** Cross-referenced all four reports and surfaced the finding no individual agent had flagged — the late-period rolling average discrepancy ($1,851 vs $1,220 weekly average) is either a structural step-change in the business or a transient surge that would make the linear forecast materially wrong. That cross-agent insight is the design argument for the architecture: synthesis across independent agents catches things that single-agent analysis misses.\n\n**CrashAgent:** Run in a separate isolation test alongside three normal agents, it booted its own sandbox, confirmed the environment was live, then raised a deliberate RuntimeError. The crash stayed contained — its sandbox terminated cleanly, and all three normal agents completed with clean results, never seeing the failure. The full test is covered in the crash isolation section below\n\nTotal pipeline time: 129.15 seconds (105.37s parallel phase + 24.16s aggregator LLM call). Theoretical sequential equivalent: 238.79 seconds. On a paid account with true parallel execution, the pipeline collapses to approximately 44 seconds.\n\nThe crash test is the most important thing to run — not because the result is uncertain, but because confirming it with real sandbox IDs and real output is worth more than a design document.\n\nThree normal agents plus one designed to fail. All four ran simultaneously via ThreadPoolExecutor. Each booted its own sandbox — confirmed by distinct sandbox IDs in the results:\n\n```\nAgent        | Sandbox ID               | Result-------------|--------------------------|-------------------------Statistician | 4zglyrupcz07t26sdediz    | completedForecaster   | 9hjtp82ra1f9kmc6qk932    | completedTrendAnalyst | 9fps4yd492jgpi0ckwoim    | completedCrashAgent   | 34vz3cgzydtpim4kmc3r6    | crashed - contained\n```\n\nThe CrashAgent was designed carefully: it first ran python3 -c \"print('sandbox live')\" to confirm it had a working sandbox, then raised RuntimeError. The crash happened inside a live, provisioned environment — not in setup, not during boot. This is the realistic failure mode: an agent that gets far enough to be dangerous before it fails.\n\nThe sequence in the logs: on the free tier, Statistician booted first and got the only available quota slot. The other three queued via the retry loop. As each normal agent finished and released its slot, the next booted. CrashAgent was last. Its RuntimeError propagated up through call(), the finally block terminated its sandbox, and the exception was caught by the orchestrator's future.result() call. The three normal agents had already completed. They never saw the crash happen.\n\n```\nISOLATION CONFIRMED - crash contained, 3/3 normal agents completedTotal time: 84.72s\n```\n\nThe critical point: the containment was structural, not a timing artifact. Even if CrashAgent had run first — or concurrently with all three normal agents — the outcome would be identical. The sandboxes have no shared memory, no shared filesystem, no shared process. There is no mechanism by which a RuntimeError in one MicroVM can reach another. The failure cannot propagate regardless of execution order or concurrency.\n\nCompare this to a shared-runtime setup. There, a crash that corrupts shared state — a half-written file in a common directory, an exception that unwinds a shared object, a process that holds a lock on a resource others need — requires the orchestrator to detect, quarantine, and recover. You write that logic once per failure mode you anticipate, and you discover the ones you didn’t anticipate in production.\n\nWith sandbox isolation, the failure modes from the opening of this article — crashed process taking down the executor, filesystem bleed, memory degradation — are structurally prevented. The orchestrator handles the exception from CrashAgent, logs it, and moves on. The three normal agents’ results are clean. No defensive code was needed to produce that outcome.\n\nThis pipeline uses linear regression for forecasting. For business-critical projections, ARIMA or Prophet captures seasonality that linear regression misses, and prediction intervals give a more honest picture than point estimates.\n\nThe parallelism speedup measured here (2.27x) reflects free-tier quota constraints, not architectural limits. On a paid account with concurrent sandbox quota, the same code delivers true parallelism: all four agents run simultaneously, and the total pipeline time drops from 129 seconds to approximately 44 seconds.\n\nThe Aggregator’s synthesis quality depends entirely on the individual agents’ output structure. Define the JSON schema each agent returns explicitly, validate it before passing to the Aggregator, and consider adding a schema enforcement step in base.py if the pipeline grows.\n\nBoot a single sandbox from a registered image and run a print(\"hello\"). That confirms the image is correctly built, tl-user exists, and your account quota is available. One working sandbox is a better foundation than a broken orchestrator.\n\nAdd one agent — the Statistician is the simplest — and verify the full loop: boot, write data, run computation, LLM call, terminate. Measure the elapsed time. If you’re seeing ~4.5s boot and ~10–15s LLM call, everything is working correctly.\n\nWire the ThreadPoolExecutor orchestrator next, and add the boot_sandbox() retry loop before the first run. The crash isolation test is the last step — but run it. Knowing that your architecture structurally contains failures is worth the 85 seconds of confirmation.\n\nThe executive summary that correctly cross-referenced four independent analytical perspectives — and surfaced a discrepancy no single agent had caught — is what this architecture is for. Build toward it.\n\n[1] Tensorlake Python SDK documentation — Image, Sandbox, rootfs builder: [https://docs.tensorlake.ai](https://docs.tensorlake.ai/)\n\n[2] Tensorlake GitHub — SDK source, Sandbox, Image, CLI reference, FAQ on MicroVM isolation vs E2B/Modal/Daytona: [https://github.com/tensorlakeai/tensorlake](https://github.com/tensorlakeai/tensorlake)\n\n[3] Tensorlake Sandbox Images — registered images, Image() builder API, base images, .tlsnap materialisation: [https://docs.tensorlake.ai/sandboxes/images](https://docs.tensorlake.ai/sandboxes/images)\n\n[4] Anthropic Messages API — /v1/messages endpoint reference, model IDs, authentication, error codes (including 401 authentication_error): [https://docs.anthropic.com/en/api/messages](https://docs.anthropic.com/en/api/messages)\n\n[I Ran 5 AI Agents in Parallel on Tensorlake. The Isolation Held. Here Is How I Built It.](https://pub.towardsai.net/i-ran-5-ai-agents-in-parallel-on-tensorlake-the-isolation-held-here-is-how-i-built-it-769384e046cf) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/i-ran-5-ai-agents-in-parallel-on-tensorlake-the-isolation-held-here-is-how-i-it", "canonical_source": "https://pub.towardsai.net/i-ran-5-ai-agents-in-parallel-on-tensorlake-the-isolation-held-here-is-how-i-built-it-769384e046cf?source=rss----98111c9905da---4", "published_at": "2026-07-10 05:25:58+00:00", "updated_at": "2026-07-10 05:39:53.280050+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools"], "entities": ["Tensorlake"], "alternates": {"html": "https://wpnews.pro/news/i-ran-5-ai-agents-in-parallel-on-tensorlake-the-isolation-held-here-is-how-i-it", "markdown": "https://wpnews.pro/news/i-ran-5-ai-agents-in-parallel-on-tensorlake-the-isolation-held-here-is-how-i-it.md", "text": "https://wpnews.pro/news/i-ran-5-ai-agents-in-parallel-on-tensorlake-the-isolation-held-here-is-how-i-it.txt", "jsonld": "https://wpnews.pro/news/i-ran-5-ai-agents-in-parallel-on-tensorlake-the-isolation-held-here-is-how-i-it.jsonld"}}