Why Your AI Experiments Keep Starting From Scratch (And How Tensorlake Fixes It)

Running parallel ML experiments often wastes time on redundant setup, such as reinstalling numpy and reloading datasets from scratch for each worker. Tensorlake addresses this with memory snapshots that capture both filesystem and VM memory state, allowing workers to resume from a warm state rather than cold booting. This approach reduces overhead by enabling workers to start from an identical execution point, similar to Unix fork(), rather than requiring full isolation from base images.

Running eight parallel ML experiments sounds efficient. Watching each one reinstall numpy from scratch does not. The logs told the story: eight sandbox workers, each spending its first 40 seconds on pip install numpy and loading the dataset before a single line of training code ran. The training scripts were different. The setup wasn’t. I had paid for the same 40 seconds of work eight times. The training scripts were the candidates. The setup was not. That is not a performance bug. It is a mental model problem. This article is the fix. If you want more such information about AI, consider subscribing to my newsletter, where you will get noise-free AI information every week Link for the newsletter: Newsletter https://aiengsimplified.beehiiv.com/ Most experiment setups quietly make an assumption: real worker isolation means booting from a clean image every time. No shared state. No residue from previous runs. That assumption is not wrong. It is just more expensive than it needs to be. Workers running parallel experiments need independent filesystems and independent process trees. They need a known starting state. What they do not need is to reinstall numpy, reload the same dataset from disk, or re-seed a Python environment that was identical across all of them at the start of the iteration. “Starting fresh” became a proxy for isolation. But the actual requirement is narrower: get every worker to the same state, then let them diverge. That is a different primitive than booting from a base image. Before I discovered how Tensorlake handles this, I assumed all snapshots worked the same way Docker images do. A filesystem snapshot captures disk state only. When you restore from one, the sandbox cold boots: the VM initializes from scratch, the OS comes up, Python starts, your process begins again. The installed packages are there on disk, so you skip the apt install and pip install steps. But you still pay for boot time and for every piece of process initialization that was not baked into the image. The interpreter loads. Modules get imported. Data gets pulled into RAM. I learned this the hard way. My first implementation used filesystem snapshots because they were the default. I launched eight workers from the same filesystem snapshot and watched every one of them cold boot, import NumPy, load the dataset, and rebuild the execution environment. I had skipped the package installation step, but I still paid for everything that came afterwards. Then I read the docs again and saw CheckpointType.MEMORY. A memory snapshot captures the filesystem plus the entire VM memory state, including all running processes at the exact moment of capture. Restore from one, and the sandbox resumes warm. The Python interpreter is already running. Modules imported before the checkpoint are already loaded. Variables that were in memory are still there. It is closer to Unix fork than to a Docker image. You are not restoring a filesystem. You are branching from a specific execution point. Tensorlake exposes filesystem and memory checkpoints as distinct primitives, making it possible to choose between cold restores and warm restores depending on the workload. python from tensorlake.sandbox import Sandbox, CheckpointType Cold-boot restore: captures disk state onlysnapshot = sandbox.checkpoint checkpoint type=CheckpointType.FILESYSTEM Warm restore: captures disk + VM memory + running processessnapshot = sandbox.checkpoint checkpoint type=CheckpointType.MEMORY The default when you call sandbox.checkpoint with no arguments is filesystem. For the fork pattern described in this article, you want memory. One tradeoff to understand upfront: memory snapshots lock the resource configuration at capture time.Image, CPU count, memory limit, entrypoint, and secrets all come from the snapshot when you restore. You cannot change them at restore time. If you need different resource allocations per worker, use filesystem snapshots and accept the cold boot. For most parallel experiment setups, the locked resources are fine since you sized the base environment for the workers when you created it. Secrets are the exception. They survive the checkpoint but are not locked to it. Secrets are passed as environment variables to run , so you can pass a new value at restore time and overwrite whatever was captured in the snapshot. Rotating an API key across forked workers does not require a new snapshot. Once you have a memory snapshot, the fork pattern is straightforward. Here is what it looks like end to end: Every worker starts from the same warm execution point. They share nothing at runtime. The base sandbox can be terminated right after the checkpoint. The snapshot id persists independently. Step 1: Create one base sandbox and do all shared setup inside it. python from tensorlake.sandbox import Sandbox, CheckpointTypebase = Sandbox.create image="tensorlake/ubuntu-minimal", cpus=2, memory mb=4096, Install shared dependencies oncebase.run "pip", "install", "numpy", "pandas", "--break-system-packages" Verify the environment is readybase.run "python3", "-c", "import numpy; import pandas; print 'Environment ready.' " Step 2: Capture the warm state. snapshot = base.checkpoint checkpoint type=CheckpointType.MEMORY print f"Snapshot captured: {snapshot.snapshot id}" The base sandbox can be terminated. The snapshot persists independently.base.terminate Step 3: Restore N workers in parallel from the same checkpoint. python from concurrent.futures import ThreadPoolExecutordef run experiment script: str - str: Each worker restores a fresh copy of the warm environment worker = Sandbox.create snapshot id=snapshot.snapshot id result = worker.run "python3", "-c", script worker.terminate return result.stdoutcandidates = script v1, script v2, script v3, script v4, script v5 with ThreadPoolExecutor max workers=len candidates as pool: results = list pool.map run experiment, candidates Each Sandbox.create snapshot id=... call restores an independent copy of the warm environment. Workers share no filesystem and no runtime state. They just started from the same point. The snapshot.snapshot id is a persistent string. Save it to a file. Use it in the next session, in the next iteration of your loop, in a different process entirely. The warm state survives until you explicitly delete the snapshot. Andrej Karpathy published the autoresearch repo in early 2026. The core idea: an LLM reads your current best training script, proposes N code modifications, you race all N in parallel sandboxes, keep the winner, and loop. The loop runs overnight . Each accepted modification becomes the new baseline for the next iteration. The naive implementation pays the setup cost N times per iteration. If the training script needs NumPy, Scipy, and a small dataset loaded into memory, that is real time per candidate, per loop. With 8 iterations and 3 candidates each, you have paid an identical setup cost 24 times. With Tensorlake’s Snapshot Fork Pattern , you do the setup once. Take a memory checkpoint. Fork all candidates from that checkpoint. They start warm and only diverge where the proposed modification changes behavior. The structure of one iteration looks like this: Build the warm baseline once per iterationbase = Sandbox.create image="tensorlake/ubuntu-minimal", cpus=2, memory mb=4096 base.run "pip", "install", "numpy", "scipy", "--break-system-packages" base.run "python3", "-c", "import dataset; dataset.load into memory " Checkpoint once: all candidates share this starting pointsnapshot = base.checkpoint checkpoint type=CheckpointType.MEMORY base.terminate Race candidates from the warm snapshot in parallel Each worker starts where the base left off, not from a fresh imagewith ThreadPoolExecutor max workers=len candidates as pool: results = list pool.map lambda s: run in sandbox snapshot.snapshot id, s , candidates Keep the winner, discard the rest, repeatwinner = min results, key=lambda r: r "val loss" The snapshot.snapshot id is reused across all candidates in the same iteration. Setup cost is paid once per loop, not once per candidate. RL rollouts have a hard requirement: same seed, same action sequence, same trajectory. Every time, without exception. That guarantee breaks the moment workers share any state. A shared pip cache can introduce version skew between runs. A shared /tmp carries residual files from previous episodes. Even calling env.reset correctly does not help when state outside the environment object persists between episodes. With per-rollout sandboxes forked from a common snapshot, the isolation is structural rather than enforced by convention. There is no shared filesystem between workers. The seed goes directly into the Python script that runs inside the sandbox, not into the host process. Keeping it there means the host’s random state stays completely out of the episode: php import jsonfrom tensorlake.sandbox import Sandboxdef gym harness seed: int - str: Script runs inside the sandbox, not the host return f"""import gymnasium as gym, jsonenv = gym.make "CartPole-v1" obs, = env.reset seed={seed} env.action space.seed {seed} trajectory, total reward = , 0.0for in range 200 : action = env.action space.sample next obs, reward, terminated, truncated, = env.step action trajectory.append obs.tolist , int action , float reward , bool terminated total reward += reward obs = next obs if terminated or truncated: breakprint json.dumps {{"seed": {seed}, "total reward": total reward, "steps": len trajectory }} """def run rollout snapshot id: str, seed: int - dict: worker = Sandbox.create snapshot id=snapshot id result = worker.run "python3", "-c", gym harness seed worker.terminate return json.loads result.stdout One specific gotcha here: env.reset seed=seed only seeds the observation and transition RNG. The action space has its own separate RNG that requires env.action space.seed seed independently. Miss the second call and trajectories vary across runs with no obvious error. The mismatch is silent and will look like flaky results until you trace it to the gymnasium source. Browser warmup is slow: login flows, OAuth, session cookies, waiting for JavaScript-heavy pages to stabilize. If you are running 20 browser workers in parallel, you do not want each one repeating that from scratch. Tensorlake’s ubuntu-vnc sandbox image makes this practical because the authenticated browser itself becomes part of the checkpoint. Complete the auth flow once, checkpoint the authenticated browser state, then fork your workers from there: python from tensorlake.sandbox import Sandbox, CheckpointType Authenticate once in a single browser sandboxbrowser base = Sandbox.create image="tensorlake/ubuntu-vnc", cpus=4, memory mb=4096 ... drive Chrome via CDP, complete login, reach stable page state ... Checkpoint the authenticated browser in memoryauth snapshot = browser base.checkpoint checkpoint type=CheckpointType.MEMORY browser base.terminate All 10 workers start with the logged-in browser state already in memoryworkers = Sandbox.create snapshot id=auth snapshot.snapshot id for in range 10 The same pattern that eliminates repeated pip install across ML workers eliminates repeated OAuth flows across browser workers. The primitive is the same. Only the warmup content changes. Suspend and snapshot both preserve sandbox state. They solve different problems and it is worth being clear about which one you reach for. Suspend pauses this specific sandbox and holds its state for later resumption under the same sandbox ID. I t uses no compute while suspended. When you resume, the sandbox comes back in under a second with the same process IDs, filesystem, and in-memory state intact. Named sandboxes auto-suspend on timeout instead of terminating, which means you do not lose a long-running agent session because a task ran slightly over its time budget. Snapshot captures a reusable artifact you can restore into new sandboxes. The artifact persists after the source sandbox is terminated. Restore from it once or many times. Suspend is for pausing and resuming a single workstream. Snapshot is for branching from a known state into N parallel workers. The sandbox keeps running while the snapshot is being captured. The snapshot artifact persists regardless of what happens to the source sandbox afterward. Unix fork exists for the same reason. Process creation is expensive, so instead of spawning a child from scratch, you copy the parent's entire memory state and let the child diverge from there. Git branches for the same reason: you checkpoint a known state and branch from it because re-deriving full history for every branch would be wasteful. AI experiment infrastructure got filesystem snapshots first. But filesystem snapshots only moved the cost boundary to disk. The running process, loaded modules, in-memory data, and interpreter state all had to rebuild on every new worker anyway. Memory checkpointing moves that boundary further. The running process is part of what gets captured. Load a 500MB dataset into a pandas DataFrame before the checkpoint, and every forked worker starts with that DataFrame already in RAM. Pre-compile JIT functions and the cache is there too. The interpreter does not restart. Nothing reloads. The expensive part of an experiment is rarely the experiment. It is everything that had to be true before the experiment could start. That cost is not fixed. It just looked fixed because the tooling treated it that way. Three numbers that matter for the fork pattern specifically. Speed. The tensorlake/ubuntu-minimal image starts up in a few hundred milliseconds; tensorlake/ubuntu-systemd full init system takes around one second. Forked workers restore from a memory snapshot warm, skipping boot and process initialization entirely. Suspended named sandboxes resume in under a second without losing any memory or filesystem state. The published SQLite benchmark https://github.com/tensorlakeai/sandbox-sqlite-bench 100k inserts, 2 vCPU / 4GB shows Tensorlake at 2.45s against Vercel at 3.00s, E2B at 3.92s, Modal at 4.66s, and Daytona at 5.51s. Scale. Tensorlake supports fanning out to thousands concurrent sandbox environments — the number the Harbor integration targets for RL rollouts and eval pipelines running from a single snapshot. The overall project limit is 5 million sandboxes. Isolation. Sandboxes run on MicroVMs backed by Firecracker and CloudHypervisor. LLM-generated code never shares a kernel with other tenants or with your host process. Tensorlake is SOC 2 Type II and HIPAA-compliant, with EU data residency and zero-data-retention options available. Install the SDK and grab an API key from cloud.tensorlake.ai https://cloud.tensorlake.ai free tier available : pip install tensorlakeexport TENSORLAKE API KEY=your api key Minimal working fork pattern you can run right now: python from tensorlake.sandbox import Sandbox, CheckpointTypefrom concurrent.futures import ThreadPoolExecutor Step 1: Warm the base environmentbase = Sandbox.create cpus=2, memory mb=4096 base.run "pip", "install", "numpy", "--break-system-packages" Step 2: Checkpoint the warm statesnapshot = base.checkpoint checkpoint type=CheckpointType.MEMORY base.terminate Step 3: Fork workers from the snapshotdef run script: str - str: w = Sandbox.create snapshot id=snapshot.snapshot id out = w.run "python3", "-c", script .stdout w.terminate return outscripts = "print 'v1' ", "print 'v2' ", "print 'v3' " with ThreadPoolExecutor max workers=3 as pool: results = list pool.map run, scripts print results All three workers start warm. snapshot.snapshot id is a persistent string you can store and reuse across sessions. Those eight pip install numpy calls were not a performance bug.They were a modeling mistake. I had built my experiment loop around "start fresh" when what I actually needed was "start identical." The difference is real. Starting fresh means paying full setup cost per worker. Starting identical means paying it once and forking from there. Tensorlake’s memory checkpoints provide one implementation of this pattern: build the environment once, capture the execution state, and fork as many isolated workers as you need from the same warm baseline. You do the expensive shared work once, freeze the execution state, and branch as many workers as you need from that exact point. Each worker gets its own isolated environment. The setup cost is paid once. This applies to most parallel AI workflows: ML candidate racing, RL rollouts, browser agent pools, CI evaluation pipelines. The pattern is the same across all of them. If I could go back to those eight workers all stuck on pip install numpy, the fix would have been four lines. Create the base. Warm it. Call checkpoint MEMORY . Restore from there. The 320 seconds of identical setup would have happened once. The other seven copies would never have existed. That is what the pattern is for. Why Your AI Experiments Keep Starting From Scratch And How Tensorlake Fixes It https://pub.towardsai.net/why-your-ai-experiments-keep-starting-from-scratch-and-how-tensorlake-fixes-it-2e6091edf143 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.