{"slug": "why-your-ai-experiments-keep-starting-from-scratch-and-how-tensorlake-fixes-it", "title": "Why Your AI Experiments Keep Starting From Scratch (And How Tensorlake Fixes It)", "summary": "Running parallel ML experiments often wastes time on redundant setup, such as reinstalling numpy and reloading datasets from scratch for each worker. Tensorlake addresses this with memory snapshots that capture both filesystem and VM memory state, allowing workers to resume from a warm state rather than cold booting. This approach reduces overhead by enabling workers to start from an identical execution point, similar to Unix fork(), rather than requiring full isolation from base images.", "body_md": "Running eight parallel ML experiments sounds efficient.\n\nWatching each one reinstall numpy from scratch does not.\n\nThe logs told the story: eight sandbox workers, each spending its first 40 seconds on pip install numpy and loading the dataset before a single line of training code ran. The training scripts were different.\n\nThe setup wasn’t. **I had paid for the same 40 seconds of work eight times.**\n\nThe training scripts were the candidates. The setup was not.\n\nThat is not a performance bug. It is a mental model problem.\n\nThis article is the fix.\n\nIf you want more such information about AI, consider subscribing to my newsletter, where you will get noise-free AI information every week\n\n**Link for the newsletter:** [Newsletter](https://aiengsimplified.beehiiv.com/)\n\n**Most experiment setups quietly make an assumption: **real worker isolation means booting from a clean image every time. No shared state. No residue from previous runs.\n\n**That assumption is not wrong.**\n\nIt is just more expensive than it needs to be.\n\nWorkers running parallel experiments need independent filesystems and independent process trees. They need a known starting state. What they do not need is to reinstall numpy, reload the same dataset from disk, or re-seed a Python environment that was identical across all of them at the start of the iteration.\n\n“Starting fresh” became a proxy for isolation.\n\nBut the actual requirement is narrower: get every worker to the same state, then let them diverge. That is a different primitive than booting from a base image.\n\nBefore I discovered how **Tensorlake** handles this, I assumed all snapshots worked the same way Docker images do.\n\nA filesystem snapshot captures disk state only. When you restore from one, the sandbox cold boots: the VM initializes from scratch, the OS comes up, Python starts, your process begins again. The installed packages are there on disk, so you skip the apt install and pip install steps. But you still pay for boot time and for every piece of process initialization that was not baked into the image.\n\nThe interpreter loads. Modules get imported. Data gets pulled into RAM.\n\n**I learned this the hard way.**\n\nMy first implementation used filesystem snapshots because they were the default. I launched eight workers from the same filesystem snapshot and watched every one of them cold boot, import NumPy, load the dataset, and rebuild the execution environment.\n\nI had skipped the package installation step, but I still paid for everything that came afterwards.\n\nThen I read the docs again and saw CheckpointType.MEMORY.\n\n**A memory snapshot captures the filesystem plus the entire VM memory state, including all running processes at the exact moment of capture.**\n\nRestore from one, and the sandbox resumes warm. The Python interpreter is already running. Modules imported before the checkpoint are already loaded. Variables that were in memory are still there.\n\n**It is closer to Unix ****fork() than to a Docker image.** You are not restoring a filesystem. You are branching from a specific execution point.\n\nTensorlake exposes filesystem and memory checkpoints as distinct primitives, making it possible to choose between cold restores and warm restores depending on the workload.\n\n``` python\nfrom tensorlake.sandbox import Sandbox, CheckpointType# Cold-boot restore: captures disk state onlysnapshot = sandbox.checkpoint(checkpoint_type=CheckpointType.FILESYSTEM)# Warm restore: captures disk + VM memory + running processessnapshot = sandbox.checkpoint(checkpoint_type=CheckpointType.MEMORY)\n```\n\nThe default when you call sandbox.checkpoint() with no arguments is filesystem. For the fork pattern described in this article, you want memory.\n\nOne tradeoff to understand upfront: memory snapshots lock the resource configuration at capture time.Image, CPU count, memory limit, entrypoint, and secrets all come from the snapshot when you restore.\n\n**You cannot change them at restore time.**\n\nIf you need different resource allocations per worker, use filesystem snapshots and accept the cold boot. For most parallel experiment setups, the locked resources are fine since you sized the base environment for the workers when you created it.\n\n**Secrets are the exception.**\n\nThey survive the checkpoint but are not locked to it. Secrets are passed as environment variables to run(), so you can pass a new value at restore time and overwrite whatever was captured in the snapshot. Rotating an API key across forked workers does not require a new snapshot.\n\nOnce you have a memory snapshot, the fork pattern is straightforward. Here is what it looks like end to end:\n\nEvery worker starts from the same warm execution point. They share nothing at runtime. The base sandbox can be terminated right after the checkpoint. The snapshot_id persists independently.\n\n**Step 1: Create one base sandbox and do all shared setup inside it.**\n\n``` python\nfrom tensorlake.sandbox import Sandbox, CheckpointTypebase = Sandbox.create(    image=\"tensorlake/ubuntu-minimal\",    cpus=2,    memory_mb=4096,)# Install shared dependencies oncebase.run(\"pip\", [\"install\", \"numpy\", \"pandas\", \"--break-system-packages\"])# Verify the environment is readybase.run(\"python3\", [\"-c\", \"import numpy; import pandas; print('Environment ready.')\"])\n```\n\n**Step 2: Capture the warm state.**\n\n```\nsnapshot = base.checkpoint(checkpoint_type=CheckpointType.MEMORY)print(f\"Snapshot captured: {snapshot.snapshot_id}\")\n# The base sandbox can be terminated. The snapshot persists independently.base.terminate()\n```\n\n**Step 3: Restore N workers in parallel from the same checkpoint.**\n\n``` python\nfrom concurrent.futures import ThreadPoolExecutordef run_experiment(script: str) -> str:    # Each worker restores a fresh copy of the warm environment    worker = Sandbox.create(snapshot_id=snapshot.snapshot_id)    result = worker.run(\"python3\", [\"-c\", script])    worker.terminate()    return result.stdoutcandidates = [script_v1, script_v2, script_v3, script_v4, script_v5]with ThreadPoolExecutor(max_workers=len(candidates)) as pool:    results = list(pool.map(run_experiment, candidates))\n```\n\nEach Sandbox.create(snapshot_id=...) call restores an independent copy of the warm environment. Workers share no filesystem and no runtime state. They just started from the same point.\n\nThe snapshot.snapshot_id is a persistent string. Save it to a file. Use it in the next session, in the next iteration of your loop, in a different process entirely. The warm state survives until you explicitly delete the snapshot.\n\n**Andrej Karpathy** published the autoresearch repo in early 2026.\n\n**The core idea:** an LLM reads your current best training script, proposes N code modifications, you race all N in parallel sandboxes, keep the winner, and loop. **The loop runs overnight**. Each accepted modification becomes the new baseline for the next iteration.\n\n**The naive implementation pays the setup cost N times per iteration.**\n\nIf the training script needs NumPy, Scipy, and a small dataset loaded into memory, that is real time per candidate, per loop. With 8 iterations and 3 candidates each, you have paid an identical setup cost 24 times.\n\n**With Tensorlake’s Snapshot Fork Pattern**, you do the setup once.\n\nTake a memory checkpoint. Fork all candidates from that checkpoint.\n\n**They start warm and only diverge where the proposed modification changes behavior.**\n\nThe structure of one iteration looks like this:\n\n```\n# Build the warm baseline once per iterationbase = Sandbox.create(image=\"tensorlake/ubuntu-minimal\", cpus=2, memory_mb=4096)base.run(\"pip\", [\"install\", \"numpy\", \"scipy\", \"--break-system-packages\"])base.run(\"python3\", [\"-c\", \"import dataset; dataset.load_into_memory()\"])\n# Checkpoint once: all candidates share this starting pointsnapshot = base.checkpoint(checkpoint_type=CheckpointType.MEMORY)base.terminate()# Race candidates from the warm snapshot in parallel# Each worker starts where the base left off, not from a fresh imagewith ThreadPoolExecutor(max_workers=len(candidates)) as pool:    results = list(pool.map(lambda s: run_in_sandbox(snapshot.snapshot_id, s), candidates))# Keep the winner, discard the rest, repeatwinner = min(results, key=lambda r: r[\"val_loss\"])\n```\n\nThe snapshot.snapshot_id is reused across all candidates in the same iteration. Setup cost is paid once per loop, not once per candidate.\n\n**RL rollouts have a hard requirement:** same seed, same action sequence, same trajectory. Every time, without exception. That guarantee breaks the moment workers share any state. A shared pip cache can introduce version skew between runs. A shared /tmp carries residual files from previous episodes. Even calling env.reset() correctly does not help when state outside the environment object persists between episodes.\n\nWith per-rollout sandboxes forked from a common snapshot, the isolation is structural rather than enforced by convention. There is no shared filesystem between workers.\n\nThe seed goes directly into the Python script that runs inside the sandbox, not into the host process.\n\n**Keeping it there means the host’s random state stays completely out of the episode:**\n\n``` php\nimport jsonfrom tensorlake.sandbox import Sandboxdef gym_harness(seed: int) -> str:    # Script runs inside the sandbox, not the host    return f\"\"\"import gymnasium as gym, jsonenv = gym.make(\"CartPole-v1\")obs, _ = env.reset(seed={seed})env.action_space.seed({seed})trajectory, total_reward = [], 0.0for _ in range(200):    action = env.action_space.sample()    next_obs, reward, terminated, truncated, _ = env.step(action)    trajectory.append((obs.tolist(), int(action), float(reward), bool(terminated)))    total_reward += reward    obs = next_obs    if terminated or truncated:        breakprint(json.dumps({{\"seed\": {seed}, \"total_reward\": total_reward, \"steps\": len(trajectory)}}))\"\"\"def run_rollout(snapshot_id: str, seed: int) -> dict:    worker = Sandbox.create(snapshot_id=snapshot_id)    result = worker.run(\"python3\", [\"-c\", gym_harness(seed)])    worker.terminate()    return json.loads(result.stdout)\n```\n\nOne specific gotcha here: env.reset(seed=seed) only seeds the observation and transition RNG. The action space has its own separate RNG that requires env.action_space.seed(seed) independently. Miss the second call and trajectories vary across runs with no obvious error. The mismatch is silent and will look like flaky results until you trace it to the gymnasium source.\n\n**Browser warmup is slow:** login flows, OAuth, session cookies, waiting for JavaScript-heavy pages to stabilize. If you are running 20 browser workers in parallel, you do not want each one repeating that from scratch.\n\nTensorlake’s ubuntu-vnc sandbox image makes this practical because the authenticated browser itself becomes part of the checkpoint.\n\nComplete the auth flow once, checkpoint the authenticated browser state, then fork your workers from there:\n\n``` python\nfrom tensorlake.sandbox import Sandbox, CheckpointType# Authenticate once in a single browser sandboxbrowser_base = Sandbox.create(image=\"tensorlake/ubuntu-vnc\", cpus=4, memory_mb=4096)# ... drive Chrome via CDP, complete login, reach stable page state ...# Checkpoint the authenticated browser in memoryauth_snapshot = browser_base.checkpoint(checkpoint_type=CheckpointType.MEMORY)browser_base.terminate()# All 10 workers start with the logged-in browser state already in memoryworkers = [Sandbox.create(snapshot_id=auth_snapshot.snapshot_id) for _ in range(10)]\n```\n\nThe same pattern that eliminates repeated pip install across ML workers eliminates repeated OAuth flows across browser workers. The primitive is the same. Only the warmup content changes.\n\nSuspend and snapshot both preserve sandbox state. They solve different problems and it is worth being clear about which one you reach for.\n\n**Suspend** pauses this specific sandbox and holds its state for later resumption under the same sandbox ID. I**t uses no compute while suspended. **When you resume, the sandbox comes back in under a second with the same process IDs, filesystem, and in-memory state intact. Named sandboxes auto-suspend on timeout instead of terminating, which means you do not lose a long-running agent session because a task ran slightly over its time budget.\n\n**Snapshot** captures a reusable artifact you can restore into new sandboxes. The artifact persists after the source sandbox is terminated. Restore from it once or many times.\n\nSuspend is for pausing and resuming a single workstream. Snapshot is for branching from a known state into N parallel workers.\n\n**The sandbox keeps running while the snapshot is being captured.** The snapshot artifact persists regardless of what happens to the source sandbox afterward.\n\nUnix fork() exists for the same reason. Process creation is expensive, so instead of spawning a child from scratch, you copy the parent's entire memory state and let the child diverge from there.\n\nGit branches for the same reason: you checkpoint a known state and branch from it because re-deriving full history for every branch would be wasteful.\n\nAI experiment infrastructure got filesystem snapshots first. But filesystem snapshots only moved the cost boundary to disk. The running process, loaded modules, in-memory data, and interpreter state all had to rebuild on every new worker anyway.\n\nMemory checkpointing moves that boundary further. The running process is part of what gets captured. Load a 500MB dataset into a pandas DataFrame before the checkpoint, and every forked worker starts with that DataFrame already in RAM. Pre-compile JIT functions and the cache is there too. The interpreter does not restart. Nothing reloads.\n\nThe expensive part of an experiment is rarely the experiment. It is everything that had to be true before the experiment could start.\n\nThat cost is not fixed. It just looked fixed because the tooling treated it that way.\n\n**Three numbers that matter for the fork pattern specifically.**\n\n**Speed.** The tensorlake/ubuntu-minimal image starts up in a few hundred milliseconds; tensorlake/ubuntu-systemd (full init system) takes around one second. Forked workers restore from a memory snapshot warm, skipping boot and process initialization entirely. Suspended named sandboxes resume in under a second without losing any memory or filesystem state. The [published SQLite benchmark](https://github.com/tensorlakeai/sandbox-sqlite-bench) (100k inserts, 2 vCPU / 4GB) shows Tensorlake at 2.45s against Vercel at 3.00s, E2B at 3.92s, Modal at 4.66s, and Daytona at 5.51s.\n\n**Scale.** Tensorlake supports fanning out to thousands concurrent sandbox environments — the number the Harbor integration targets for RL rollouts and eval pipelines running from a single snapshot. The overall project limit is 5 million sandboxes.\n\n**Isolation.** Sandboxes run on MicroVMs backed by Firecracker and CloudHypervisor. LLM-generated code never shares a kernel with other tenants or with your host process. Tensorlake is SOC 2 Type II and HIPAA-compliant, with EU data residency and zero-data-retention options available.\n\nInstall the SDK and grab an API key from [cloud.tensorlake.ai](https://cloud.tensorlake.ai) (free tier available):\n\n```\npip install tensorlakeexport TENSORLAKE_API_KEY=your_api_key\n```\n\nMinimal working fork pattern you can run right now:\n\n``` python\nfrom tensorlake.sandbox import Sandbox, CheckpointTypefrom concurrent.futures import ThreadPoolExecutor# Step 1: Warm the base environmentbase = Sandbox.create(cpus=2, memory_mb=4096)base.run(\"pip\", [\"install\", \"numpy\", \"--break-system-packages\"])# Step 2: Checkpoint the warm statesnapshot = base.checkpoint(checkpoint_type=CheckpointType.MEMORY)base.terminate()# Step 3: Fork workers from the snapshotdef run(script: str) -> str:    w = Sandbox.create(snapshot_id=snapshot.snapshot_id)    out = w.run(\"python3\", [\"-c\", script]).stdout    w.terminate()    return outscripts = [\"print('v1')\", \"print('v2')\", \"print('v3')\"]with ThreadPoolExecutor(max_workers=3) as pool:    results = list(pool.map(run, scripts))print(results)\n```\n\nAll three workers start warm. snapshot.snapshot_id is a persistent string you can store and reuse across sessions.\n\nThose eight pip install numpy calls were not a performance bug.They were a modeling mistake. I had built my experiment loop around \"start fresh\" when what I actually needed was \"start identical.\"\n\n**The difference is real.**\n\nStarting fresh means paying full setup cost per worker. Starting identical means paying it once and forking from there.\n\n**Tensorlake’s memory checkpoints provide one implementation of this pattern: **build the environment once, capture the execution state, and fork as many isolated workers as you need from the same warm baseline.\n\nYou do the expensive shared work once, freeze the execution state, and branch as many workers as you need from that exact point. Each worker gets its own isolated environment. The setup cost is paid once.\n\n**This applies to most parallel AI workflows: **ML candidate racing, RL rollouts, browser agent pools, CI evaluation pipelines. The pattern is the same across all of them.\n\nIf I could go back to those eight workers all stuck on pip install numpy, the fix would have been four lines. Create the base. Warm it.\n\nCall checkpoint(MEMORY). Restore from there. The 320 seconds of identical setup would have happened once. The other seven copies would never have existed.\n\nThat is what the pattern is for.\n\n[Why Your AI Experiments Keep Starting From Scratch (And How Tensorlake Fixes It)](https://pub.towardsai.net/why-your-ai-experiments-keep-starting-from-scratch-and-how-tensorlake-fixes-it-2e6091edf143) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/why-your-ai-experiments-keep-starting-from-scratch-and-how-tensorlake-fixes-it", "canonical_source": "https://pub.towardsai.net/why-your-ai-experiments-keep-starting-from-scratch-and-how-tensorlake-fixes-it-2e6091edf143?source=rss----98111c9905da---4", "published_at": "2026-07-14 02:44:11+00:00", "updated_at": "2026-07-14 02:53:21.021670+00:00", "lang": "en", "topics": ["machine-learning", "ai-infrastructure", "developer-tools"], "entities": ["Tensorlake", "NumPy"], "alternates": {"html": "https://wpnews.pro/news/why-your-ai-experiments-keep-starting-from-scratch-and-how-tensorlake-fixes-it", "markdown": "https://wpnews.pro/news/why-your-ai-experiments-keep-starting-from-scratch-and-how-tensorlake-fixes-it.md", "text": "https://wpnews.pro/news/why-your-ai-experiments-keep-starting-from-scratch-and-how-tensorlake-fixes-it.txt", "jsonld": "https://wpnews.pro/news/why-your-ai-experiments-keep-starting-from-scratch-and-how-tensorlake-fixes-it.jsonld"}}