# Stop Killing Your Agents: How to Move AI Workflows from Local Laptops to Cloud Runtimes > Source: > Published: 2026-11-15 14:03:52+00:00 # Stop Killing Your Agents: How to Move AI Workflows from Local Laptops to Cloud Runtimes Why local agent execution fails in production, what PostHog learned building a serverless agent runtime, and the architecture of cloud agent sandboxes. If you have spent any real time working with coding agents like Claude Code, Codex, or Aider, you have almost certainly killed an agent mid-task by closing your laptop lid on the way to a meeting. Or maybe you opened three terminal tabs, kicked off three parallel tasks across different branches, and listened to your MacBook fans spin up like a jet engine while two language servers, a local Postgres container, and a headless Chromium instance fought over 32 GB of RAM. Running an agent on your own machine feels magical for a ten-minute demo. As soon as you rely on agents across an engineering team, localhost becomes a bottleneck: - **Real tasks take longer than a coffee break:** Reproducing a flaky test, patching the code, and running verification suites often takes 30 to 90 minutes. A single Wi-Fi drop or sleep timer wipes out a local terminal session. - **Local sessions are single-player:** If an agent gets halfway through a refactor on your laptop and needs a teammate’s input in Slack, nobody else can inspect or resume that run. - **Shell access on developer hardware is dangerous:** Developer machines hold SSH keys, cloud CLI sessions, and`.env` files. A prompt injection hidden inside a GitHub issue or error log can trick a local agent into reading`~/.ssh` or hitting internal services. Recently, PostHog engineers shared how they solved this by building the **PostHog Cloud Agent Runtime**. Today, that shared runtime powers agents across four entry points: the PostHog Desktop app, Slack threads, PostHog AI in the browser, and background agents that autonomously triage and fix production bugs. ## Why Naive Worker Containers Die on Every Deploy When teams first move agents off laptops, the instinct is usually to wrap the agent loop in a backend web service, run it in a Kubernetes or ECS container, and keep conversation state in memory. PostHog started there too, and hit a wall immediately. PostHog ships backend updates constantly. In a stateless web app, rolling deploys are painless: load balancers drain short HTTP requests in seconds and swap containers. An agent task, however, can run for an hour. Every time PostHog deployed a backend update, the deploy terminated active worker containers and killed in-flight runs. That failure led to the core design rule behind their architecture: **nothing that matters can live exclusively inside the execution sandbox.** To make cloud agents reliable, you have to split the system into two halves: 1. **A durable control plane** that owns the task lifecycle, tracks conversation turns, and survives server restarts. 2. **An ephemeral execution sandbox** that holds a Linux kernel, a cloned repo, and local tools, but is treated as disposable at any second. If a sandbox crashes, runs out of memory, or gets preempted, the control plane provisions a fresh sandbox, mounts the last saved filesystem state, replays the event log, and picks up where the agent left off. ## Coordinating Long-Running Turns With Temporal To stop deployments from killing active runs, PostHog moved orchestration to **Temporal**. Each agent session runs as a durable workflow: if a worker restarts during a deploy, Temporal pauses the run and resumes it on a fresh worker without losing state. Temporal also solved two headaches that surface whenever humans interact with agents asynchronously in Slack: - **Follow-up messages mid-turn:** Engineers rarely wait 15 minutes for an agent to finish before adding context like*“Oh, also check the billing webhook.”* Temporal queues incoming messages in order and feeds them into the agent at clean turn boundaries, while routing cancelation signals directly to the active sandbox. - **Webhook retries:** Slack retries webhooks if an endpoint takes more than a few seconds to respond. By keying workflows on message IDs, Temporal deduplicates triggers automatically so one Slack message never spawns three competing agents. ## Why Containers Were Not Enough: Moving from gVisor to MicroVMs Once orchestration is durable, the next question is where the agent runs bash commands and tests. PostHog initially used Google’s **gVisor** container runtime to isolate untrusted code, only to hit a hard blocker: **gVisor cannot run Docker.** In a real production codebase like PostHog’s, integration tests require Postgres, ClickHouse, Redis, and Kafka spun up via Docker Compose. An agent that cannot start a Docker daemon cannot run tests to verify its own work. To give agents a complete development environment, PostHog migrated to **Modal MicroVMs**, giving each sandbox a dedicated Linux kernel and a working Docker daemon. To avoid waiting several minutes for a fresh VM to clone the monorepo and install packages, PostHog relies on two optimizations: - **Filesystem snapshots:** A background job periodically clones the repo, installs dependencies, and saves a base snapshot. Agents boot from that snapshot in seconds, and pause states snapshot the modified working tree so follow-up turns resume instantly. - **Speculative pre-warming:** In the desktop and web apps, the system warms a sandbox as soon as a user focuses the prompt input and starts typing, so the VM is ready before they hit Enter. Inside every VM image, a baked-in **Agent Server** handles authenticated commands from Temporal, streams logs, and refreshes credentials. ## When an Error Log Tries to Hack Your Agent PostHog’s autonomous bug-fixing pipeline watches production error logs, session replays, and support tickets, then spins up an agent to open a pull request. Because anyone on the internet can trigger a frontend error whose stack trace contains a prompt injection payload, those error logs must be treated as untrusted input. PostHog initially added an “LLM as a judge” classifier to filter incoming error signals. In a single week, it blocked **1,374 malicious or suspicious signals**. Yet an LLM filter that catches 98% of attacks still lets 2% through at scale. Assuming an agent will eventually be tricked into executing a malicious command, PostHog enforced deterministic kernel-level firewall rules on every sandbox: - **Strict domain allowlists:** Outbound traffic is capped at 100 pre-approved domains (such as`github.com` ,`pypi.org` , and`registry.npmjs.org` ). - **No raw IP connections:** Agents cannot open sockets to arbitrary IP addresses. - **Blocked metadata routes:** Routing rules block`169.254.169.254` so an agent can never query cloud metadata to steal IAM credentials. ## What Quietly Breaks Two Hours Into a Run Long-running agent sessions also expose operational failure modes that never appear in short benchmarks: - **Orphaned background processes:** Failed Playwright browser tests or dev servers often leave headless Chromium and Python processes running in the background until the VM runs out of memory. Every command needs process-group tracking (`pgid` ) and strict timeouts so the Agent Server can reap the entire process tree (`SIGTERM` then`SIGKILL` ) after each step. - **Mid-run token expiration:** GitHub App installation tokens expire after 60 minutes. If a refactor takes 70 minutes, static tokens injected at boot will fail on the final`git push` . The internal Agent Server refreshes OAuth and GitHub tokens out-of-band right before git operations run. - **Checkpoint bloat:** Tarring`/workspace` (including`node_modules` and Docker layers) after every turn creates multi-gigabyte uploads. Combining copy-on-write VM snapshots with lightweight`git diff` patches in Postgres keeps checkpoints fast. ## Building a Minimal Cloud Sandbox Runner in Python You do not need a large infrastructure team to adopt this architecture. You can build the execution half of this pattern in a few dozen lines of Python using Modal’s sandbox primitives, pre-baking dependencies, enforcing process-group timeouts, and locking down outbound network traffic: ``` python import subprocess from typing import Any, Dict import modal app = modal.App("cloud-agent-runtime") sandbox_image = ( modal.Image.debian_slim() .apt_install("git", "curl", "ripgrep") .pip_install("pytest", "pydantic") ) allowlist = modal.NetworkAllowlist( domains=["github.com", "api.github.com", "pypi.org", "files.pythonhosted.org"] ) @app.function( image=sandbox_image, timeout=3600, network_allowlist=allowlist, volumes={"/workspace": modal.Volume.from_name("agent-repo-snapshots", create_if_missing=True)}, ) def execute_agent_step(command: str, step_timeout: int = 120) -> Dict[str, Any]: try: proc = subprocess.run( command, shell=True, cwd="/workspace", capture_output=True, text=True, timeout=step_timeout, start_new_session=True, ) return { "exit_code": proc.returncode, "stdout": proc.stdout[-4000:], "stderr": proc.stderr[-4000:], } except subprocess.TimeoutExpired: return {"exit_code": -1, "error": f"Command exceeded {step_timeout}s timeout"} ``` Pair this execution worker with a durable queue or workflow engine (like Temporal, Inngest, or Celery backed by Postgres), and you have the core foundation of a cloud agent runtime: isolated execution, bounded blast radius, and sessions that survive when you close your laptop and head home. ## Where Agent Infrastructure Is Heading For the past year, most of the industry’s attention has gone into picking the best model or tweaking system prompts. PostHog’s experience shows why the real differentiator in production is shifting toward the **runtime harness**. As long as agents live inside a developer’s local terminal, they remain single-player power tools: useful when you are babysitting them, fragile when you look away. Moving agents into durable, network-isolated cloud sandboxes turns them into shared engineering infrastructure. A developer can kick off an investigation from a Slack thread on their phone, let a teammate chime in twenty minutes later, and wake up to a tested pull request without a single laptop fan spinning up. ### Further Reading & Resources - **[PostHog Engineering: We built our own cloud agents runtime. Here’s what we learned](https://posthog.com/blog/cloud-agents-runtime)** : The original teardown by Cleo Lant and Alessandro Pogliaghi covering PostHog’s migration from local workers to Temporal and Modal. - **[Temporal Documentation](https://docs.temporal.io/)** : Open-source durable execution platform for orchestrating long-running stateful workflows. - **[Modal Sandboxes Guide](https://modal.com/docs/guide/sandbox)** : Documentation on running untrusted code inside serverless MicroVMs with filesystem snapshots and network allowlists. - **[PostHog Agent Infrastructure Team](https://posthog.com/teams/agent-infrastructure)** : How PostHog structures their internal agent platform and roadmap.