Fly.io AI Agents: Moving from LLMs to Virtual Machines

Fly.io advocates moving AI agents from LLM function-calling to dedicated virtual machines, arguing that a full POSIX environment reduces brittleness and error-handling complexity. The approach replaces tool definitions with direct shell commands, provides state persistence and dependency management, and uses ephemeral VMs to balance cost and performance. Fly.io recommends a spin-up-execute-destroy lifecycle for production deployments.

Fly.io AI Agents: Moving from LLMs to Virtual Machines The "Sandboxed OS" Architecture Standard prompt engineering for agents usually involves giving the LLM a set of tools functions it can call. The problem is that these tools are brittle. If a tool fails or returns an unexpected format, the agent loops or crashes. By deploying agents into their own dedicated VMs, the agent gains a full POSIX-compliant environment. Instead of calling a read file function, the agent simply executes cat filename.txt in a bash shell. This shifts the burden of reliability from the tool-definition layer to the operating system layer. To implement this locally for testing before deploying to a cloud provider, you can simulate this "agent-in-a-box" using Docker. Here is a basic configuration for a Python-based agent environment that restricts the agent's capabilities while providing a full shell: docker-compose.yml for Agent Sandbox services: agent-env: image: python:3.11-slim volumes: - ./agent workspace:/home/agent/workspace environment: - OPENAI API KEY=${OPENAI API KEY} working dir: /home/agent/workspace deploy: resources: limits: cpus: '0.5' memory: 512M command: /bin/bash Why VMs Beat Function Calling When you move from tool-calling to a VM-based AI workflow, you change the error-handling paradigm. In a traditional LLM agent setup, a syntax error in a generated script usually requires a complex feedback loop to the model. In a VM environment, the agent receives the actual stderr from the shell. State Persistence: A VM maintains a filesystem. The agent can install a library via pip , run a script, check the output, and then modify the script based on the result without the developer having to explicitly manage a "state" object in the prompt. Dependency Management: If an agent needs a specific version of a tool e.g., ffmpeg for video processing , it can attempt to install it via apt-get rather than the developer needing to pre-build every possible tool into the agent's API. Security Isolation: Running an agent in a VM provides a hard boundary. If the LLM generates a destructive command like rm -rf / , it only kills the ephemeral instance, not the host server. Real-World Deployment Logic For those building an LLM agent deployment, the key is managing the lifecycle of these VMs. You cannot keep a VM running 24/7 for every single user query due to cost. The optimal pattern is "Spin up - Execute - Snapshot/Destroy." Here is a conceptual Python snippet showing how to manage an ephemeral agent session via a CLI interface: python import subprocess class AgentSession: def init self, session id : self.session id = session id self.vm id = None def start vm self : Example command to spin up a lightweight Firecracker VM or Docker container cmd = f"docker run -d --name agent {self.session id} agent-base-image" self.vm id = subprocess.check output cmd, shell=True .decode .strip def execute command self, command : Executing a command directly in the agent's "body" exec cmd = f"docker exec agent {self.session id} /bin/bash -c '{command}'" result = subprocess.run exec cmd, shell=True, capture output=True, text=True return result.stdout if result.returncode == 0 else result.stderr def terminate self : subprocess.run f"docker rm -f agent {self.session id}", shell=True Usage session = AgentSession "user 123" session.start vm print session.execute command "ls -la" session.terminate The Performance Trade-off The shift to VM-based agents introduces latency. Cold-starting a VM takes longer than a simple API request. However, for complex tasks—like autonomous coding or data analysis—the time spent booting a VM is negligible compared to the time spent in a "hallucination loop" where a tool-based agent fails repeatedly because it lacks the proper environment context. This approach turns the agent from a "chatbot with plugins" into a "remote operator." By providing a real filesystem and a real shell, we stop trying to simulate a computer inside a prompt and simply give the AI an actual computer. Claude Code Workflow: Open Weights vs. Closed Models 1h ago /en/news/2807/ Claude Code: My Experience with Local Credential Scanning 2h ago /en/news/2787/ Claude Code Workflow: Leveraging Open Weights for Local Dev 2h ago /en/news/2773/ Oracle AI Pivot: 21,000 Layoffs to Fund Infrastructure 3h ago /en/news/2749/ DeepSQL: Self-Hostable DBA Agent for Postgres & MySQL 4h ago /en/news/2739/ AI Chips: The Great Hardware Sprint 5h ago /en/news/2719/ Next Claude Code Workflow: Open Weights vs. Closed Models → /en/news/2807/ All Replies （3） @CameronWizard /en/users/CameronWizard/ Probably. New leadership usually means a hard reset on the roadmap, especially when the tech landscape flips this fast.