Fly.io AI Agents: Moving from LLMs to Virtual Machines

The "Sandboxed OS" Architecture #

Standard prompt engineering for agents usually involves giving the LLM a set of tools (functions) it can call. The problem is that these tools are brittle. If a tool fails or returns an unexpected format, the agent loops or crashes. By deploying agents into their own dedicated VMs, the agent gains a full POSIX-compliant environment.

Instead of calling a read_file()

function, the agent simply executes cat filename.txt

in a bash shell. This shifts the burden of reliability from the tool-definition layer to the operating system layer.

To implement this locally for testing before deploying to a cloud provider, you can simulate this "agent-in-a-box" using Docker. Here is a basic configuration for a Python-based agent environment that restricts the agent's capabilities while providing a full shell:

services:
  agent-env:
    image: python:3.11-slim
    volumes:
      - ./agent_workspace:/home/agent/workspace
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
    working_dir: /home/agent/workspace
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
    command: /bin/bash

Why VMs Beat Function Calling #

When you move from tool-calling to a VM-based AI workflow, you change the error-handling paradigm. In a traditional LLM agent setup, a syntax error in a generated script usually requires a complex feedback loop to the model. In a VM environment, the agent receives the actual stderr from the shell.

State Persistence: A VM maintains a filesystem. The agent can install a library viapip

, run a script, check the output, and then modify the script based on the result without the developer having to explicitly manage a "state" object in the prompt.Dependency Management: If an agent needs a specific version of a tool (e.g.,ffmpeg

for video processing), it can attempt to install it viaapt-get

rather than the developer needing to pre-build every possible tool into the agent's API.Security Isolation: Running an agent in a VM provides a hard boundary. If the LLM generates a destructive command likerm -rf /

, it only kills the ephemeral instance, not the host server.

Real-World Deployment Logic #

For those building an LLM agent deployment, the key is managing the lifecycle of these VMs. You cannot keep a VM running 24/7 for every single user query due to cost. The optimal pattern is "Spin up -> Execute -> Snapshot/Destroy."

Here is a conceptual Python snippet showing how to manage an ephemeral agent session via a CLI interface:

import subprocess

class AgentSession:
    def __init__(self, session_id):
        self.session_id = session_id
        self.vm_id = None

    def start_vm(self):
        cmd = f"docker run -d --name agent_{self.session_id} agent-base-image"
        self.vm_id = subprocess.check_output(cmd, shell=True).decode().strip()

    def execute_command(self, command):
        exec_cmd = f"docker exec agent_{self.session_id} /bin/bash -c '{command}'"
        result = subprocess.run(exec_cmd, shell=True, capture_output=True, text=True)
        return result.stdout if result.returncode == 0 else result.stderr

    def terminate(self):
        subprocess.run(f"docker rm -f agent_{self.session_id}", shell=True)

session = AgentSession("user_123")
session.start_vm()
print(session.execute_command("ls -la"))
session.terminate()

The Performance Trade-off #

The shift to VM-based agents introduces latency. Cold-starting a VM takes longer than a simple API request. However, for complex tasks—like autonomous coding or data analysis—the time spent booting a VM is negligible compared to the time spent in a "hallucination loop" where a tool-based agent fails repeatedly because it lacks the proper environment context.

This approach turns the agent from a "chatbot with plugins" into a "remote operator." By providing a real filesystem and a real shell, we stop trying to simulate a computer inside a prompt and simply give the AI an actual computer.

Claude Code Workflow: Open Weights vs. Closed Models 1h ago

Claude Code: My Experience with Local Credential Scanning 2h ago

Claude Code Workflow: Leveraging Open Weights for Local Dev 2h ago

Oracle AI Pivot: 21,000 Layoffs to Fund Infrastructure 3h ago

DeepSQL: Self-Hostable DBA Agent for Postgres & MySQL 4h ago

AI Chips: The Great Hardware Sprint 5h ago

Next Claude Code Workflow: Open Weights vs. Closed Models →

All Replies （3） #

@CameronWizardProbably. New leadership usually means a hard reset on the roadmap, especially when the tech landscape flips this fast.