I Built a JIT Compiler for AI Agents: How We Turned 30s LLM Chains into 0.1ms Deterministic Python

In 2026, AI agents have become the default paradigm for automating complex workflows: DevOps orchestration, customer support, database triage, and e-commerce transactions.

Yet, every engineering team deploying autonomous agents in production eventually hits the same brick wall:

think -> tool -> observe -> think) easily burns user_id, order_id, or date). Why are we invoking massive 70-billion-parameter neural networks over HTTP dozens of times just to parse an ID and pass it into a database query?

In computer science, this problem was solved decades ago:

torch.compile): What if we did the exact same thing for AI Agent Trajectories?

Today, I’m open-sourcing AgentJIT — a Just-In-Time trajectory compiler for AI agents that traces dynamic tool chains and compiles them into sub-millisecond, deterministic Python AST pipelines with zero runtime token cost.

       [Dynamic Agent Task]
                │
         (1st run / warmup)
                ▼
     ┌──────────────────────┐
     │   AgentJIT Tracer    │ ── Captures tool calls, data flow & variables
     └──────────────────────┘
                │
                ▼
     ┌──────────────────────┐
     │  DAG Flow Analyzer   │ ── Resolves dependencies & arithmetic expressions
     └──────────────────────┘
                │
                ▼
     ┌──────────────────────┐
     │  AST Code Generator  │ ── Synthesizes pure Python AST + Runtime Guards
     └──────────────────────┘
                │
                ▼
  ┌────────────────────────────┐
  │   Compiled JIT Pipeline    │ ──► Subsequent runs: < 0.1ms, $0 tokens!
  └────────────────────────────┘
                │
       (Guard failure? Deopt!)
                ▼
     [Fall back to LLM Agent]

AgentJIT operates in three distinct phases:

When an agent decorated with @jit runs for the first time, the Tracer records every tool invocation, its inputs, outputs, and execution timings. It builds a directed acyclic graph (DAG) of the data flow, distinguishing between static parameters and dynamic runtime inputs.

The compiler examines the trace and synthesizes a pure Python Abstract Syntax Tree (AST). It converts dynamic tool dispatching into hard-wired, type-checked Python function calls, resolving nested dictionaries and mathematical operators.

What happens if the user inputs an anomalous value or unexpected format?

AgentJIT automatically inserts Runtime Input Guards. If any input violates the expected structure, the compiled pipeline immediately triggers a speculative bailout (de-optimization), gracefully falling back to the original LLM agent.

Zero crashes, zero regressions, pure speedup.

AgentJIT is 100% self-contained in a single library with zero mandatory external dependencies.

pip install agentjit

(or uv add agentjit)

Simply decorate your agent with @jit and mark your tools with @trace_tool:

from agentjit import jit, trace_tool

@trace_tool()
def fetch_product(product_id: str):
    return {"id": product_id, "price": 89.0, "category": "electronics"}

@trace_tool()
def calculate_vat(price: float, tax_rate: float):
    return round(price * (1.0 + tax_rate), 2)

@trace_tool()
def generate_invoice(product_id: str, total_price: float):
    return {"invoice_id": f"INV-{product_id}", "total": total_price}

@jit
def checkout_agent(product_id: str, tax_rate: float):
    product = fetch_product(product_id=product_id)
    total = calculate_vat(price=product["price"], tax_rate=tax_rate)
    return generate_invoice(product_id=product["id"], total_price=total)

order1 = checkout_agent("SKU-100", 0.20)

order2 = checkout_agent("SKU-200", 0.20)

You can even inspect the generated Python code at runtime:

print(checkout_agent.source_code)

We ran a 100-iteration benchmark in Google Colab simulating an uncompiled multi-step LLM chain versus the compiled AgentJIT pipeline:

AgentJIT was architected with Python 3.13 and Python 3.14 in mind:

threading.Lock. 3.13t and 3.14t). You can spin up hundreds of concurrent agent threads without GIL contention. You don't need to configure an environment or install dependencies locally. You can run the interactive demo and benchmark right now in your browser:

👉 Launch Interactive Google Colab Demo

If you're building autonomous agents in production and want to slash your latency and API costs, give AgentJIT a spin!

If you find the project interesting, please consider dropping a Star ⭐ on GitHub — it helps other developers discover the library!