I Built a JIT Compiler for AI Agents: How We Turned 30s LLM Chains into 0.1ms Deterministic Python

A developer has open-sourced AgentJIT, a just-in-time trajectory compiler that traces an AI agent's dynamic tool chains and compiles them into deterministic Python AST pipelines, cutting execution time from roughly 30 seconds to under 0.1 milliseconds with zero runtime token cost. The library records tool invocations on a first warmup run, builds a DAG of the data flow, and synthesizes type-checked Python calls, inserting runtime input guards that trigger a speculative bailout back to the original LLM agent when inputs deviate from the traced structure. AgentJIT is distributed as a self-contained Python package with no mandatory external dependencies and is used via @jit and @trace_tool decorators.

In 2026, AI agents have become the default paradigm for automating complex workflows: DevOps orchestration, customer support, database triage, and e-commerce transactions. Yet, every engineering team deploying autonomous agents in production eventually hits the same brick wall: think - tool - observe - think easily burns user id , order id , or date . Why are we invoking massive 70-billion-parameter neural networks over HTTP dozens of times just to parse an ID and pass it into a database query? In computer science, this problem was solved decades ago: torch.compile : What if we did the exact same thing for AI Agent Trajectories ? Today, I’m open-sourcing AgentJIT https://github.com/eminsk/agentjit — a Just-In-Time trajectory compiler for AI agents that traces dynamic tool chains and compiles them into sub-millisecond, deterministic Python AST pipelines with zero runtime token cost . Dynamic Agent Task │ 1st run / warmup ▼ ┌──────────────────────┐ │ AgentJIT Tracer │ ── Captures tool calls, data flow & variables └──────────────────────┘ │ ▼ ┌──────────────────────┐ │ DAG Flow Analyzer │ ── Resolves dependencies & arithmetic expressions └──────────────────────┘ │ ▼ ┌──────────────────────┐ │ AST Code Generator │ ── Synthesizes pure Python AST + Runtime Guards └──────────────────────┘ │ ▼ ┌────────────────────────────┐ │ Compiled JIT Pipeline │ ──► Subsequent runs: < 0.1ms, $0 tokens └────────────────────────────┘ │ Guard failure? Deopt ▼ Fall back to LLM Agent AgentJIT operates in three distinct phases: When an agent decorated with @jit runs for the first time, the Tracer records every tool invocation, its inputs, outputs, and execution timings. It builds a directed acyclic graph DAG of the data flow, distinguishing between static parameters and dynamic runtime inputs. The compiler examines the trace and synthesizes a pure Python Abstract Syntax Tree AST . It converts dynamic tool dispatching into hard-wired, type-checked Python function calls, resolving nested dictionaries and mathematical operators. What happens if the user inputs an anomalous value or unexpected format? AgentJIT automatically inserts Runtime Input Guards . If any input violates the expected structure, the compiled pipeline immediately triggers a speculative bailout de-optimization , gracefully falling back to the original LLM agent. Zero crashes, zero regressions, pure speedup. AgentJIT is 100% self-contained in a single library with zero mandatory external dependencies . pip install agentjit or uv add agentjit Simply decorate your agent with @jit and mark your tools with @trace tool : python from agentjit import jit, trace tool 1. Define your tools @trace tool def fetch product product id: str : return {"id": product id, "price": 89.0, "category": "electronics"} @trace tool def calculate vat price: float, tax rate: float : return round price 1.0 + tax rate , 2 @trace tool def generate invoice product id: str, total price: float : return {"invoice id": f"INV-{product id}", "total": total price} 2. Decorate your multi-step agent @jit def checkout agent product id: str, tax rate: float : This dynamic workflow could call an LLM Claude, GPT, Gemini product = fetch product product id=product id total = calculate vat price=product "price" , tax rate=tax rate return generate invoice product id=product "id" , total price=total Run 1: Warmup & Tracing captures trajectory, compiles AST order1 = checkout agent "SKU-100", 0.20 Run 2+: Instant JIT execution < 0.1ms, ZERO tokens consumed order2 = checkout agent "SKU-200", 0.20 You can even inspect the generated Python code at runtime: print checkout agent.source code We ran a 100-iteration benchmark in Google Colab simulating an uncompiled multi-step LLM chain versus the compiled AgentJIT pipeline: | Metric | Uncompiled Agent | AgentJIT Pipeline | Advantage | |---|---|---|---| | Mean Latency | 37.21 ms | 0.1044 ms | 356.4x Faster ⚡ | | Token Cost 1k runs | $7.50 2.5M tokens | $0.00 0 tokens | 100% Free 💰 | | Determinism | ~94% LLM hallucinations | 100.0% Verified AST | Rock-Solid 🛡️ | | Fallback Safety | N/A | Automatic Speculative Deopt | Zero Crashes ✅ | AgentJIT was architected with Python 3.13 and Python 3.14 in mind: threading.Lock . 3.13t and 3.14t . You can spin up hundreds of concurrent agent threads without GIL contention. You don't need to configure an environment or install dependencies locally. You can run the interactive demo and benchmark right now in your browser: 👉 Launch Interactive Google Colab Demo https://colab.research.google.com/github/eminsk/agentjit/blob/main/notebooks/AgentJIT Interactive Demo.ipynb If you're building autonomous agents in production and want to slash your latency and API costs, give AgentJIT a spin If you find the project interesting, please consider dropping a Star ⭐ on GitHub https://github.com/eminsk/agentjit — it helps other developers discover the library