# Show HN: AgentJIT – Compile dynamic LLM agent workflows into 0.1ms Python

> Source: <https://github.com/eminsk/agentjit>
> Published: 2026-09-13 12:39:35+00:00

**Compile flaky, 30-second multi-step AI Agent workflows into 5-millisecond deterministic code.**

[**Quickstart**](#quickstart) • [** Why AgentJIT?**](#the-problem-in-2026-why-agentjit) • [** Architecture**](#architecture) • [** Benchmarks**](#benchmarks) • **Speculative Execution**

In 2026, autonomous AI agents solve real-world workflows across business, DevOps, and data analysis. However, running stochastic LLM loops in production faces four critical barriers:

1. **Massive Latency:** A standard 4-step agent workflow (`think -> tool -> observe -> think` ) takes**15 to 45 seconds** .
2. **Exponential Costs:** Running the loop 10,000 times/day costs thousands of dollars in redundant API tokens.
3. **Flakiness & Hallucinations:** Even 98% reliability per step leads to compounding errors across multi-turn trajectories.
4. **Redundant Reasoning:** Most agent invocations execute the*exact same structural trajectory* with slightly different input parameters (e.g. different user IDs or dates).

Just like **V8** compiles hot JavaScript into machine code, and **PyTorch `torch.compile`** traces dynamic tensors into optimized CUDA kernels, **AgentJIT traces dynamic agent trajectories and compiles them into pure, type-safe, ultra-fast Python code.**

```
       [Dynamic Agent Task]
                │
         (1st run / warmup)
                ▼
     ┌──────────────────────┐
     │   AgentJIT Tracer    │ ── (Captures tool calls, data flow, variables)
     └──────────────────────┘
                │
                ▼
     ┌──────────────────────┐
     │ DAG Flow Analyzer    │ ── (Parameter generalization, dependency graph)
     └──────────────────────┘
                │
                ▼
     ┌──────────────────────┐
     │ AST Code Generator   │ ── (Synthesizes pure Python pipeline + Guards)
     └──────────────────────┘
                │
                ▼
  ┌────────────────────────────┐
  │   Compiled JIT Pipeline    │ ──► Subsequent runs: <1ms, $0 tokens!
  └────────────────────────────┘
                │
       (Guard failure? Deopt!)
                ▼
     [Fall back to LLM Agent]
```

- 🏎️ **Up to 100,000x Speedup:** Hot paths drop from ~20,000 ms to**< 0.1 ms** .
- 💸 **100% Token Savings:** Once compiled, recurring workflows run completely locally with**0 LLM tokens consumed** .
- 🛡️ **Speculative De-Optimization (Bailout):** Automatically generates runtime input guards. If unexpected data formats or divergent branches appear, AgentJIT transparently falls back to the dynamic LLM agent.
- 🔍 **Transparent & Inspectable:** Inspect the exact Python code generated by the JIT with`agent.source_code` .
- 🧵 **Free-Threaded / No-GIL (PEP 703) Ready:** Thread-safe runtime fully tested on Python 3.13t and 3.14t for true multi-core parallel agent execution without GIL contention.
- 🔌 **Framework Agnostic:** Seamlessly wraps LangChain, CrewAI, AutoGen, OpenAI Tool calls, or native Python functions.

```
pip install agentjit
# or with uv
uv add agentjit
```

Decorate your agent with `@jit` and mark your tools with `@trace_tool`:

``` python
from agentjit import jit, trace_tool

# 1. Define your tools
@trace_tool()
def search_product(name: str):
    return {"name": name, "price": 49.99, "stock": 120}

@trace_tool()
def apply_tax(price: float, tax_rate: float):
    return round(price * (1.0 + tax_rate), 2)

# 2. Decorate your agent with @jit
@jit
def checkout_agent(product_name: str, tax_rate: float):
    # This dynamic workflow could call an LLM (Claude, GPT, Gemini)
    item = search_product(name=product_name)
    total = apply_tax(price=item["price"], tax_rate=tax_rate)
    return {"item": item["name"], "total": total}

# --- Run 1: Warmup & Tracing (runs dynamic agent, compiles to Python) ---
order1 = checkout_agent("Mechanical Keyboard", 0.19)

# --- Run 2+: Instant compiled execution (ZERO tokens, sub-millisecond!) ---
order2 = checkout_agent("Wireless Mouse", 0.19)  # Takes 0.05 ms!
```

You can view the exact synthesized Python code generated by the JIT at any time:

```
print(checkout_agent.source_code)
```

**Synthesized Output:**

``` python
def compiled_checkout_agent(product_name, tax_rate):
    """JIT-compiled trajectory pipeline generated by AgentJIT.
    Executes deterministically in sub-millisecond time with zero token cost.
    """
    # --- Speculative Guards ---
    if not (product_name is not None):
        raise GuardViolation("Argument 'product_name' must not be None", param="product_name")
    if not (isinstance(product_name, str)):
        raise GuardViolation("Argument 'product_name' must be of type str", param="product_name")

    # --- Execution Steps ---
    step_1_out = _tools['search_product'](name=product_name)
    step_2_out = _tools['apply_tax'](price=step_1_out['price'], tax_rate=tax_rate)

    # --- Return Final Result ---
    return {'item': step_1_out['name'], 'total': step_2_out}
```

Benchmark comparing a simulated 3-step reasoning agent (15s latency, 2,500 tokens) vs AgentJIT compiled execution over 100 runs:

| Execution Mode | Mean Latency | 99th Percentile | Cost per 1k runs | Token Usage | Determinism | 
|---|---|---|---|---|---|
| **Standard LLM Agent** | `14,820 ms` | `22,400 ms` | **$75.00** | 2,500,000 | ~94% | 
| **AgentJIT (Warm Path)** | **`0.08 ms`** | **` 0.12 ms`** | **$0.00** | **0** | **100%** | 
| **Improvement** | **185,000x faster** | **186,000x faster** | **100% savings** | **Zero tokens** | **Rock-solid** | 

What happens when an input is unusual or triggers an unexpected branch?

AgentJIT uses **Speculative De-Optimization**:

1. Input variables are validated against synthesized guards.
2. If any guard fails (e.g. wrong type, missing required key) or a tool raises an unhandled exception, AgentJIT catches `GuardViolation` .
3. It seamlessly bails out to the dynamic LLM agent to handle the edge case.
4. Telemetry records the bailout for future multi-branch specialization.

```
# Normal input: runs compiled pipeline in 0.08ms
checkout_agent("Monitor", 0.19)

# Divergent input (e.g. invalid type): automatically bails out to dynamic agent
checkout_agent(12345, None)  # Transparently de-optimizes, no crash!
```

Monitor your compiled agents in real time:

```
print(checkout_agent.stats)
# Output:
# {
#     "total_calls": 1500,
#     "compiled_hits": 1492,
#     "bailouts": 8,
#     "compiled_hit_rate": 99.47,
#     "total_time_saved_ms": 22380000.0,
#     "total_tokens_saved": 3730000
# }
```

-  **Core Tracer & DAG Flow Analyzer**
-  **AST Code Generation with Speculative Guards**
-  **De-optimization / Bailout Runtime**
-  **`@jit` Decorator with Auto-Warmup**
-  **Multi-Branch Polyhedral JIT:** Merge multiple execution paths into a unified control-flow graph (`if/else` branching synthesis).
-  **eBPF-Isolated Micro-Sandbox:** Ultra-fast sub-millisecond process sandbox for compiled shell actions.
-  **WebAssembly (Wasm) Export:** Compile agent trajectories into standalone Wasm binaries for browser and edge runtime.

AgentJIT is open-source software licensed under the [Apache 2.0 License](/eminsk/agentjit/blob/main/LICENSE).