{"slug": "show-hn-agentjit-compile-dynamic-llm-agent-workflows-into-0-1ms-python", "title": "Show HN: AgentJIT – Compile dynamic LLM agent workflows into 0.1ms Python", "summary": "AgentJIT, a new open-source Python library, compiles dynamic LLM agent workflows into deterministic Python code that runs in under 0.1 milliseconds with zero LLM tokens consumed on repeat executions, according to its Show HN launch. The tool traces an agent's first run, analyzes the resulting DAG, and generates type-safe Python with runtime guards that fall back to the dynamic LLM agent on unexpected inputs, and it is compatible with LangChain, CrewAI, AutoGen, and OpenAI tool calls. AgentJIT reports hot-path speedups of up to 100,000x over the 15-to-45-second latency of a standard 4-step agent workflow.", "body_md": "**Compile flaky, 30-second multi-step AI Agent workflows into 5-millisecond deterministic code.**\n\n[**Quickstart**](#quickstart) • [** Why AgentJIT?**](#the-problem-in-2026-why-agentjit) • [** Architecture**](#architecture) • [** Benchmarks**](#benchmarks) • **Speculative Execution**\n\nIn 2026, autonomous AI agents solve real-world workflows across business, DevOps, and data analysis. However, running stochastic LLM loops in production faces four critical barriers:\n\n1. **Massive Latency:** A standard 4-step agent workflow (`think -> tool -> observe -> think` ) takes**15 to 45 seconds** .\n2. **Exponential Costs:** Running the loop 10,000 times/day costs thousands of dollars in redundant API tokens.\n3. **Flakiness & Hallucinations:** Even 98% reliability per step leads to compounding errors across multi-turn trajectories.\n4. **Redundant Reasoning:** Most agent invocations execute the*exact same structural trajectory* with slightly different input parameters (e.g. different user IDs or dates).\n\nJust like **V8** compiles hot JavaScript into machine code, and **PyTorch `torch.compile`** traces dynamic tensors into optimized CUDA kernels, **AgentJIT traces dynamic agent trajectories and compiles them into pure, type-safe, ultra-fast Python code.**\n\n```\n       [Dynamic Agent Task]\n                │\n         (1st run / warmup)\n                ▼\n     ┌──────────────────────┐\n     │   AgentJIT Tracer    │ ── (Captures tool calls, data flow, variables)\n     └──────────────────────┘\n                │\n                ▼\n     ┌──────────────────────┐\n     │ DAG Flow Analyzer    │ ── (Parameter generalization, dependency graph)\n     └──────────────────────┘\n                │\n                ▼\n     ┌──────────────────────┐\n     │ AST Code Generator   │ ── (Synthesizes pure Python pipeline + Guards)\n     └──────────────────────┘\n                │\n                ▼\n  ┌────────────────────────────┐\n  │   Compiled JIT Pipeline    │ ──► Subsequent runs: <1ms, $0 tokens!\n  └────────────────────────────┘\n                │\n       (Guard failure? Deopt!)\n                ▼\n     [Fall back to LLM Agent]\n```\n\n- 🏎️ **Up to 100,000x Speedup:** Hot paths drop from ~20,000 ms to**< 0.1 ms** .\n- 💸 **100% Token Savings:** Once compiled, recurring workflows run completely locally with**0 LLM tokens consumed** .\n- 🛡️ **Speculative De-Optimization (Bailout):** Automatically generates runtime input guards. If unexpected data formats or divergent branches appear, AgentJIT transparently falls back to the dynamic LLM agent.\n- 🔍 **Transparent & Inspectable:** Inspect the exact Python code generated by the JIT with`agent.source_code` .\n- 🧵 **Free-Threaded / No-GIL (PEP 703) Ready:** Thread-safe runtime fully tested on Python 3.13t and 3.14t for true multi-core parallel agent execution without GIL contention.\n- 🔌 **Framework Agnostic:** Seamlessly wraps LangChain, CrewAI, AutoGen, OpenAI Tool calls, or native Python functions.\n\n```\npip install agentjit\n# or with uv\nuv add agentjit\n```\n\nDecorate your agent with `@jit` and mark your tools with `@trace_tool`:\n\n``` python\nfrom agentjit import jit, trace_tool\n\n# 1. Define your tools\n@trace_tool()\ndef search_product(name: str):\n    return {\"name\": name, \"price\": 49.99, \"stock\": 120}\n\n@trace_tool()\ndef apply_tax(price: float, tax_rate: float):\n    return round(price * (1.0 + tax_rate), 2)\n\n# 2. Decorate your agent with @jit\n@jit\ndef checkout_agent(product_name: str, tax_rate: float):\n    # This dynamic workflow could call an LLM (Claude, GPT, Gemini)\n    item = search_product(name=product_name)\n    total = apply_tax(price=item[\"price\"], tax_rate=tax_rate)\n    return {\"item\": item[\"name\"], \"total\": total}\n\n# --- Run 1: Warmup & Tracing (runs dynamic agent, compiles to Python) ---\norder1 = checkout_agent(\"Mechanical Keyboard\", 0.19)\n\n# --- Run 2+: Instant compiled execution (ZERO tokens, sub-millisecond!) ---\norder2 = checkout_agent(\"Wireless Mouse\", 0.19)  # Takes 0.05 ms!\n```\n\nYou can view the exact synthesized Python code generated by the JIT at any time:\n\n```\nprint(checkout_agent.source_code)\n```\n\n**Synthesized Output:**\n\n``` python\ndef compiled_checkout_agent(product_name, tax_rate):\n    \"\"\"JIT-compiled trajectory pipeline generated by AgentJIT.\n    Executes deterministically in sub-millisecond time with zero token cost.\n    \"\"\"\n    # --- Speculative Guards ---\n    if not (product_name is not None):\n        raise GuardViolation(\"Argument 'product_name' must not be None\", param=\"product_name\")\n    if not (isinstance(product_name, str)):\n        raise GuardViolation(\"Argument 'product_name' must be of type str\", param=\"product_name\")\n\n    # --- Execution Steps ---\n    step_1_out = _tools['search_product'](name=product_name)\n    step_2_out = _tools['apply_tax'](price=step_1_out['price'], tax_rate=tax_rate)\n\n    # --- Return Final Result ---\n    return {'item': step_1_out['name'], 'total': step_2_out}\n```\n\nBenchmark comparing a simulated 3-step reasoning agent (15s latency, 2,500 tokens) vs AgentJIT compiled execution over 100 runs:\n\n| Execution Mode | Mean Latency | 99th Percentile | Cost per 1k runs | Token Usage | Determinism | \n|---|---|---|---|---|---|\n| **Standard LLM Agent** | `14,820 ms` | `22,400 ms` | **$75.00** | 2,500,000 | ~94% | \n| **AgentJIT (Warm Path)** | **`0.08 ms`** | **` 0.12 ms`** | **$0.00** | **0** | **100%** | \n| **Improvement** | **185,000x faster** | **186,000x faster** | **100% savings** | **Zero tokens** | **Rock-solid** | \n\nWhat happens when an input is unusual or triggers an unexpected branch?\n\nAgentJIT uses **Speculative De-Optimization**:\n\n1. Input variables are validated against synthesized guards.\n2. If any guard fails (e.g. wrong type, missing required key) or a tool raises an unhandled exception, AgentJIT catches `GuardViolation` .\n3. It seamlessly bails out to the dynamic LLM agent to handle the edge case.\n4. Telemetry records the bailout for future multi-branch specialization.\n\n```\n# Normal input: runs compiled pipeline in 0.08ms\ncheckout_agent(\"Monitor\", 0.19)\n\n# Divergent input (e.g. invalid type): automatically bails out to dynamic agent\ncheckout_agent(12345, None)  # Transparently de-optimizes, no crash!\n```\n\nMonitor your compiled agents in real time:\n\n```\nprint(checkout_agent.stats)\n# Output:\n# {\n#     \"total_calls\": 1500,\n#     \"compiled_hits\": 1492,\n#     \"bailouts\": 8,\n#     \"compiled_hit_rate\": 99.47,\n#     \"total_time_saved_ms\": 22380000.0,\n#     \"total_tokens_saved\": 3730000\n# }\n```\n\n-  **Core Tracer & DAG Flow Analyzer**\n-  **AST Code Generation with Speculative Guards**\n-  **De-optimization / Bailout Runtime**\n-  **`@jit` Decorator with Auto-Warmup**\n-  **Multi-Branch Polyhedral JIT:** Merge multiple execution paths into a unified control-flow graph (`if/else` branching synthesis).\n-  **eBPF-Isolated Micro-Sandbox:** Ultra-fast sub-millisecond process sandbox for compiled shell actions.\n-  **WebAssembly (Wasm) Export:** Compile agent trajectories into standalone Wasm binaries for browser and edge runtime.\n\nAgentJIT is open-source software licensed under the [Apache 2.0 License](/eminsk/agentjit/blob/main/LICENSE).", "url": "https://wpnews.pro/news/show-hn-agentjit-compile-dynamic-llm-agent-workflows-into-0-1ms-python", "canonical_source": "https://github.com/eminsk/agentjit", "published_at": "2026-09-13 12:39:35+00:00", "updated_at": "2026-09-13 13:10:30.363343+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "developer-tools", "ai-tools", "ai-infrastructure"], "entities": ["AgentJIT", "LangChain", "CrewAI", "AutoGen", "OpenAI", "Python", "V8", "PyTorch"], "alternates": {"html": "https://wpnews.pro/news/show-hn-agentjit-compile-dynamic-llm-agent-workflows-into-0-1ms-python", "markdown": "https://wpnews.pro/news/show-hn-agentjit-compile-dynamic-llm-agent-workflows-into-0-1ms-python.md", "text": "https://wpnews.pro/news/show-hn-agentjit-compile-dynamic-llm-agent-workflows-into-0-1ms-python.txt", "jsonld": "https://wpnews.pro/news/show-hn-agentjit-compile-dynamic-llm-agent-workflows-into-0-1ms-python.jsonld"}}