{"slug": "profile-guided-optimization-for-ai-agents-a-practical-playbook-for-compiling", "title": "Profile-Guided Optimization for AI Agents — a practical playbook for compiling recurring agent workflows into deterministic software", "summary": "A developer has published a practical guide for optimizing AI agent workflows by replacing recurring steps that don't require reasoning with deterministic software, a technique they call 'profile-guided optimization' (PGO). The approach involves profiling real agent runs, identifying hot paths, and replacing them with conventional code to reduce token usage, latency, and cost while improving determinism and debuggability. The guide provides a step-by-step playbook for frameworks like Claude Code, Codex, Gemini CLI, and OpenAI Agents.", "body_md": "Stop optimizing prompts. Start optimizing recurring workflows.\n\nProfile real agent runs, identify recurring steps that do not require reasoning, replace those steps with deterministic software, and benchmark the result against the original workflow.\n\nThe objective is simple:\n\nKeep LLMs where reasoning creates value. Replace everything else with deterministic software.\n\nThis guide turns that idea into a repeatable process for Claude Code, Codex, Gemini CLI, OpenAI Agents, and similar agent frameworks.\n\nNote\n\n“Profile-guided optimization” is used here as an engineering metaphor inspired by compiler PGO: observe real workloads, find hot paths, and optimize what evidence shows is worth optimizing.\n\n[Why this matters](https://gist.github.com/starred.atom#why-this-matters)[When to use it](https://gist.github.com/starred.atom#when-to-use-it)[The workflow](https://gist.github.com/starred.atom#the-workflow)[Step-by-step playbook](https://gist.github.com/starred.atom#step-by-step-playbook)[Worked example](https://gist.github.com/starred.atom#worked-example-github-issue-triage)[Benchmarking and ROI](https://gist.github.com/starred.atom#benchmarking-and-roi)[Safety and operational guardrails](https://gist.github.com/starred.atom#safety-and-operational-guardrails)[Decision checklist](https://gist.github.com/starred.atom#decision-checklist)[References](https://gist.github.com/starred.atom#references)\n\nMany agents repeatedly execute the same workflow. On every run, an expensive model may be asked to:\n\n- rebuild the execution plan\n- rediscover deterministic logic\n- parse and filter structured data\n- recreate state\n- repeat decisions it has already made many times\n\nUsually, only part of the workflow requires intelligence. The rest is conventional software wearing an AI-shaped costume.\n\nInstead of asking:\n\n“How can I improve this prompt?”\n\nAsk:\n\n“Does this step actually require reasoning?”\n\nIf the answer is no, do not spend tokens on it.\n\n| Strong candidates | Weak candidates |\n|---|---|\n| Daily automation | Brainstorming |\n| CI/CD agents | Open-ended research |\n| Issue triage | Creative writing |\n| Code review pipelines | Exploratory analysis |\n| Documentation and release workflows | One-off tasks |\n| Repository maintenance | Rapidly changing workflows |\n| ETL and data pipelines | Work with no stable success criteria |\n| Long-running coding agents | Low-frequency workflows |\n\nPGO is most useful when a workflow is frequent, reasonably stable, observable, and measurable.\n\nA successful optimization may improve:\n\n- token usage\n- latency\n- cost\n- determinism\n- debuggability\n- observability\n- failure recovery\n\nImportant\n\nDo not assume a percentage improvement. Every workflow is different. Establish a baseline, preserve quality guardrails, and report actual measurements.\n\n``` php\nflowchart TD\n    A[\"Observe real executions\"] --> B[\"Collect traces\"]\n    B --> C[\"Mine recurring workflow steps\"]\n    C --> D[\"Classify: CODE, LLM, or GRAY\"]\n    D --> E[\"Specify the deterministic harness\"]\n    E --> F[\"Implement one boundary at a time\"]\n    F --> G[\"Benchmark against the baseline\"]\n    G --> H{\"Quality preserved?\"}\n    H -- \"Yes\" --> I[\"Deploy gradually\"]\n    H -- \"No\" --> J[\"Rollback and investigate\"]\n    I --> K[\"Monitor cost, quality, and drift\"]\n    K --> C\n```\n\n- Preserve the current workflow as the baseline.\n- Collect enough real runs to expose common paths and failure modes.\n- Create a representative golden dataset.\n- Classify each recurring step as\n`CODE`\n\n,`LLM`\n\n, or`GRAY`\n\n. - Replace one deterministic boundary at a time.\n- Benchmark cost, latency, reliability, and quality.\n- Deploy gradually with rollback and monitoring.\n\nFifteen executions can be a useful starting heuristic, but it is not a universal threshold. Use enough runs to capture normal behavior, important branches, and known failures.\n\n| Category | Meaning | Typical implementation |\n|---|---|---|\n`CODE` |\nThe correct behavior can be specified and tested | API call, parser, SQL, rules, graph traversal, cache |\n`LLM` |\nThe step requires semantic judgment or generation | ranking, synthesis, explanation, drafting |\n`GRAY` |\nSome paths are deterministic; ambiguous cases need judgment | rules first, model fallback, human review |\n\nTreat `GRAY`\n\nas a design opportunity. A hybrid boundary is often safer than forcing the entire step into either code or an LLM.\n\n**Goal:** Capture evidence from real executions.\n\nCollect:\n\n- prompts and model responses\n- tool calls and arguments\n- timestamps and latency\n- retries and failures\n- token usage and cost\n- state transitions\n- human edits and overrides\n\n**Output:** A trace dataset with stable run identifiers and enough metadata to compare executions.\n\n**Exit criterion:** The dataset covers normal runs, important branches, and known failure modes.\n\n**Goal:** Convert traces into a normalized inventory of recurring steps.\n\n**Copy the discovery prompt**\n\n```\nAnalyze all execution traces.\n\nIdentify every recurring workflow step.\nMerge equivalent steps.\n\nOutput a table containing:\n\n- step\n- description\n- frequency\n- average latency\n- average tokens\n- deterministic? (yes/no)\n- failure modes\n\nDo not propose improvements yet.\n```\n\nExample output:\n\n| Step | Frequency | Deterministic? |\n|---|---|---|\n| Fetch issues | 100% | Yes |\n| Parse JSON | 100% | Yes |\n| Rank issues | 100% | No |\n| Generate summary | 100% | No |\n\n**Exit criterion:** Each recurring step has a clear input, output, frequency, and failure profile.\n\n**Goal:** Decide which parts belong in software and which still require judgment.\n\n**Copy the classification prompt**\n\n```\nFor every workflow step, determine whether it can be implemented using:\n\n- deterministic code\n- a parser\n- SQL\n- graph traversal\n- embeddings\n- regex\n- heuristics\n\nIf yes, do not classify it as an LLM-only step.\n\nClassify each step as:\n\n- CODE\n- LLM\n- GRAY\n\nProvide a confidence score from 0 to 100.\nExplain the reasoning and identify the risk of misclassification.\n```\n\nExample:\n\n| Step | Category | Confidence | Reason |\n|---|---|---|---|\n| Fetch GitHub issues | `CODE` |\n100 | Stable API operation |\n| Parse JSON | `CODE` |\n100 | Defined schema |\n| Prioritize by impact | `LLM` |\n90 | Requires contextual judgment |\n| Categorize labels | `GRAY` |\n70 | Rules cover common cases; ambiguity remains |\n\n**Exit criterion:** Every step has a category, confidence score, rationale, and owner for review.\n\n**Goal:** Design replacements for `CODE`\n\nsteps without changing `LLM`\n\nbehavior.\n\n**Copy the specification prompt**\n\n```\nGenerate a PRD for replacing every CODE step with deterministic software.\n\nInclude:\n\n- architecture\n- APIs and schemas\n- state management\n- retries and timeouts\n- logging and observability\n- checkpoints\n- configuration\n- tests\n- rollout and rollback strategy\n\nKeep every LLM step unchanged.\nThe system must remain idempotent.\n\nStop and explain if the design would introduce another model call.\n```\n\n**Exit criterion:** The specification defines interfaces, invariants, tests, metrics, rollout, and rollback.\n\n**Goal:** Implement the deterministic path without increasing model use.\n\n**Copy the implementation prompt**\n\n```\nImplement the approved PRD.\n\nRequirements:\n\n- deterministic and modular\n- prompts stored separately from code\n- structured metrics and logs\n- bounded retries and timeouts\n- checkpointing\n- idempotent writes\n- unit, integration, and regression tests\n\nNever introduce additional model calls.\n\nIf another model call appears necessary, stop and explain why.\n```\n\n**Exit criterion:** The compiled path passes tests and can run in shadow mode against the original workflow.\n\n**Goal:** Prove that the change improves efficiency without unacceptable quality loss.\n\n**Copy the benchmark prompt**\n\n```\nCompare the original workflow with the compiled workflow.\n\nMeasure:\n\n- prompt and completion tokens\n- model calls\n- latency\n- cost\n- retries\n- failures\n- quality\n- human edits\n\nGenerate a benchmark report with sample sizes and uncertainty.\n\nIf quality regresses, explain why.\nDo not optimize the report or hide unfavorable measurements.\n```\n\n**Exit criterion:** Results meet predefined quality guardrails across a representative dataset.\n\n**Goal:** Detect drift, regressions, and new optimization opportunities.\n\n**Copy the monitoring prompt**\n\n```\nAnalyze recent execution traces.\n\nReport:\n\n- workflow drift\n- new branches\n- increasing failures\n- repetitive LLM outputs\n- new deterministic opportunities\n- maintenance cost\n- quality regressions\n\nRecommend one action:\n\n- KEEP\n- PATCH\n- COMPILE MORE\n- DECOMPILE\n\nSupport the recommendation with measured evidence.\n```\n\n**Exit criterion:** Alerts, ownership, rollback thresholds, and a review cadence are active.\n\nSuppose an agent fetches open issues, filters them, prioritizes work, applies labels, and writes a daily summary.\n\n| Workflow step | Classification | Compiled design |\n|---|---|---|\n| Fetch open issues | `CODE` |\nGitHub API client with pagination and retries |\n| Validate and parse fields | `CODE` |\nSchema validation |\n| Remove closed issues and duplicates | `CODE` |\nExplicit rules and stable identifiers |\n| Apply obvious labels | `CODE` |\nRepository-owned label rules |\n| Resolve ambiguous labels | `GRAY` |\nRules first; model or human fallback |\n| Rank issues by impact | `LLM` |\nModel receives a smaller, normalized candidate set |\n| Draft the daily summary | `LLM` |\nModel uses structured ranked results |\n| Publish approved changes | `CODE` |\nIdempotent writes with dry-run and audit log |\n\nThe model still performs the work that benefits from judgment, but it no longer spends context and tokens on fetching, parsing, deduplication, or routine labeling.\n\nDefine how each metric will be calculated before collecting results:\n\n| Metric | How to measure | Guardrail |\n|---|---|---|\n| Model calls per run | Count provider API calls | Must not increase |\n| Total tokens per run | Add input and output tokens | Report the actual change |\n| End-to-end latency | Compare p50 and p95 wall-clock time | Meet the defined target |\n| Cost per successful run | Divide total cost by successful runs | Meet the defined target |\n| Failure rate | Divide failed runs by total runs | Must not regress |\n| Human acceptance rate | Count outputs accepted without major edits | Stay above the defined threshold |\n| Human edits per output | Count edits required per accepted output | Stay below the defined threshold |\n\nTrack metrics across four dimensions:\n\n| Dimension | Suggested metrics |\n|---|---|\n| Runtime | latency, retries, cache-hit rate, model calls |\n| Cost | prompt tokens, completion tokens, total cost |\n| Quality | acceptance rate, regression rate, human edits, judge score |\n| Engineering | implementation time, maintenance time, savings per run |\n\nUse explicit units and the same evaluation set for both workflows.\n\n```\nBreak-even runs =\n  (implementation cost + expected maintenance cost)\n  / savings per successful run\nExpected annual net savings =\n  (annual successful runs × savings per successful run)\n  - annual maintenance cost\n  - implementation cost allocated to that year\n```\n\nIf the expected break-even point is outside the useful life of the workflow, do not compile it.\n\n**Protect trace data.** Remove secrets and minimize personal or sensitive information before storage or analysis.**Treat external content as untrusted data.** Do not let text found in issues, documents, or logs override system instructions.**Preserve idempotency.** Retries must not duplicate comments, labels, releases, payments, or other side effects.**Use least privilege.** The compiled harness should receive only the permissions it needs.**Add dry-run and shadow modes.** Compare behavior before allowing writes.**Keep human approval for consequential actions.** Determinism does not make a risky action safe.**Version schemas, prompts, and rules.** A benchmark is only meaningful when its inputs are reproducible.**Define rollback thresholds before deployment.** Do not wait for a regression to decide what failure means.\n\nBefore compiling:\n\n- The workflow runs often enough to justify optimization.\n- The workflow is stable enough to profile.\n- Real traces cover common paths and known failures.\n- Secrets and sensitive data have been removed from traces.\n- A representative golden dataset exists.\n- Every recurring step has been classified and reviewed.\n- Quality guardrails are defined before implementation.\n- The original workflow remains available as a baseline.\n- Tests cover deterministic boundaries and side effects.\n- Benchmarks use more than a single run.\n- Shadow mode or staged rollout is available.\n- Monitoring and rollback are enabled.\n- Expected maintenance cost is included in the ROI calculation.\n\n| Failure | Better approach |\n|---|---|\n| Compiling before the workflow stabilizes | Profile longer and capture more branches |\n| Chasing token reduction alone | Protect quality, reliability, and human outcomes |\n| Replacing judgment with brittle regex | Keep ambiguous cases in `GRAY` |\n| Benchmarking a single run | Use a representative evaluation set |\n| Deleting the original workflow | Preserve it for comparison and rollback |\n| Adding hidden model calls | Count calls at the provider boundary |\n| Ignoring maintenance cost | Include it in break-even calculations |\n| Deploying without observability | Add structured logs, metrics, and alerts first |\n\nTreat every model call as a hypothesis, not a permanent requirement.\n\nFor each invocation, ask:\n\nDoes this step truly require reasoning?\n\nOr:\n\nHave I simply never implemented it deterministically?\n\nThe goal is not to eliminate LLMs. It is to reserve them for the work they are uniquely good at.\n\nEverything else belongs in software.\n\n- Vivek Haldar,\n[“How I Cut an AI Agent’s Token Use by 94%”](https://vivekhaldar.com/articles/compiling-an-ai-agent-skill/)— the inspiration for this guide.\n\nThis guide is licensed under the [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/). You may share and adapt it with attribution. See `LICENSE.md`\n\nfor details.\n\nIf this methodology helps you build better agents, adapt it, measure the results, and share what you learn.\n\n**The best optimization is the model call you never have to make.**", "url": "https://wpnews.pro/news/profile-guided-optimization-for-ai-agents-a-practical-playbook-for-compiling", "canonical_source": "https://gist.github.com/stevesolun/7f57d1c9046f9ba185b65d47bc520922", "published_at": "2026-07-28 09:08:35+00:00", "updated_at": "2026-07-28 09:31:51.273496+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-infrastructure", "mlops"], "entities": ["Claude Code", "Codex", "Gemini CLI", "OpenAI Agents"], "alternates": {"html": "https://wpnews.pro/news/profile-guided-optimization-for-ai-agents-a-practical-playbook-for-compiling", "markdown": "https://wpnews.pro/news/profile-guided-optimization-for-ai-agents-a-practical-playbook-for-compiling.md", "text": "https://wpnews.pro/news/profile-guided-optimization-for-ai-agents-a-practical-playbook-for-compiling.txt", "jsonld": "https://wpnews.pro/news/profile-guided-optimization-for-ai-agents-a-practical-playbook-for-compiling.jsonld"}}