Optimize an AI agent to sound human, judged by an AI detector

LaunchDarkly's agent optimization feature was used to automatically tune an AI email-drafting agent to sound human, judged by the AI detector GPTZero. The developer built a workflow where Claude drafts replies and generates prompt variations, while GPTZero scores AI-likeness, driving the optimizer to reduce the score. Each iteration costs about $0.002, and the reusable GPTZero integration can be adapted for other external scorers.

You can tell when an LLM wrote an email. The "I hope this email finds you well" opener, the three polite paragraphs answering a one-line question. I wanted a reply-drafting agent that didn't do that, and "don't sound like an AI" turned out to be hard to put in a prompt. Banning a few phrases is easy. The rest is judgment, and a single prompt that holds across a friendly dinner invite and a recruiter cold-email took more iterations than I'd guessed. This is not only an email problem. Some platforms down-rank content that reads as AI-generated, so teams publishing at scale have a real stake in prose that clears a detector, even when a human wrote it. The workflow here applies to any of that. So I stopped hand-tuning and let LaunchDarkly agent optimization https://launchdarkly.com/docs/home/agentcontrol search for the prompt. You give it a judge that scores "better," and it generates prompt variations and keeps the ones that beat the bar. For the reasoning behind the feature, read the agent optimization announcement https://launchdarkly.com/blog/agentcontrol-agent-optimization/ . This tutorial is the how. If you don't have an account yet, sign up for LaunchDarkly https://app.launchdarkly.com/signup to follow along. Two pieces do the work here. Claude claude-haiku-4-5-20251001 runs both roles: it drafts the replies, and it writes each new candidate prompt when the loop asks for one. Scoring comes from GPTZero, which isn't a language model at all but a closed AI detector. I wired it in inverted, so the score is the probability a reply reads as AI and the optimizer drives it down. I went with a detector instead of an LLM-as-a-judge for a reason: grading one model's prose by asking another model whether it sounds human is exactly the call language models are unreliable at, and a tool trained for that one question gives a number you can defend. A run is cheap. Each iteration costs around $0.002 and a few seconds, so a full run lands near a penny or two, and the loop tries variations I'd never sit down and type by hand. This tutorial runs from a saved configYou bootstrap the agent, the judge, and the optimization, then work in the UI. Every iteration streams to the optimization Results tab, the winner lands on the agent Variations tab, and you tune thresholds and inputs on the optimization itself. The code does two things: it scores AI-likeness with GPTZero, and it runs the optimization. email-agent , seeded with one deliberately thin instruction ai-likeness , scored by GPTZero in code rather than by a prompt email-agent-opt , that runs as a candidate generator and streams to the Results tabThe GPTZero integration is the reusable part. The same shape works for any external scorer you might bring, whether a moderation API, a classifier you host, or a scoring endpoint of your own, so what you learn here isn't limited to email or to AI detection. The companion repo is agent-optimization-sample https://github.com/launchdarkly-labs/agent-optimization-sample . Clone it to follow along. .env Install the project and its dependencies: Terminal uv sync .env holds LD SDK KEY, LD API KEY, LD PROJECT KEY, ANTHROPIC API KEY, AI LIKENESS API KEY The repo aliases the short LD names to the LAUNCHDARKLY names the SDK expects, so the short names in .env are enough. Agent optimization runs an iterative loop against an AgentControl config https://launchdarkly.com/docs/home/agentcontrol/create . It measures your current variation as a baseline, generates candidate variations, and scores each against your acceptance criteria. The loop has a simple shape: ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ │ Define │────▶│ Explore │────▶│ Commit │ │ "Better" │ │ candidates │ │ a winner │ └──────▲──────┘ └──────────────┘ └──────┬──────┘ │ │ └────────────────────────────────────────┘ Define better. You set acceptance criteria with a judge and a threshold. A judge scores a response on one dimension. You reference a judge https://launchdarkly.com/docs/home/agentcontrol/judges saved as an AgentControl config by its key. Explore candidates. Each iteration drafts against your inputs, scores the result, and writes the next candidate from what the scores tell it. The threshold here is a gate that keeps the loop generating. When a candidate clears it, the optimizer re-runs that same prompt against a few more of your input samples and keeps it only if it passes those too, so a prompt that got lucky on one message doesn't win. Even then, clearing the gate doesn't certify that a candidate is good enough to ship. A fuller eval decides that, later. Commit the winner. The recommended variation shows up in LaunchDarkly, and with autoCommit it publishes back to the agent's Variations tab https://launchdarkly.com/docs/home/agentcontrol/create-variation so you can read what the optimizer wrote. You also pick an evaluation mode. Exploratory mode infers quality from the judge alone, which suits open-ended inputs that have no single correct output. Expected Output mode scores against known-correct answers. Replies have no single right answer, so this tutorial stays in Exploratory mode. The technique behind itAgent optimization is an instance of OPRO Optimization by PROmpting , introduced in Google DeepMind's Large Language Models as Optimizers . A model reads the history of prompts and their scores, then writes the next candidate to try. It searches over prompts, not model weights, so each candidate is cheap to run and nothing is ever trained. The companion repo keeps the moving parts in small files, so each piece is easy to find and swap: bootstrap.py : seeds the three LaunchDarkly objects, the email-agent config, the ai-likeness judge, and the email-agent-opt optimization. It's safe to re-run, and it prints links to the configs and the Results tab. optimize from config.py : the one run command. It reads the saved optimization, runs it, streams each iteration to the Results tab, and prints the tab's link at the end. optimize.py : the two callbacks the run needs. handle agent call drafts replies on Claude, and writes the next candidate prompt on Claude too when the SDK asks for one. handle judge call scores AI-likeness with GPTZero and hands the optimizer the per-reply detector output. detector.py : the GPTZero client. score with response returns the number the judge gates on and the full GPTZero JSON. messages.py : the synthetic input messages. gptzero test.py : a standalone probe for scoring a draft by hand. clients.py and env.py : the LaunchDarkly and Anthropic clients, built once each, and the .env loader.The saved optimization holds what you're optimizing for: the judge, the threshold, the inputs, and the model choices. The code holds how the work happens, drafting on Claude and scoring with GPTZero. You edit the what in the UI and the how in code, and the run command brings the two together. One command seeds everything this tutorial needs. bootstrap.py creates three objects in LaunchDarkly, and it's idempotent, so anything that already exists is left alone: email-agent : the agent {"replies": ... } envelope and the {{messages}} variable, and nothing about tone. That's on purpose. It leaves the humanization, the part you want optimized, to the optimizer. ai-likeness : the inverted judge config. GPTZero scores it from code, which leaves the judge prompt as a placeholder. email-agent-opt : the saved optimization the Results tab runs. Thresholds, inputs, and model choices all live here.The baseline is thin on tone but carries the output contract the parser needs, on the Claude model the agent drafts with: bootstrap.py variation { "key": "baseline", "name": "Baseline", "model": {"modelName": "claude-haiku-4-5-20251001", "parameters": {}}, "instructions": "You are an email assistant. Write a reply to each message below. " 'Return ONLY a JSON object {"replies": "<reply to Message 1 ", ... } ' "with one reply per message, in order.\n\n{{messages}}" , } Run the bootstrap: Terminal uv run python bootstrap.py It prints links to the two configs and the one command that runs the optimization. One judge by designThe New optimization form in the UI attaches a single judge, so this tutorial uses one: AI-likeness. The bootstrap creates the same single-judge optimization in code, so the run matches what the UI supports. To build it by hand instead, open Agent optimization, thenNew optimization, target email-agent , add the ai-likeness judge, set the threshold and the input messages, and save. The agent drafts against a fixed set of messages, injected into the prompt through the {{messages}} variable. Keep them diverse on purpose, so a winning prompt generalizes instead of overfitting to one kind of message. The agent replies to all of them in one call and the judge averages the AI-likeness scores, and Step 3 explains why that averaging matters. messages.py holds the message list: messages.py SYNTHETIC MESSAGES = "Hey Are you free Saturday for dinner? A few of us are getting together and it'd be great to see you.", "Hi, I came across your background and I'm hiring for a senior role that looks like a strong fit. Open to a quick call this week?", "It's been way too long I'll be in town Thursday and Friday. Any chance you're around to grab coffee?", "Just a reminder that your dentist appointment is Tuesday at 2pm. Reply to confirm or reschedule.", "Hey neighbor, a package addressed to you was left at my door by mistake. Want to swing by this weekend?", 5 more, spanning tone, intent, and length AI-likeness is the target, and it's the one judge that has to live in code, because it isn't an LLM. You score the reply with GPTZero https://gptzero.me/ , an AI detector. The score is 1 - P human : near 0 when GPTZero reads the reply as human, which is your goal, and near 1 for AI or mixed text. Lower is better, so the bootstrap marks the judge inverted and the optimizer drives the score down. GPTZero is also the more useful integration to learn, because it generalizes to any external scorer. The SDK gives a judge config no hook to reach outside code, so the config exists only so the optimization can attach a judge, its prompt stays a placeholder, and the real number comes from a small client you call yourself: detector.py python def score with response text: str : data = api call text POST the reply to GPTZero doc = data.get "documents" or {} 0 return 1.0 - doc "class probabilities" "human" , data 1 - P human , + full JSON score with response returns both the number the judge gates on and the full GPTZero JSON, so the callback can forward the detail to the optimizer. Set AI LIKENESS API KEY to your GPTZero key. Two things here cost me time. GPTZero sits behind Cloudflare, and a plain urllib request comes back as a 403 with error code: 1010 before it reaches the API. That reads like a bad key, but it isn't. Sending any real User-Agent header clears it. The score also comes from 1 - P human rather than the average generated prob field, which is the fraction of sentences flagged AI and reports 1.0 even on text the detector still classifies as human. Gating on that field punishes replies that already passed. GPTZero is the only judge, so handle judge call doesn't branch on the judge key. Every call pulls the batch of replies, scores each one with GPTZero, and averages: optimize.py python async def handle judge call key, config, context, is evaluation=True : text = extract candidate context.user input or "" replies = r for r in json.loads text or {} .get "replies", if isinstance r, str and r.strip if not replies: empty draft must FAIL the inverted gate return OptimizationResponse output=json.dumps {"score": 1.0, "rationale": "empty or degenerate candidate"} results = for reply in replies: one GPTZero call per reply in the batch s, raw = detector.score with response reply 1 - P human ; + full JSON results.append {"reply": reply, "ai likeness": s, "gptzero": raw} avg = round sum r "ai likeness" for r in results / len results , 4 The gate is the AVERAGE; the full per-reply GPTZero JSON becomes the rationale. return OptimizationResponse output=json.dumps {"score": avg, "rationale": f"Average AI-likeness = {avg}. {json.dumps results }"} Two choices in there are what make optimizing against a detector actually work. Score a batch and average. A single short reply's GPTZero score is noisy. The very same prompt can produce a reply it calls 99% human on one message and 99% AI on the next. Each turn drafts replies to all the messages in one batch, and the judge averages those scores into something steady enough to optimize against. Forward the whole detector response. The rationale you return goes straight to the model that writes the next prompt. Instead of a bare number, return the full GPTZero JSON, with its per-sentence probabilities and predicted class. The optimizer reads that directly and revises around whatever scored as AI, with no parsing on your side. The empty-draft case is the one I got wrong first. An empty reply scores 0.0 , which the inverted gate reads as perfectly human, so an early version let the optimizer win by drafting nothing. Now an empty or malformed batch, or a detector error, returns 1.0 and fails the gate. To build intuition before you run the loop, probe GPTZero on a draft by hand: Terminal uv run python gptzero test.py "your draft reply" A detector is a black box, so gate accordinglyA detection service gives you a defensible score immediately, with no model to train. It also has real limits. You can't tune it, it leans toward calling short LLM text AI, and it bills per call across every iteration. That confidence is the reason to average over a batch instead of gating on a single reply. The saved optimization holds the judge, the threshold, the inputs, and the model choices, so the run command takes none of them. The agent drafts on Claude, the optimizer writes each new prompt on Claude too, and the threshold keeps the loop generating rather than picking a winner. Here is the saved optimization: bootstrap.py optimization { "key": "email-agent-opt", "aiConfigKey": "email-agent", "maxAttempts": 10, "judgeModel": "claude-haiku-4-5-20251001", required by the API; the GPTZero judge never calls it "modelChoices": "claude-haiku-4-5-20251001" , the Claude model the agent drafts with "judges": {"key": "ai-likeness", "threshold": 0.5} , generator gate, not a winner test "variableChoices": interpolated into the instructions; the optimizer must use every one {"sender type": "friend", "respondent name": "Jordan Lee", "messages": MESSAGES BLOCK}, {"sender type": "professional contact", "respondent name": "Jordan Lee", "messages": MESSAGES BLOCK}, , "userInputOptions": "Draft the replies now." , trigger turn; messages come from {{messages}} "autoCommit": True, } MESSAGES BLOCK is the message list from Step 2, formatted and fed in through {{messages}} . respondent name and sender type are the other two variables, so replies come out signed and pitched to the right register. The optimizer has to use every variable you declare, which is what keeps {{messages}} and the JSON envelope intact through every rewrite. The threshold is 0.5 , and that number came from watching GPTZero, not from theory. A confident detector scores even clearly human-sounding short replies well above 0 , so a gate near 0 never trips and the loop never finds anything to keep. At 0.5 , this run passed at iteration 6 with 0.43 , while the earlier iterations landed between 0.54 and 1.00 . That gave the loop room to explore without rubber-stamping every candidate. Run it from the saved config: Terminal OPTIMIZATION KEY=email-agent-opt uv run python optimize from config.py Settings live on the optimizationThresholds, inputs, models, and maxAttempts are baked into email-agent-opt at bootstrap time. Change them by editing the optimization in the UI, or by deleting it and re-running bootstrap.py with the matching environment variables set. Committing the winner back as a variation needs the REST API key. The command prints a link to the Results tab. One callback drafts replies on Claude, and the SDK reuses that same callback to write the next prompt, also on Claude. Each iteration posts its prompt and score to the Results tab as a candidate. Open the Results tab from the printed link. Each iteration posts as it runs, with its candidate prompt and AI-likeness average alongside the variation the run currently recommends. The Results tab after a passing run. Iteration 6 cleared the 0.50 gate at 0.43 and committed optimistic-coyote, with per-iteration charts for AI-likeness, latency, tokens, and cost. Click any iteration to drill into its candidate prompt, the input it ran against, and the replies it produced. Iteration 1 is the baseline template itself, scoring 0.64 . Its replies already read decently "Thanks for the invite I'd love to come to dinner Saturday." , but the thin prompt left enough AI signal for the detector to flag. Iteration 1, the baseline template: the JSON-and-{{messages}} instruction, the "Draft the replies now." trigger input, and the replies it produced. At 0.64 it didn't clear the gate. The optimization sets autoCommit , so on success the winner publishes back to email-agent as a new variation. Open the agent Variations tab to read what the optimizer wrote. The run committed a new variation, optimistic-coyote . The Variations tab shows it next to the baseline, so you can read the change directly. The optimizer kept the JSON envelope and the {{sender type}} and {{messages}} variables, and built a full humanization spec around them: The Variations tab: the thin baseline above the committed winner optimistic-coyote, both on claude-haiku-4-5-20251001. The optimizer preserved the {{sender type}} and {{messages}} variables and the JSON envelope while adding the humanization guidance. Every line of it is a humanization lever, and they map onto how GPTZero separates the two classes: {{respondent name}} instead of a placeholder.The optimizer took a prompt that said nothing about tone and built out a detailed spec, and GPTZero scored the resulting replies as more human. That's one of the candidates the loop surfaced. To decide whether it's worth shipping, take it into a fuller offline eval https://launchdarkly.com/docs/home/agentcontrol/offline-evaluations for real data, which the sections below cover. You can run all of this from the UI or from code. The New optimization form builds the same optimization by hand and streams to the same Results tab. This tutorial used optimize from config , which runs a saved optimization from code while its judge, threshold, inputs, and models stay editable in the UI. To define everything in code instead, optimize from options takes the settings directly and accepts more than one judge, and optimize from ground truth options handles Expected Output mode when you have correct answers to match. This demo leaves several controls unused: token optimization and latency optimization for a cost-and-latency pass, token limit for a spend cap, variation key , output key , context choices , and the on turn , on passing result , on failing result , and on status update callbacks. Offline evals and optimization sit next to each other in AgentControl and do opposite jobs. An offline eval https://launchdarkly.com/docs/home/agentcontrol/offline-evaluations measures a configuration you already have: you run your agent over a dataset https://launchdarkly.com/docs/home/agentcontrol/datasets , score it with judges https://launchdarkly.com/docs/home/agentcontrol/judges , and answer "how good is this, and did anything regress?" Optimization runs the other direction, generating new variations until one clears that same judge. Running a surfaced candidate back through an eval is that measuring job again, telling you how it actually performs on a fuller set. They're strongest together. Here's a path through AgentControl that gets the most out of both: Run it long enough and production signals become the next round's eval data, so each optimization starts from what you actually saw in production rather than a guess. You started with a prompt that said nothing about tone and let agent optimization rewrite it against GPTZero https://gptzero.me/ , then read the winning humanization spec off the Variations tab. The loop's job was to explore cheaply, and the trustworthy verdict comes from a proper offline eval https://launchdarkly.com/docs/home/agentcontrol/offline-evaluations over a comprehensive dataset https://launchdarkly.com/docs/home/agentcontrol/datasets . Agent optimization https://launchdarkly.com/docs/home/agentcontrol is one step in a larger workflow. You define judges https://launchdarkly.com/docs/home/agentcontrol/judges for what "better" means, optimize against them to surface candidates, then check those candidates with offline https://launchdarkly.com/docs/tutorials/offline-evals and online evals https://launchdarkly.com/docs/tutorials/when-to-add-online-evals . What you learn in production feeds the next round. If you're getting started, Build a LangGraph multi-agent system https://launchdarkly.com/docs/tutorials/agents-langgraph is a good place to begin. Sign up for LaunchDarkly https://app.launchdarkly.com/signup and point the loop at your own prompts.