You can tell when an LLM wrote an email. The "I hope this email finds you well" opener, the three polite paragraphs answering a one-line question. I wanted a reply-drafting agent that didn't do that, and "don't sound like an AI" turned out to be hard to put in a prompt. Banning a few phrases is easy. The rest is judgment, and a single prompt that holds across a friendly dinner invite and a recruiter cold-email took more iterations than I'd guessed.
This is not only an email problem. Some platforms down-rank content that reads as AI-generated, so teams publishing at scale have a real stake in prose that clears a detector, even when a human wrote it. The workflow here applies to any of that.
So I stopped hand-tuning and let LaunchDarkly agent optimization search for the prompt. You give it a judge that scores "better," and it generates prompt variations and keeps the ones that beat the bar. For the reasoning behind the feature, read the agent optimization announcement. This tutorial is the how. If you don't have an account yet, sign up for LaunchDarkly to follow along.
Two pieces do the work here. Claude (claude-haiku-4-5-20251001
) runs both roles: it drafts the replies, and it writes each new candidate prompt when the loop asks for one. Scoring comes from GPTZero, which isn't a language model at all but a closed AI detector. I wired it in inverted, so the score is the probability a reply reads as AI and the optimizer drives it down. I went with a detector instead of an LLM-as-a-judge for a reason: grading one model's prose by asking another model whether it sounds human is exactly the call language models are unreliable at, and a tool trained for that one question gives a number you can defend.
A run is cheap. Each iteration costs around $0.002 and a few seconds, so a full run lands near a penny or two, and the loop tries variations I'd never sit down and type by hand.
This tutorial runs from a saved configYou bootstrap the agent, the judge, and the optimization, then work in the UI. Every iteration streams to the optimization Results tab, the winner lands on the agent Variations tab, and you tune thresholds and inputs on the optimization itself. The code does two things: it scores AI-likeness with GPTZero, and it runs the optimization.
email-agent
, seeded with one deliberately thin instructionai-likeness
, scored by GPTZero in code rather than by a promptemail-agent-opt
, that runs as a candidate generator and streams to the Results tabThe GPTZero integration is the reusable part. The same shape works for any external scorer you might bring, whether a moderation API, a classifier you host, or a scoring endpoint of your own, so what you learn here isn't limited to email or to AI detection.
The companion repo is agent-optimization-sample. Clone it to follow along.
.env
Install the project and its dependencies:
Terminal
uv sync
The repo aliases the short LD_*
names to the LAUNCHDARKLY_*
names the SDK expects, so the short names in .env
are enough.
Agent optimization runs an iterative loop against an AgentControl config. It measures your current variation as a baseline, generates candidate variations, and scores each against your acceptance criteria. The loop has a simple shape:
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
│ Define │────▶│ Explore │────▶│ Commit │
│ "Better" │ │ candidates │ │ a winner │
└──────▲──────┘ └──────────────┘ └──────┬──────┘
│ │
└────────────────────────────────────────┘
Define better. You set acceptance criteria with a judge and a threshold. A judge scores a response on one dimension. You reference a judge saved as an AgentControl config by its key.
Explore candidates. Each iteration drafts against your inputs, scores the result, and writes the next candidate from what the scores tell it. The threshold here is a gate that keeps the loop generating. When a candidate clears it, the optimizer re-runs that same prompt against a few more of your input samples and keeps it only if it passes those too, so a prompt that got lucky on one message doesn't win. Even then, clearing the gate doesn't certify that a candidate is good enough to ship. A fuller eval decides that, later.
Commit the winner. The recommended variation shows up in LaunchDarkly, and with autoCommit
it publishes back to the agent's Variations tab so you can read what the optimizer wrote.
You also pick an evaluation mode. Exploratory mode infers quality from the judge alone, which suits open-ended inputs that have no single correct output. Expected Output mode scores against known-correct answers. Replies have no single right answer, so this tutorial stays in Exploratory mode.
The technique behind itAgent optimization is an instance of OPRO (Optimization by PROmpting), introduced in Google DeepMind's
[Large Language Models as Optimizers]. A model reads the history of prompts and their scores, then writes the next candidate to try. It searches over prompts, not model weights, so each candidate is cheap to run and nothing is ever trained.
The companion repo keeps the moving parts in small files, so each piece is easy to find and swap:
bootstrap.py
: seeds the three LaunchDarkly objects, the email-agent
config, the ai-likeness
judge, and the email-agent-opt
optimization. It's safe to re-run, and it prints links to the configs and the Results tab.optimize_from_config.py
: the one run command. It reads the saved optimization, runs it, streams each iteration to the Results tab, and prints the tab's link at the end.optimize.py
: the two callbacks the run needs. handle_agent_call
drafts replies on Claude, and writes the next candidate prompt on Claude too when the SDK asks for one. handle_judge_call
scores AI-likeness with GPTZero and hands the optimizer the per-reply detector output.detector.py
: the GPTZero client. score_with_response
returns the number the judge gates on and the full GPTZero JSON.messages.py
: the synthetic input messages.gptzero_test.py
: a standalone probe for scoring a draft by hand.clients.py
and env.py
: the LaunchDarkly and Anthropic clients, built once each, and the .env
.The saved optimization holds what you're optimizing for: the judge, the threshold, the inputs, and the model choices. The code holds how the work happens, drafting on Claude and scoring with GPTZero. You edit the what in the UI and the how in code, and the run command brings the two together.
One command seeds everything this tutorial needs. bootstrap.py
creates three objects in LaunchDarkly, and it's idempotent, so anything that already exists is left alone:
email-agent
: the agent {"replies": [...]}
envelope and the {{messages}}
variable, and nothing about tone. That's on purpose. It leaves the humanization, the part you want optimized, to the optimizer.ai-likeness
: the inverted judge config. GPTZero scores it from code, which leaves the judge prompt as a placeholder.email-agent-opt
: the saved optimization the Results tab runs. Thresholds, inputs, and model choices all live here.The baseline is thin on tone but carries the output contract the parser needs, on the Claude model the agent drafts with:
bootstrap.py (variation)
{
"key": "baseline",
"name": "Baseline",
"model": {"modelName": "claude-haiku-4-5-20251001", "parameters": {}},
"instructions": (
"You are an email assistant. Write a reply to each message below. "
'Return ONLY a JSON object {"replies": ["<reply to Message 1>", ...]} '
"with one reply per message, in order.\n\n{{messages}}"
),
}
Run the bootstrap:
Terminal
uv run python bootstrap.py
It prints links to the two configs and the one command that runs the optimization.
One judge by designThe New optimization form in the UI attaches a single judge, so this tutorial uses one: AI-likeness. The bootstrap creates the same single-judge optimization in code, so the run matches what the UI supports. To build it by hand instead, open
Agent optimization, thenNew optimization, targetemail-agent
, add theai-likeness
judge, set the threshold and the input messages, and save.
The agent drafts against a fixed set of messages, injected into the prompt through the {{messages}}
variable. Keep them diverse on purpose, so a winning prompt generalizes instead of overfitting to one kind of message. The agent replies to all of them in one call and the judge averages the AI-likeness scores, and Step 3 explains why that averaging matters. messages.py
holds the message list:
messages.py
SYNTHETIC_MESSAGES = [
"Hey! Are you free Saturday for dinner? A few of us are getting together and it'd be great to see you.",
"Hi, I came across your background and I'm hiring for a senior role that looks like a strong fit. Open to a quick call this week?",
"It's been way too long! I'll be in town Thursday and Friday. Any chance you're around to grab coffee?",
"Just a reminder that your dentist appointment is Tuesday at 2pm. Reply to confirm or reschedule.",
"Hey neighbor, a package addressed to you was left at my door by mistake. Want to swing by this weekend?",
]
AI-likeness is the target, and it's the one judge that has to live in code, because it isn't an LLM. You score the reply with GPTZero, an AI detector. The score is 1 - P(human)
: near 0 when GPTZero reads the reply as human, which is your goal, and near 1 for AI or mixed text. Lower is better, so the bootstrap marks the judge inverted and the optimizer drives the score down.
GPTZero is also the more useful integration to learn, because it generalizes to any external scorer. The SDK gives a judge config no hook to reach outside code, so the config exists only so the optimization can attach a judge, its prompt stays a placeholder, and the real number comes from a small client you call yourself:
detector.py
def score_with_response(text: str):
data = _api_call(text) # POST the reply to GPTZero
doc = (data.get("documents") or [{}])[0]
return 1.0 - doc["class_probabilities"]["human"], data # 1 - P(human), + full JSON
score_with_response
returns both the number the judge gates on and the full GPTZero JSON, so the callback can forward the detail to the optimizer. Set AI_LIKENESS_API_KEY
to your GPTZero key.
Two things here cost me time. GPTZero sits behind Cloudflare, and a plain urllib
request comes back as a 403 with error code: 1010
before it reaches the API. That reads like a bad key, but it isn't. Sending any real User-Agent
header clears it. The score also comes from 1 - P(human)
rather than the average_generated_prob
field, which is the fraction of sentences flagged AI and reports 1.0
even on text the detector still classifies as human. Gating on that field punishes replies that already passed.
GPTZero is the only judge, so handle_judge_call
doesn't branch on the judge key. Every call pulls the batch of replies, scores each one with GPTZero, and averages:
optimize.py
async def handle_judge_call(key, config, context, is_evaluation=True):
text = _extract_candidate(context.user_input or "")
replies = [r for r in (json.loads(text) or {}).get("replies", [])
if isinstance(r, str) and r.strip()]
if not replies: # empty draft must FAIL the inverted gate
return OptimizationResponse(output=json.dumps(
{"score": 1.0, "rationale": "empty or degenerate candidate"}))
results = []
for reply in replies: # one GPTZero call per reply in the batch
s, raw = detector.score_with_response(reply) # 1 - P(human); + full JSON
results.append({"reply": reply, "ai_likeness": s, "gptzero": raw})
avg = round(sum(r["ai_likeness"] for r in results) / len(results), 4)
return OptimizationResponse(output=json.dumps(
{"score": avg, "rationale": f"Average AI-likeness = {avg}. {json.dumps(results)}"}))
Two choices in there are what make optimizing against a detector actually work.
Score a batch and average. A single short reply's GPTZero score is noisy. The very same prompt can produce a reply it calls 99% human on one message and 99% AI on the next. Each turn drafts replies to all the messages in one batch, and the judge averages those scores into something steady enough to optimize against.
Forward the whole detector response. The rationale you return goes straight to the model that writes the next prompt. Instead of a bare number, return the full GPTZero JSON, with its per-sentence probabilities and predicted class. The optimizer reads that directly and revises around whatever scored as AI, with no parsing on your side.
The empty-draft case is the one I got wrong first. An empty reply scores 0.0
, which the inverted gate reads as perfectly human, so an early version let the optimizer win by drafting nothing. Now an empty or malformed batch, or a detector error, returns 1.0
and fails the gate.
To build intuition before you run the loop, probe GPTZero on a draft by hand:
Terminal
uv run python gptzero_test.py "your draft reply"
A detector is a black box, so gate accordinglyA detection service gives you a defensible score immediately, with no model to train. It also has real limits. You can't tune it, it leans toward calling short LLM text AI, and it bills per call across every iteration. That confidence is the reason to average over a batch instead of gating on a single reply.
The saved optimization holds the judge, the threshold, the inputs, and the model choices, so the run command takes none of them. The agent drafts on Claude, the optimizer writes each new prompt on Claude too, and the threshold keeps the loop generating rather than picking a winner. Here is the saved optimization:
bootstrap.py (optimization)
{
"key": "email-agent-opt",
"aiConfigKey": "email-agent",
"maxAttempts": 10,
"judgeModel": "claude-haiku-4-5-20251001", # required by the API; the GPTZero judge never calls it
"modelChoices": ["claude-haiku-4-5-20251001"], # the Claude model the agent drafts with
"judges": [{"key": "ai-likeness", "threshold": 0.5}], # generator gate, not a winner test
"variableChoices": [ # interpolated into the instructions; the optimizer must use every one
{"sender_type": "friend", "respondent_name": "Jordan Lee", "messages": MESSAGES_BLOCK},
{"sender_type": "professional contact", "respondent_name": "Jordan Lee", "messages": MESSAGES_BLOCK},
],
"userInputOptions": ["Draft the replies now."], # trigger turn; messages come from {{messages}}
"autoCommit": True,
}
MESSAGES_BLOCK
is the message list from Step 2, formatted and fed in through {{messages}}
. respondent_name
and sender_type
are the other two variables, so replies come out signed and pitched to the right register. The optimizer has to use every variable you declare, which is what keeps {{messages}}
and the JSON envelope intact through every rewrite.
The threshold is 0.5
, and that number came from watching GPTZero, not from theory. A confident detector scores even clearly human-sounding short replies well above 0
, so a gate near 0
never trips and the loop never finds anything to keep. At 0.5
, this run passed at iteration 6 with 0.43
, while the earlier iterations landed between 0.54
and 1.00
. That gave the loop room to explore without rubber-stamping every candidate.
Run it from the saved config:
Terminal
OPTIMIZATION_KEY=email-agent-opt uv run python optimize_from_config.py
Settings live on the optimizationThresholds, inputs, models, and
maxAttempts
are baked intoemail-agent-opt
at bootstrap time. Change them by editing the optimization in the UI, or by deleting it and re-runningbootstrap.py
with the matching environment variables set. Committing the winner back as a variation needs the REST API key.
The command prints a link to the Results tab. One callback drafts replies on Claude, and the SDK reuses that same callback to write the next prompt, also on Claude. Each iteration posts its prompt and score to the Results tab as a candidate.
Open the Results tab from the printed link. Each iteration posts as it runs, with its candidate prompt and AI-likeness average alongside the variation the run currently recommends.
The Results tab after a passing run. Iteration 6 cleared the 0.50 gate at 0.43 and committed optimistic-coyote, with per-iteration charts for AI-likeness, latency, tokens, and cost.
Click any iteration to drill into its candidate prompt, the input it ran against, and the replies it produced. Iteration 1 is the baseline template itself, scoring 0.64
. Its replies already read decently ("Thanks for the invite! I'd love to come to dinner Saturday."), but the thin prompt left enough AI signal for the detector to flag.
Iteration 1, the baseline template: the JSON-and-{{messages}} instruction, the "Draft the replies now." trigger input, and the replies it produced. At 0.64 it didn't clear the gate.
The optimization sets autoCommit
, so on success the winner publishes back to email-agent
as a new variation. Open the agent Variations tab to read what the optimizer wrote.
The run committed a new variation, optimistic-coyote
. The Variations tab shows it next to the baseline, so you can read the change directly. The optimizer kept the JSON envelope and the {{sender_type}}
and {{messages}}
variables, and built a full humanization spec around them:
The Variations tab: the thin baseline above the committed winner optimistic-coyote, both on claude-haiku-4-5-20251001. The optimizer preserved the {{sender_type}} and {{messages}} variables and the JSON envelope while adding the humanization guidance.
Every line of it is a humanization lever, and they map onto how GPTZero separates the two classes:
{{respondent_name}}
instead of a placeholder.The optimizer took a prompt that said nothing about tone and built out a detailed spec, and GPTZero scored the resulting replies as more human.
That's one of the candidates the loop surfaced. To decide whether it's worth shipping, take it into a fuller offline eval for real data, which the sections below cover.
You can run all of this from the UI or from code. The New optimization form builds the same optimization by hand and streams to the same Results tab. This tutorial used optimize_from_config
, which runs a saved optimization from code while its judge, threshold, inputs, and models stay editable in the UI. To define everything in code instead, optimize_from_options
takes the settings directly and accepts more than one judge, and optimize_from_ground_truth_options
handles Expected Output mode when you have correct answers to match.
This demo leaves several controls unused: token_optimization
and latency_optimization
for a cost-and-latency pass, token_limit
for a spend cap, variation_key
, output_key
, context_choices
, and the on_turn
, on_passing_result
, on_failing_result
, and on_status_update
callbacks.
Offline evals and optimization sit next to each other in AgentControl and do opposite jobs. An offline eval measures a configuration you already have: you run your agent over a dataset, score it with judges, and answer "how good is this, and did anything regress?" Optimization runs the other direction, generating new variations until one clears that same judge. Running a surfaced candidate back through an eval is that measuring job again, telling you how it actually performs on a fuller set.
They're strongest together. Here's a path through AgentControl that gets the most out of both:
Run it long enough and production signals become the next round's eval data, so each optimization starts from what you actually saw in production rather than a guess.
You started with a prompt that said nothing about tone and let agent optimization rewrite it against GPTZero, then read the winning humanization spec off the Variations tab. The loop's job was to explore cheaply, and the trustworthy verdict comes from a proper offline eval over a comprehensive dataset.
Agent optimization is one step in a larger workflow. You define judges for what "better" means, optimize against them to surface candidates, then check those candidates with offline and online evals. What you learn in production feeds the next round. If you're getting started, Build a LangGraph multi-agent system is a good place to begin.
Sign up for LaunchDarkly and point the loop at your own prompts.