{"slug": "optimize-an-ai-agent-to-sound-human-judged-by-an-ai-detector", "title": "Optimize an AI agent to sound human, judged by an AI detector", "summary": "LaunchDarkly's agent optimization feature was used to automatically tune an AI email-drafting agent to sound human, judged by the AI detector GPTZero. The developer built a workflow where Claude drafts replies and generates prompt variations, while GPTZero scores AI-likeness, driving the optimizer to reduce the score. Each iteration costs about $0.002, and the reusable GPTZero integration can be adapted for other external scorers.", "body_md": "You can tell when an LLM wrote an email. The \"I hope this email finds you well\" opener, the three polite paragraphs answering a one-line question. I wanted a reply-drafting agent that didn't do that, and \"don't sound like an AI\" turned out to be hard to put in a prompt. Banning a few phrases is easy. The rest is judgment, and a single prompt that holds across a friendly dinner invite and a recruiter cold-email took more iterations than I'd guessed.\n\nThis is not only an email problem. Some platforms down-rank content that reads as AI-generated, so teams publishing at scale have a real stake in prose that clears a detector, even when a human wrote it. The workflow here applies to any of that.\n\nSo I stopped hand-tuning and let LaunchDarkly [agent optimization](https://launchdarkly.com/docs/home/agentcontrol) search for the prompt. You give it a judge that scores \"better,\" and it generates prompt variations and keeps the ones that beat the bar. For the reasoning behind the feature, read the [agent optimization announcement](https://launchdarkly.com/blog/agentcontrol-agent-optimization/). This tutorial is the how. If you don't have an account yet, [sign up for LaunchDarkly](https://app.launchdarkly.com/signup) to follow along.\n\nTwo pieces do the work here. Claude (`claude-haiku-4-5-20251001`\n\n) runs both roles: it drafts the replies, and it writes each new candidate prompt when the loop asks for one. Scoring comes from GPTZero, which isn't a language model at all but a closed AI detector. I wired it in inverted, so the score is the probability a reply reads as AI and the optimizer drives it down. I went with a detector instead of an LLM-as-a-judge for a reason: grading one model's prose by asking another model whether it sounds human is exactly the call language models are unreliable at, and a tool trained for that one question gives a number you can defend.\n\nA run is cheap. Each iteration costs around $0.002 and a few seconds, so a full run lands near a penny or two, and the loop tries variations I'd never sit down and type by hand.\n\nThis tutorial runs from a saved configYou bootstrap the agent, the judge, and the optimization, then work in the UI. Every iteration streams to the optimization Results tab, the winner lands on the agent Variations tab, and you tune thresholds and inputs on the optimization itself. The code does two things: it scores AI-likeness with GPTZero, and it runs the optimization.\n\n`email-agent`\n\n, seeded with one deliberately thin instruction`ai-likeness`\n\n, scored by GPTZero in code rather than by a prompt`email-agent-opt`\n\n, that runs as a candidate generator and streams to the Results tabThe GPTZero integration is the reusable part. The same shape works for any external scorer you might bring, whether a moderation API, a classifier you host, or a scoring endpoint of your own, so what you learn here isn't limited to email or to AI detection.\n\nThe companion repo is [agent-optimization-sample](https://github.com/launchdarkly-labs/agent-optimization-sample). Clone it to follow along.\n\n`.env`\n\nInstall the project and its dependencies:\n\n**Terminal**\n\n```\nuv sync\n# .env holds LD_SDK_KEY, LD_API_KEY, LD_PROJECT_KEY,\n#            ANTHROPIC_API_KEY, AI_LIKENESS_API_KEY\n```\n\nThe repo aliases the short `LD_*`\n\nnames to the `LAUNCHDARKLY_*`\n\nnames the SDK expects, so the short names in `.env`\n\nare enough.\n\nAgent optimization runs an iterative loop against an [AgentControl config](https://launchdarkly.com/docs/home/agentcontrol/create). It measures your current variation as a baseline, generates candidate variations, and scores each against your acceptance criteria. The loop has a simple shape:\n\n```\n┌─────────────┐     ┌──────────────┐     ┌─────────────┐\n│   Define    │────▶│   Explore    │────▶│   Commit    │\n│  \"Better\"   │     │  candidates  │     │  a winner   │\n└──────▲──────┘     └──────────────┘     └──────┬──────┘\n       │                                        │\n       └────────────────────────────────────────┘\n```\n\n**Define better.** You set acceptance criteria with a judge and a threshold. A judge scores a response on one dimension. You reference a [judge](https://launchdarkly.com/docs/home/agentcontrol/judges) saved as an AgentControl config by its key.\n\n**Explore candidates.** Each iteration drafts against your inputs, scores the result, and writes the next candidate from what the scores tell it. The threshold here is a gate that keeps the loop generating. When a candidate clears it, the optimizer re-runs that same prompt against a few more of your input samples and keeps it only if it passes those too, so a prompt that got lucky on one message doesn't win. Even then, clearing the gate doesn't certify that a candidate is good enough to ship. A fuller eval decides that, later.\n\n**Commit the winner.** The recommended variation shows up in LaunchDarkly, and with `autoCommit`\n\nit publishes back to the agent's [Variations tab](https://launchdarkly.com/docs/home/agentcontrol/create-variation) so you can read what the optimizer wrote.\n\nYou also pick an evaluation mode. **Exploratory** mode infers quality from the judge alone, which suits open-ended inputs that have no single correct output. **Expected Output** mode scores against known-correct answers. Replies have no single right answer, so this tutorial stays in Exploratory mode.\n\nThe technique behind itAgent optimization is an instance of OPRO (Optimization by PROmpting), introduced in Google DeepMind's\n\n[Large Language Models as Optimizers]. A model reads the history of prompts and their scores, then writes the next candidate to try. It searches over prompts, not model weights, so each candidate is cheap to run and nothing is ever trained.\n\nThe companion repo keeps the moving parts in small files, so each piece is easy to find and swap:\n\n`bootstrap.py`\n\n: seeds the three LaunchDarkly objects, the `email-agent`\n\nconfig, the `ai-likeness`\n\njudge, and the `email-agent-opt`\n\noptimization. It's safe to re-run, and it prints links to the configs and the Results tab.`optimize_from_config.py`\n\n: the one run command. It reads the saved optimization, runs it, streams each iteration to the Results tab, and prints the tab's link at the end.`optimize.py`\n\n: the two callbacks the run needs. `handle_agent_call`\n\ndrafts replies on Claude, and writes the next candidate prompt on Claude too when the SDK asks for one. `handle_judge_call`\n\nscores AI-likeness with GPTZero and hands the optimizer the per-reply detector output.`detector.py`\n\n: the GPTZero client. `score_with_response`\n\nreturns the number the judge gates on and the full GPTZero JSON.`messages.py`\n\n: the synthetic input messages.`gptzero_test.py`\n\n: a standalone probe for scoring a draft by hand.`clients.py`\n\nand `env.py`\n\n: the LaunchDarkly and Anthropic clients, built once each, and the `.env`\n\nloader.The saved optimization holds *what* you're optimizing for: the judge, the threshold, the inputs, and the model choices. The code holds *how* the work happens, drafting on Claude and scoring with GPTZero. You edit the what in the UI and the how in code, and the run command brings the two together.\n\nOne command seeds everything this tutorial needs. `bootstrap.py`\n\ncreates three objects in LaunchDarkly, and it's idempotent, so anything that already exists is left alone:\n\n`email-agent`\n\n: the agent `{\"replies\": [...]}`\n\nenvelope and the `{{messages}}`\n\nvariable, and nothing about tone. That's on purpose. It leaves the humanization, the part you want optimized, to the optimizer.`ai-likeness`\n\n: the inverted judge config. GPTZero scores it from code, which leaves the judge prompt as a placeholder.`email-agent-opt`\n\n: the saved optimization the Results tab runs. Thresholds, inputs, and model choices all live here.The baseline is thin on tone but carries the output contract the parser needs, on the Claude model the agent drafts with:\n\n**bootstrap.py (variation)**\n\n```\n{\n    \"key\": \"baseline\",\n    \"name\": \"Baseline\",\n    \"model\": {\"modelName\": \"claude-haiku-4-5-20251001\", \"parameters\": {}},\n    \"instructions\": (\n        \"You are an email assistant. Write a reply to each message below. \"\n        'Return ONLY a JSON object {\"replies\": [\"<reply to Message 1>\", ...]} '\n        \"with one reply per message, in order.\\n\\n{{messages}}\"\n    ),\n}\n```\n\nRun the bootstrap:\n\n**Terminal**\n\n```\nuv run python bootstrap.py\n```\n\nIt prints links to the two configs and the one command that runs the optimization.\n\nOne judge by designThe New optimization form in the UI attaches a single judge, so this tutorial uses one: AI-likeness. The bootstrap creates the same single-judge optimization in code, so the run matches what the UI supports. To build it by hand instead, open\n\nAgent optimization, thenNew optimization, target`email-agent`\n\n, add the`ai-likeness`\n\njudge, set the threshold and the input messages, and save.\n\nThe agent drafts against a fixed set of messages, injected into the prompt through the `{{messages}}`\n\nvariable. Keep them diverse on purpose, so a winning prompt generalizes instead of overfitting to one kind of message. The agent replies to all of them in one call and the judge averages the AI-likeness scores, and Step 3 explains why that averaging matters. `messages.py`\n\nholds the message list:\n\n**messages.py**\n\n```\nSYNTHETIC_MESSAGES = [\n    \"Hey! Are you free Saturday for dinner? A few of us are getting together and it'd be great to see you.\",\n    \"Hi, I came across your background and I'm hiring for a senior role that looks like a strong fit. Open to a quick call this week?\",\n    \"It's been way too long! I'll be in town Thursday and Friday. Any chance you're around to grab coffee?\",\n    \"Just a reminder that your dentist appointment is Tuesday at 2pm. Reply to confirm or reschedule.\",\n    \"Hey neighbor, a package addressed to you was left at my door by mistake. Want to swing by this weekend?\",\n    # 5 more, spanning tone, intent, and length\n]\n```\n\nAI-likeness is the target, and it's the one judge that has to live in code, because it isn't an LLM. You score the reply with [GPTZero](https://gptzero.me/), an AI detector. The score is `1 - P(human)`\n\n: near 0 when GPTZero reads the reply as human, which is your goal, and near 1 for AI or mixed text. Lower is better, so the bootstrap marks the judge inverted and the optimizer drives the score down.\n\nGPTZero is also the more useful integration to learn, because it generalizes to any external scorer. The SDK gives a judge config no hook to reach outside code, so the config exists only so the optimization can attach a judge, its prompt stays a placeholder, and the real number comes from a small client you call yourself:\n\n**detector.py**\n\n``` python\ndef score_with_response(text: str):\n    data = _api_call(text)                                    # POST the reply to GPTZero\n    doc = (data.get(\"documents\") or [{}])[0]\n    return 1.0 - doc[\"class_probabilities\"][\"human\"], data    # 1 - P(human), + full JSON\n```\n\n`score_with_response`\n\nreturns both the number the judge gates on and the full GPTZero JSON, so the callback can forward the detail to the optimizer. Set `AI_LIKENESS_API_KEY`\n\nto your GPTZero key.\n\nTwo things here cost me time. GPTZero sits behind Cloudflare, and a plain `urllib`\n\nrequest comes back as a 403 with `error code: 1010`\n\nbefore it reaches the API. That reads like a bad key, but it isn't. Sending any real `User-Agent`\n\nheader clears it. The score also comes from `1 - P(human)`\n\nrather than the `average_generated_prob`\n\nfield, which is the fraction of sentences flagged AI and reports `1.0`\n\neven on text the detector still classifies as human. Gating on that field punishes replies that already passed.\n\nGPTZero is the only judge, so `handle_judge_call`\n\ndoesn't branch on the judge key. Every call pulls the batch of replies, scores each one with GPTZero, and averages:\n\n**optimize.py**\n\n``` python\nasync def handle_judge_call(key, config, context, is_evaluation=True):\n    text = _extract_candidate(context.user_input or \"\")\n    replies = [r for r in (json.loads(text) or {}).get(\"replies\", [])\n               if isinstance(r, str) and r.strip()]\n    if not replies:                              # empty draft must FAIL the inverted gate\n        return OptimizationResponse(output=json.dumps(\n            {\"score\": 1.0, \"rationale\": \"empty or degenerate candidate\"}))\n    results = []\n    for reply in replies:                        # one GPTZero call per reply in the batch\n        s, raw = detector.score_with_response(reply)        # 1 - P(human); + full JSON\n        results.append({\"reply\": reply, \"ai_likeness\": s, \"gptzero\": raw})\n    avg = round(sum(r[\"ai_likeness\"] for r in results) / len(results), 4)\n    # The gate is the AVERAGE; the full per-reply GPTZero JSON becomes the rationale.\n    return OptimizationResponse(output=json.dumps(\n        {\"score\": avg, \"rationale\": f\"Average AI-likeness = {avg}. {json.dumps(results)}\"}))\n```\n\nTwo choices in there are what make optimizing against a detector actually work.\n\n**Score a batch and average.** A single short reply's GPTZero score is noisy. The very same prompt can produce a reply it calls 99% human on one message and 99% AI on the next. Each turn drafts replies to all the messages in one batch, and the judge averages those scores into something steady enough to optimize against.\n\n**Forward the whole detector response.** The rationale you return goes straight to the model that writes the next prompt. Instead of a bare number, return the full GPTZero JSON, with its per-sentence probabilities and predicted class. The optimizer reads that directly and revises around whatever scored as AI, with no parsing on your side.\n\nThe empty-draft case is the one I got wrong first. An empty reply scores `0.0`\n\n, which the inverted gate reads as perfectly human, so an early version let the optimizer win by drafting nothing. Now an empty or malformed batch, or a detector error, returns `1.0`\n\nand fails the gate.\n\nTo build intuition before you run the loop, probe GPTZero on a draft by hand:\n\n**Terminal**\n\n```\nuv run python gptzero_test.py \"your draft reply\"\n```\n\nA detector is a black box, so gate accordinglyA detection service gives you a defensible score immediately, with no model to train. It also has real limits. You can't tune it, it leans toward calling short LLM text AI, and it bills per call across every iteration. That confidence is the reason to average over a batch instead of gating on a single reply.\n\nThe saved optimization holds the judge, the threshold, the inputs, and the model choices, so the run command takes none of them. The agent drafts on Claude, the optimizer writes each new prompt on Claude too, and the threshold keeps the loop generating rather than picking a winner. Here is the saved optimization:\n\n**bootstrap.py (optimization)**\n\n```\n{\n    \"key\": \"email-agent-opt\",\n    \"aiConfigKey\": \"email-agent\",\n    \"maxAttempts\": 10,\n    \"judgeModel\": \"claude-haiku-4-5-20251001\",       # required by the API; the GPTZero judge never calls it\n    \"modelChoices\": [\"claude-haiku-4-5-20251001\"],   # the Claude model the agent drafts with\n    \"judges\": [{\"key\": \"ai-likeness\", \"threshold\": 0.5}],   # generator gate, not a winner test\n    \"variableChoices\": [   # interpolated into the instructions; the optimizer must use every one\n        {\"sender_type\": \"friend\", \"respondent_name\": \"Jordan Lee\", \"messages\": MESSAGES_BLOCK},\n        {\"sender_type\": \"professional contact\", \"respondent_name\": \"Jordan Lee\", \"messages\": MESSAGES_BLOCK},\n    ],\n    \"userInputOptions\": [\"Draft the replies now.\"],  # trigger turn; messages come from {{messages}}\n    \"autoCommit\": True,\n}\n```\n\n`MESSAGES_BLOCK`\n\nis the message list from Step 2, formatted and fed in through `{{messages}}`\n\n. `respondent_name`\n\nand `sender_type`\n\nare the other two variables, so replies come out signed and pitched to the right register. The optimizer has to use every variable you declare, which is what keeps `{{messages}}`\n\nand the JSON envelope intact through every rewrite.\n\nThe threshold is `0.5`\n\n, and that number came from watching GPTZero, not from theory. A confident detector scores even clearly human-sounding short replies well above `0`\n\n, so a gate near `0`\n\nnever trips and the loop never finds anything to keep. At `0.5`\n\n, this run passed at iteration 6 with `0.43`\n\n, while the earlier iterations landed between `0.54`\n\nand `1.00`\n\n. That gave the loop room to explore without rubber-stamping every candidate.\n\nRun it from the saved config:\n\n**Terminal**\n\n```\nOPTIMIZATION_KEY=email-agent-opt uv run python optimize_from_config.py\n```\n\nSettings live on the optimizationThresholds, inputs, models, and\n\n`maxAttempts`\n\nare baked into`email-agent-opt`\n\nat bootstrap time. Change them by editing the optimization in the UI, or by deleting it and re-running`bootstrap.py`\n\nwith the matching environment variables set. Committing the winner back as a variation needs the REST API key.\n\nThe command prints a link to the Results tab. One callback drafts replies on Claude, and the SDK reuses that same callback to write the next prompt, also on Claude. Each iteration posts its prompt and score to the Results tab as a candidate.\n\nOpen the Results tab from the printed link. Each iteration posts as it runs, with its candidate prompt and AI-likeness average alongside the variation the run currently recommends.\n\n*The Results tab after a passing run. Iteration 6 cleared the 0.50 gate at 0.43 and committed optimistic-coyote, with per-iteration charts for AI-likeness, latency, tokens, and cost.*\n\nClick any iteration to drill into its candidate prompt, the input it ran against, and the replies it produced. Iteration 1 is the baseline template itself, scoring `0.64`\n\n. Its replies already read decently (\"Thanks for the invite! I'd love to come to dinner Saturday.\"), but the thin prompt left enough AI signal for the detector to flag.\n\n*Iteration 1, the baseline template: the JSON-and-{{messages}} instruction, the \"Draft the replies now.\" trigger input, and the replies it produced. At 0.64 it didn't clear the gate.*\n\nThe optimization sets `autoCommit`\n\n, so on success the winner publishes back to `email-agent`\n\nas a new variation. Open the agent **Variations** tab to read what the optimizer wrote.\n\nThe run committed a new variation, `optimistic-coyote`\n\n. The **Variations** tab shows it next to the baseline, so you can read the change directly. The optimizer kept the JSON envelope and the `{{sender_type}}`\n\nand `{{messages}}`\n\nvariables, and built a full humanization spec around them:\n\n*The Variations tab: the thin baseline above the committed winner optimistic-coyote, both on claude-haiku-4-5-20251001. The optimizer preserved the {{sender_type}} and {{messages}} variables and the JSON envelope while adding the humanization guidance.*\n\nEvery line of it is a humanization lever, and they map onto how GPTZero separates the two classes:\n\n`{{respondent_name}}`\n\ninstead of a placeholder.The optimizer took a prompt that said nothing about tone and built out a detailed spec, and GPTZero scored the resulting replies as more human.\n\nThat's one of the candidates the loop surfaced. To decide whether it's worth shipping, take it into a fuller [offline eval](https://launchdarkly.com/docs/home/agentcontrol/offline-evaluations) for real data, which the sections below cover.\n\nYou can run all of this from the UI or from code. The **New optimization** form builds the same optimization by hand and streams to the same Results tab. This tutorial used `optimize_from_config`\n\n, which runs a saved optimization from code while its judge, threshold, inputs, and models stay editable in the UI. To define everything in code instead, `optimize_from_options`\n\ntakes the settings directly and accepts more than one judge, and `optimize_from_ground_truth_options`\n\nhandles Expected Output mode when you have correct answers to match.\n\nThis demo leaves several controls unused: `token_optimization`\n\nand `latency_optimization`\n\nfor a cost-and-latency pass, `token_limit`\n\nfor a spend cap, `variation_key`\n\n, `output_key`\n\n, `context_choices`\n\n, and the `on_turn`\n\n, `on_passing_result`\n\n, `on_failing_result`\n\n, and `on_status_update`\n\ncallbacks.\n\nOffline evals and optimization sit next to each other in AgentControl and do opposite jobs. An [offline eval](https://launchdarkly.com/docs/home/agentcontrol/offline-evaluations) measures a configuration you already have: you run your agent over a [dataset](https://launchdarkly.com/docs/home/agentcontrol/datasets), score it with [judges](https://launchdarkly.com/docs/home/agentcontrol/judges), and answer \"how good is this, and did anything regress?\" Optimization runs the other direction, generating new variations until one clears that same judge. Running a surfaced candidate back through an eval is that measuring job again, telling you how it actually performs on a fuller set.\n\nThey're strongest together. Here's a path through AgentControl that gets the most out of both:\n\nRun it long enough and production signals become the next round's eval data, so each optimization starts from what you actually saw in production rather than a guess.\n\nYou started with a prompt that said nothing about tone and let agent optimization rewrite it against [GPTZero](https://gptzero.me/), then read the winning humanization spec off the Variations tab. The loop's job was to explore cheaply, and the trustworthy verdict comes from a proper [offline eval](https://launchdarkly.com/docs/home/agentcontrol/offline-evaluations) over a comprehensive [dataset](https://launchdarkly.com/docs/home/agentcontrol/datasets).\n\n[Agent optimization](https://launchdarkly.com/docs/home/agentcontrol) is one step in a larger workflow. You define [judges](https://launchdarkly.com/docs/home/agentcontrol/judges) for what \"better\" means, optimize against them to surface candidates, then check those candidates with [offline](https://launchdarkly.com/docs/tutorials/offline-evals) and [online evals](https://launchdarkly.com/docs/tutorials/when-to-add-online-evals). What you learn in production feeds the next round. If you're getting started, [Build a LangGraph multi-agent system](https://launchdarkly.com/docs/tutorials/agents-langgraph) is a good place to begin.\n\n[Sign up for LaunchDarkly](https://app.launchdarkly.com/signup) and point the loop at your own prompts.", "url": "https://wpnews.pro/news/optimize-an-ai-agent-to-sound-human-judged-by-an-ai-detector", "canonical_source": "https://dev.to/launchdarkly/optimize-an-ai-agent-to-sound-human-judged-by-an-ai-detector-42po", "published_at": "2026-08-03 17:56:04+00:00", "updated_at": "2026-08-03 18:11:39.619746+00:00", "lang": "en", "topics": ["ai-agents", "ai-products", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["LaunchDarkly", "Claude", "GPTZero", "AgentControl", "agent-optimization-sample"], "alternates": {"html": "https://wpnews.pro/news/optimize-an-ai-agent-to-sound-human-judged-by-an-ai-detector", "markdown": "https://wpnews.pro/news/optimize-an-ai-agent-to-sound-human-judged-by-an-ai-detector.md", "text": "https://wpnews.pro/news/optimize-an-ai-agent-to-sound-human-judged-by-an-ai-detector.txt", "jsonld": "https://wpnews.pro/news/optimize-an-ai-agent-to-sound-human-judged-by-an-ai-detector.jsonld"}}