{"slug": "migrating-a-production-ai-agent-to-gpt-5-6", "title": "Migrating a production AI agent to GPT 5.6", "summary": "Ploy, an AI agent that builds marketing websites, has migrated its production model from Claude Opus 4.8 to OpenAI's GPT-5.6 Sol after extensive evaluation. The switch resulted in builds finishing in less than half the time at 27% lower cost while maintaining or improving quality scores. The migration required fixing eval harness assumptions and adapting to provider-specific behaviors like parallel tool calls and caching.", "body_md": "As of today, Ploy’s agent runs on GPT-5.6 Sol, the flagship tier of the model family OpenAI released this morning. For months, we couldn’t find a model that challenges Claude Opus given our incredibly high bar for quality. That changed with GPT 5.6 Sol. After running it head-to-head against Claude Opus, we’ve made GPT 5.6 Sol the default model powering every Ploy workspace.\n\nThat’s a bigger switch than it sounds. Ploy’s agent builds and edits real marketing websites. It plans a page, reads the codebase, writes components, generates imagery, screenshots its own work, and decides when it’s done. That job description sets a very high bar for a model, and we test every frontier release against it. For the four months Opus held the default slot (first Opus 4.7, then 4.8), nothing we tested beat it. GPT-5.6 is the first model that did.\n\nNot that the first eval run was perfect. It had real failure modes, which we’ll show you. But it did extremely well, and the promise was immediate and specific: builds finishing in less than half the wall-clock time, at 27% lower cost, scoring at or above our incumbent on completed work. Numbers like that buy a model a real migration effort.\n\nDespite using Vercel’s [AI SDK](https://ai-sdk.dev/docs/introduction), a universal LLM SDK, switching from Claude Opus 4.8 to GPT 5.6 Sol required discovering, one eval failure at a time, that the things we think of as “the model” are provider-specific behaviors our whole stack has quietly specialized around: how it fills in tool arguments, how its prompt cache works, how it replays its own reasoning between turns.\n\nHere’s what it took: fix the eval harness, then the tool schemas, then caching, then reasoning replay.\n\n## Step 0: Fix your harness before you trust a single number\n\nOur eval suite runs the real agent against real fixture workspaces. Hundreds of cases, from “build a homepage from scratch” to “is this clone request safe to execute.” Build cases are scored by a visual judge running binary checks against a reference design, ten yes/no questions like *“the hero is a full-bleed photographic scene”* or *“primary CTAs are rounded rectangles, not pills”*, plus content checks, tool-trajectory checks, and file assertions. Every failed case gets triaged against its full trace: the actual tool calls and model text, not just the score.\n\nRunning that suite across two model families surprised us more than any individual result:\n\n**Your harness is tuned to your incumbent model, and you don’t know it.** Our tool-call budgets were sized for Opus’s sequential style; GPT-5.6 fans out parallel calls and blew through them on cases it was solving correctly. Our eval executor didn’t support batched file reads, which Opus rarely used and GPT-5.6 uses constantly. Roughly a third of the raw failures in the first cross-model run traced back to harness assumptions, not model behavior, and they were not evenly distributed between the models. If you’re evaluating a challenger model against an incumbent, **triage the traces before you trust the pass rate**. Otherwise you’re grading the new model on how well it imitates the old one.\n\n**Make sure you’re grading models fairly in evals.** A dataset that omitted its `minScore`\n\nthreshold silently inherited a default of 1.0, so GPT-5.6 “failed” a hero it scored 0.98 on, and Opus “failed” a case while passing every individual check. Two defensible design directions; one invisible threshold.\n\n## First impression: immediately promising\n\nWith the harness cleaned up, here’s a sample from our redesign suite, where the agent rebuilds a brand’s homepage against a reference design:\n\n| Mean per completed build | Claude Opus 4.8 (n=11) | GPT-5.6 (n=10) |\n|---|---|---|\n| Cost | $3.06 | $2.22 |\n| Wall-clock time | 8m 00s | 3m 42s |\n| Input tokens | 2.60M | 1.70M |\n| Output tokens | 33.0K | 17.1K |\n| Visual score | 0.936 | 0.970 |\n\nThis is the shape of the promise: 2.2× faster to a finished page, 27% cheaper, and about half the output tokens. GPT-5.6 writes lean code. On one matched pair, Opus produced a 17,957-character `globals.css`\n\nwith 174 CSS variables (full color ramps, mostly unused) where GPT-5.6 wrote 2,508 characters and 45 variables for a comparable (and sometimes better) rendered page.\n\n### Claude Opus 4.8\n\n### GPT-5.6 Sol\n\n### Design: sharp, clean, but a little bit uniform\n\nOur overall read on GPT-5.6’s design work: it is *very* good at clean, modern, tightly-gridded layouts, but it tends to converge towards that look unless you steer it well. With our older harness designed for Opus 4.8, GPT 5.6 Sol tends to ignore existing design systems and instead produces sharp, restrained, and visibly generic output.\n\nThe details of how we fixed this are worth a separate blog post of its own. With the expertise of our design and engineering teams, we are able to steer models to achieve world-class brand adherence that you can’t get out of the box.\n\n## Step 1: Check your tool calls\n\nHere’s the one that was silently corrupting results before we caught it.\n\nOur agent’s `code`\n\ntool has 25 top-level parameters, one required (`action`\n\n) and the rest optional. Claude sends the two or three it’s using and omits the rest. GPT-5.6 sends **all 25, every time**, inventing plausible values for the ones it doesn’t need: `offset: 0`\n\n, `timeout: 120000`\n\n, `siteId: \"00000000-0000-0000-0000-000000000000\"`\n\n.\n\nThree days of production traces, `code(read)`\n\ncalls carrying every property:\n\n| Model | Calls | Carrying all 25 properties |\n|---|---|---|\n| gpt-5.6 | 6,635 | 6,635 (100%) |\n| claude-opus-4.8 | 2,898 | 4 (0.1%) |\n| claude-sonnet-5 | 1,933 | 0 |\n\nThe problem isn’t verbosity. It’s that **an invented value is indistinguishable from an intended one**. `offset: 0`\n\nlooks like a real argument. Our file-read implementation treated it as one, and 52% to 64% of GPT-5.6’s file reads were coming back empty because of it. The tool returned `success: true`\n\nboth ways, so the model had no way to know it was reading blank files. It just did the work worse, with more calls.\n\nPrompting doesn’t fix this. A tool-description directive to “omit unused parameters”: still 25/25. Per-property “OPTIONAL, omit if unused” hints: still 25/25. OpenAI’s `strict`\n\nmode: identical behavior (we measured), and adopting it would have forced us to strip `pattern`\n\n, `format`\n\n, and array-bound validation from every schema. This is baked into how the model emits [function calls](https://developers.openai.com/api/docs/guides/function-calling). You don’t instruct it away; you design around it.\n\nThe fix that worked is a schema transform at the provider boundary. For OpenAI-family models only, we rewrite every optional property to be **required but nullable**, using `anyOf: [T, null]`\n\n, which gives the model an explicit way to say “not using this.” Then, at the single seam every tool invocation passes through, we strip the nulls back out before validation, so no tool implementation changes at all. Round trip: the model sees a schema where honesty is expressible; the tools see the same inputs they always did.\n\n```\n// Before: 25 keys, every one carrying an invented value\n{ \"action\": \"read\", \"file_paths\": [...], \"offset\": 0, \"timeout\": 120000, ... }\n\n// After: 25 keys, 4 real values, 21 explicit nulls (stripped before the tool runs)\n{ \"action\": \"read\", \"file_paths\": [...], \"offset\": null, \"timeout\": null, ... }\n```\n\nResults: empty file reads went from 52% to 0%, and the agent needed roughly 30% fewer tool calls for the same work, because it was no longer re-reading files that came back blank.\n\n## Step 2: Rebuild prompt caching\n\nThis was the most instructive engineering difference, because on the surface both providers offer “prompt caching” and the words hide two entirely different designs. If you migrate one thing carefully, make it this: before we did, GPT-5.6 looked about 50% more expensive than Opus. It wasn’t the model’s pricing; it was our cache configuration.\n\nOur agent’s prompt opens with a static prefix of roughly 29K tokens (tool schemas plus the core system prompt) that’s identical for every conversation. On Claude, we mark cache breakpoints with `cache_control`\n\nand that prefix caches **across the whole organization**: any conversation, any workspace, one shared entry, no throughput budget to think about. Cache hit rates run 92% to 96% and caching fades into the background.\n\nGPT-5.6 changed OpenAI’s caching model out from under us. Earlier GPT models cached implicitly on partial prefix matches, which gave decent hit rates for free. GPT-5.6 [dropped partial-prefix matching](https://developers.openai.com/api/docs/guides/prompt-caching): implicit caching now only creates whole-prompt entries keyed on the latest message. A *new* conversation sharing our 29K static prefix cached **0%** of it. Every conversation re-billed the full prefix at the uncached rate, and on GPT-5.6 every uncached prompt also pays a 1.25× cache-*write* surcharge, whether or not you use caching.\n\nThe intended mechanism is explicit: `prompt_cache_breakpoint`\n\nmarkers plus a mandatory `prompt_cache_key`\n\n. And the key is where the design really diverges, because it’s part of cache identity. Identical prompt, different key: zero cache hits. Each key maps to a cache node that sustains roughly **15 requests per minute** before OpenAI fans traffic to other nodes with independent, cold caches.\n\nThat turns “enable caching” into an actual design decision: what entity do you scope the key to?\n\n**Per-conversation key** means a new conversation never hits the shared prefix. First-call hit rate: 0%. (We measured this mistake. It’s expensive.)**One global key** means every request hashes to one cache node, and production traffic obliterates the 15 rpm budget; requests spill to cold nodes and you’re back to misses.**Per-workspace key** is the sweet spot. All conversations in a customer workspace share entries; per-key traffic stays low.\n\nWe ship the workspace-scoped key and split the system prompt into breakpointed layers, mirroring the structure we already used for Anthropic:\n\n```\nrequest ──► hash(prompt head + prompt_cache_key) ──► cache node (~15 req/min per key)\n                                                          │\n   ┌──────────────────────────────────────────────────────┴───────────────┐\n   │  entries on the node, all namespaced by key ws:{workspaceId}         │\n   │                                                                      │\n   │   [ tools + static prefix ]······················ A  every session   │\n   │   [ tools + static prefix + workspace context ]·· B  same context    │\n   │   [ ····················· + turn 1 + … + latest ] C  this session    │\n   └──────────────────────────────────────────────────────────────────────┘\n```\n\nEntry A is what makes a session’s *first* call cheap. Entry B self-heals: when workspace memory changes, the request misses B but still hits A, then writes a fresh B. One context-sized write instead of a full 29K re-bill. Entry C is OpenAI’s implicit whole-prompt chain, which works fine within a session because our prompts are strictly append-only.\n\nOne consequence has no workaround: **cross-workspace sharing of the static prefix is structurally impossible on OpenAI.** Anthropic can share it because its cache is org-scoped without key partitioning. On GPT-5.6, every workspace pays one 29K cold write per idle window, about $0.18. A real cost, but bounded and predictable.\n\nResults after the change: first-call cache hits went from roughly 0% to **83.7%**, total uncached input tokens dropped 28%, and GPT-5.6’s per-suite cost landed *below* Opus’s. Every dollar of the gap we’d been staring at was cache misconfiguration, not model pricing. If you’re cost-comparing models and one of them has a cold cache, you are comparing your config, not the models.\n\n## Step 3: Make reasoning replay self-contained\n\nShorter, but it broke real conversations. GPT-5.6’s Responses API replays prior-turn reasoning as server-side item references by default; ours started intermittently failing mid-conversation with `Item 'rs_...' not found`\n\n. The fix is `store: false`\n\n, which makes the SDK request [encrypted reasoning content](https://developers.openai.com/api/docs/guides/reasoning) and replay self-contained blobs instead of pointers to server state. A corollary that cost us a debugging afternoon: with server-side reasoning state in the loop, the effective prompt can change upstream of you even when the bytes you send are append-only.\n\n## GPT 5.6 Sol is ready to Ploy\n\n**Try it yourself.** GPT-5.6 launched today, and it’s already live on Ploy. You can try it for free right now: give it a website to build and see what a sub-four-minute build looks like. [Start free at ploy.ai](https://ploy.ai/auth/sign-up).\n\n*Ploy is marketing run on autopilot: an AI layer that plans, builds, publishes, and optimizes your website and campaigns end-to-end. If debugging cache-node fan-out at 2am sounds like fun, we’re hiring.*", "url": "https://wpnews.pro/news/migrating-a-production-ai-agent-to-gpt-5-6", "canonical_source": "https://ploy.ai/blog/migrating-a-production-ai-agent-to-gpt-5-6", "published_at": "2026-07-10 20:40:34+00:00", "updated_at": "2026-07-10 21:06:02.420353+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "large-language-models", "ai-products", "ai-tools"], "entities": ["Ploy", "OpenAI", "GPT-5.6 Sol", "Claude Opus", "Vercel", "AI SDK"], "alternates": {"html": "https://wpnews.pro/news/migrating-a-production-ai-agent-to-gpt-5-6", "markdown": "https://wpnews.pro/news/migrating-a-production-ai-agent-to-gpt-5-6.md", "text": "https://wpnews.pro/news/migrating-a-production-ai-agent-to-gpt-5-6.txt", "jsonld": "https://wpnews.pro/news/migrating-a-production-ai-agent-to-gpt-5-6.jsonld"}}