{"slug": "ai-coding-agents-still-write-your-sdk-s-old-api-so-i-built-a-type-checker-to-it", "title": "AI coding agents still write your SDK's old API — so I built a type-checker to measure it", "summary": "A developer built SDKProof, a tool that uses TypeScript's compiler (tsc --noEmit) to measure how often AI coding agents write code against outdated library APIs. Testing Claude Opus 4 on three SDKs, it found scores of 80/100 for Prisma 7, 90/100 for Vercel AI SDK 7, and 90/100 for Zod 4, with errors stemming from renamed options like 'parameters' to 'inputSchema'.", "body_md": "Here's a bug I kept hitting. I'd ask an AI assistant to write some code against a library — Prisma, the Vercel AI SDK, Zod — and the code would look completely right. Clean, idiomatic, exactly the shape I expected. Then I'd run it, and it wouldn't compile.\n\nThe reason was always the same: the library had shipped a new major version, and the model was writing the *previous* major's API from memory. `parameters`\n\ninstead of `inputSchema`\n\n. `required_error`\n\ninstead of `error`\n\n. A `new PrismaClient({ datasources })`\n\ncall that no longer exists. Small things — but enough to break the build on the first try.\n\nModels are frozen at their training cutoff. Libraries are not. So there's a gap between \"the API the model reaches for\" and \"the API you actually have installed\" — and that gap is widest right after a library ships a breaking change.\n\nI wanted to know: **how big is that gap, exactly — and can I measure it objectively?** So I built [SDKProof](https://sdkproof.dev).\n\nThe trick to measuring this without hand-waving is to *not* use an LLM to grade an LLM. Instead:\n\n`tsc --noEmit`\n\n.No LLM judge. No \"looks plausible.\" The installed package's type definitions are the ground truth, and `tsc`\n\nis the referee. A pass means the code would actually build against the version you have.\n\nOne design detail that matters: the prompts name the *functions* but never the *option names*. I ask for \"a tool with a description and an input schema,\" not \"use `inputSchema`\n\n.\" That way I'm measuring what the model naturally reaches for — not whether it can echo back a name I already handed it.\n\nRunning `claude-opus-4-8`\n\nacross three SDKs:\n\n| SDK | Package | Score | Where it breaks |\n|---|---|---|---|\n| Prisma 7 | `@prisma/client` |\n80 / 100 |\nstill writes removed v6 setup — `new PrismaClient()` with `datasources` , `$use` middleware |\n| Vercel AI SDK 7 | `ai` |\n90 / 100 |\nold tool wiring — `parameters` (now `inputSchema` ), removed `maxSteps`\n|\n| Zod 4 | `zod` |\n90 / 100 |\nremoved `required_error` (now `error` ) — but nails the new 2-arg `z.record()`\n|\n\nHere's a representative miss. Ask for a tool definition with the AI SDK and you tend to get this:\n\n``` js\nimport { tool } from 'ai';\nimport { z } from 'zod';\n\nconst weatherTool = tool({\n  description: 'Get the weather for a city',\n  parameters: z.object({ city: z.string() }), // ✗ this was the v4 API\n  execute: async ({ city }) => getWeather(city),\n});\n```\n\nLooks right. But against `ai`\n\nv7, `tsc`\n\nsays:\n\n```\nerror TS2353: Object literal may only specify known properties,\nand 'parameters' does not exist in type 'Tool<...>'.\n```\n\nBecause the option was renamed to `inputSchema`\n\n:\n\n``` js\nconst weatherTool = tool({\n  description: 'Get the weather for a city',\n  inputSchema: z.object({ city: z.string() }), // ✓ v5+\n  execute: async ({ city }) => getWeather(city),\n});\n```\n\nEverything else about the code is fine. It's one renamed key — and it's exactly the kind of thing that slips past a quick read but stops the build cold.\n\nThe interesting part isn't any single score — it's *why* they differ.\n\nNotice the newest breaking change scores worst. The Vercel AI SDK and Zod shipped their renames in 2025; by now the model has largely absorbed them and mostly gets them right (90/100). Prisma 7 is more recent, and the model hasn't caught up (80/100) — it still writes setup calls that were removed.\n\nThat's the whole thesis: **a model's \"readiness\" for an SDK tracks how recently that SDK changed.** Which means this isn't a one-time audit. The gap:\n\nSo it's something to *monitor*, not measure once. A library that scores 95 today can drop to 70 the week it ships v-next — then climb back as the next model generation learns it.\n\nI'd rather you trust the number than oversell it:\n\nNone of that breaks the signal — it just scopes it. \"Does the model reach for API surface that still exists?\" is a real, useful question, and the compiler answers it without opinion.\n\nAdding an SDK is about an afternoon: install the package, add a `tsconfig`\n\n, write a tasks file, register it. The harness handles generation, type-checking, and scoring.\n\nI'm looking for the next SDKs to score. **What library have you watched an AI assistant get wrong since its last major?** Tell me and I'll run it.", "url": "https://wpnews.pro/news/ai-coding-agents-still-write-your-sdk-s-old-api-so-i-built-a-type-checker-to-it", "canonical_source": "https://dev.to/kalpitrathore/ai-coding-agents-still-write-your-sdks-old-api-so-i-built-a-type-checker-to-measure-it-alk", "published_at": "2026-07-20 19:32:26+00:00", "updated_at": "2026-07-20 20:02:10.507179+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools", "large-language-models", "ai-agents"], "entities": ["SDKProof", "Prisma", "Vercel AI SDK", "Zod", "Claude Opus 4"], "alternates": {"html": "https://wpnews.pro/news/ai-coding-agents-still-write-your-sdk-s-old-api-so-i-built-a-type-checker-to-it", "markdown": "https://wpnews.pro/news/ai-coding-agents-still-write-your-sdk-s-old-api-so-i-built-a-type-checker-to-it.md", "text": "https://wpnews.pro/news/ai-coding-agents-still-write-your-sdk-s-old-api-so-i-built-a-type-checker-to-it.txt", "jsonld": "https://wpnews.pro/news/ai-coding-agents-still-write-your-sdk-s-old-api-so-i-built-a-type-checker-to-it.jsonld"}}