Wrong answers were the rarest way AI features broke. I went through 480 public reports of AI features failing inside real apps, from OpenAI's developer forum, Reddit and Hacker News, and sorted each by its cause. Only 29 were about the AI saying something wrong. The rest were slow calls, answers the app couldn't read, billing limits, surprise bills, model changes and leaked keys.
That's good news if you're working out how to integrate AI into an app. Wrong answers are hard to fix. The mechanical failures are not: each one has a known safeguard, and most take an afternoon to add.
I build AI into products for a living. I built an AI research tool that cut hours of reading SEC filings to under 5 minutes, one of my own products runs a daily AI job, and in the Trust Identity Protocol I shipped a classifier pipeline that scans content in the background so it never slows down the user. This is the setup I use, in the order I add it.
Under the hood, adding AI to an app is one network call: your app sends text (and sometimes images) to a model from OpenAI, Anthropic or Google, and gets text back. You pay per token, roughly per word, in both directions. Everything else in this post is about where that call happens and what you check around it.
The shape that holds up is five stops, and each safeguard lives at one of them:
Each safeguard lines up with a group in the 480 reports. The AI drafts; a person decides.
The reports run from January 2025 to October 2026. Most come from the OpenAI developer forum, where people post problems with their own integrations, plus Reddit and Hacker News. I kept only cases of an AI feature inside someone's app or website, not people using ChatGPT itself, and grouped them by the cause the thread gave, or by the main symptom when it gave none.
Wrong answers were only 29. Most failures were mechanical: slow calls, unreadable output, billing limits and model changes.
Wrong answers and retrieval problems matter, and I wrote a separate guide on why AI chatbots give wrong answers. But if you only test whether the answers are good, you'll miss most of what breaks in production.
The most expensive reports in the set were all leaked keys. One developer posted on Hacker News about $82,314 in 48 hours from a Gemini key that had been uploaded to GitHub, against a normal spend of $180 a month. Another got a €54,000 bill in 13 hours from a browser-side Firebase key that had no restrictions and could call Gemini. On the OpenAI forum, a key shipped inside a published VS Code extension quietly drained someone's credits.
The rule is simple: the browser never holds an AI key. Anything sent to a browser or bundled into a mobile app can be read by whoever uses it, which Google's Gemini key guide says plainly, and OpenAI's official Node library refuses to run in a browser unless you switch that protection off. Anthropic's key safety guide compares a leaked API key to a leaked credit card number.
Instead, your app calls a route on your own server, and only that route talks to the AI provider. The key lives in the server's environment variables, never in the code or the repo. This is a minimal version in Next.js:
// app/api/draft/route.ts: runs on your server, never in the browser
import OpenAI from 'openai';
import { zodTextFormat } from 'openai/helpers/zod';
import { z } from 'zod';
const openai = new OpenAI({ timeout: 30_000, maxRetries: 2 }); // key from OPENAI_API_KEY
const Draft = z.object({ subject: z.string().max(120), body: z.string().max(2000) });
export async function POST(req: Request) {
const { message } = await req.json();
const res = await openai.responses.create({
model: process.env.AI_MODEL!, // a dated version, not a "latest" alias
input: [{ role: 'user', content: String(message).slice(0, 4000) }],
max_output_tokens: 800,
text: { format: zodTextFormat(Draft, 'draft') }, // strict structured output
});
const parsed = Draft.safeParse(JSON.parse(res.output_text));
if (!parsed.success) return Response.json({ error: 'try again' }, { status: 502 });
return Response.json({ draft: parsed.data, status: 'needs_approval' });
}
It also needs a login check and a per-user limit, which depend on your app. The point is the shape: the key, the limits and the checks all sit on your side.
Fifty-three reports were surprise bills and 64 more were rate limit or quota errors, and 45 of those were really billing: the account had run out of credit or hit its cap. One of my own products had AI features with no spending limit at all, so one bad day could have cost any amount. I added a hard cap and a switch to turn the feature off, and I now add both before launch.
Every provider has a limit, but they behave differently, and none of them is instant:
So the provider's limit is the last line, not the first. In your own app, cap how many AI calls each user can make per day and how long each prompt and reply can be, and give each app its own project or workspace with its own limit, so one leak can't drain everything. OWASP has a name for the attack where someone runs up your bill on purpose: denial of wallet.
Slow responses and timeouts were the biggest group, at 91. AI calls take seconds, sometimes far longer, and some hang: one developer found about 1% of calls hung until timeout with no error at all. Serverless hosts make it worse. A GPT-5 call on AWS Lambda kept timing out because idle connections get dropped, and one side project got a $1,500 Vercel bill, mostly for functions sitting idle while they waited on slow streamed replies.
The second biggest group, 82 reports, was output the app couldn't use: broken JSON, missing fields, tool calls with the wrong shape. One developer found their JSON had an extra closing brace about 40% of the time until they switched on strict mode. Another saw 8 broken replies in 10,871 calls even with structured output on, and fixed it by checking and retrying.
Use the provider's structured output in strict mode: OpenAI and Anthropic both constrain the reply to a schema you define. Then check it anyway, the way the safeParse line in the example does, and retry or fall back when it fails. OWASP's advice is to treat model output like untrusted user input: never put it straight into a database query, a page's HTML or a shell command.
A quieter version of this broke one of my own products. A daily AI job published broken content about nine days a month, and nobody was told, because nothing threw an error. A better prompt didn't fix it. Checking the result before publishing did, plus an alert when the check fails. It's the same lesson as in why n8n workflows stop working: the failures that hurt are the silent ones.
Forty-four reports broke when the model changed under them. Some called a model that had been retired. Others got an update that behaved differently: a production voice agent went from almost every command working (98 to 100%) to about 25% because the new version answered instead of calling its tools, and another team's feature quietly got worse for about 10 days until a user mentioned it.
Models are retired on a schedule, and each provider gives different notice:
| Provider | Notice before a model is retired |
|---|---|
| OpenAI | At least 6 months for generally available models, as little as 2 weeks for previews |
| Anthropic | At least 60 days for public models |
| Google Gemini | At least 2 weeks for previews; "latest" aliases are swapped with each release |
So use a dated model version in production, not an alias that moves; Google's model guide says most production apps should use a specific stable model. And keep a small set of test questions with the answers you expect, taken from real use. Anthropic's testing guide says to mirror real tasks and include edge cases, and OpenAI recommends running your tests on every change. Run them before you switch models, and you find the 25% before your users do.
The wrong answers that did happen were the ones with real cost: a scheduling assistant picked the wrong dates because the model has no clock (the fix is putting today's date in the prompt), and a voice assistant skipped phone number validation and quietly corrected input to pass. In the best-known case, Air Canada's chatbot told a customer the wrong refund policy, and a tribunal held the airline to it.
No prompt makes a model always right; Anthropic's own guide says its techniques reduce but do not eliminate errors. The fix is deciding what the AI may do on its own. OWASP's guidance for AI apps is to require a human to approve high-impact actions before they are taken, and OpenAI recommends a person review outputs before use in high-stakes areas.
In practice that's one extra state on the record, "needs approval", and a screen that lists drafts with Approve and Edit buttons. I built Client Approvals around the same idea for client work: nothing goes out until the right person signs it off, and there's a record of who did.
Start with the cheapest model that passes your test questions and move up only where it fails. The price gap is large. From each provider's pricing page, per million tokens, as of October 2026:
| Model | Input | Output | Notes |
|---|---|---|---|
| GPT-6 Luna (OpenAI) | $0.10 | $0.50 | OpenAI's cheapest current model |
| Claude Haiku 5.5 (Anthropic) | $0.10 | $0.50 | Prompts up to 100,000 tokens |
| Gemini 3.8 Flash (Google) | $0.75 | $3.75 | Doubles from January 1, 2027 |
| Claude Sonnet 5.5 (Anthropic) | $2 | $10 | Mid tier |
| Gemini 3.1 Pro Preview (Google) | $2 | $12 | No free tier |
| GPT-6 Astra (OpenAI) | $10 | $50 | OpenAI's most capable |
A feature that sends 2,000 tokens and gets 500 back, 10,000 times a month, costs about $4.50 on GPT-6 Luna and about $450 on GPT-6 Astra. Work that can wait up to a day, like nightly summaries, is half price through the batch APIs of OpenAI, Anthropic and Google. Watch images: one developer found gpt-4o-mini bills image tokens at about 33 times the rate, so a picture cost as much as on the bigger model.
On your users' data: OpenAI has not trained on API data by default since March 1, 2023, and Anthropic doesn't train on commercial API data by default. Google's Gemini free tier is the exception: its terms say human reviewers may read input and output, so keep personal data on a paid tier.
Integrating AI into an app is mostly ordinary engineering: keys on the server, limits, time limits, checks and alerts. The model is the easy part to swap. What keeps the feature safe is that it drafts, a person decides where it matters, and your code never trusts a reply it hasn't checked.
If you're adding AI to an app built with Lovable, Replit or Base44, start with how to build an MVP with AI tools for where those apps break, and the AI chatbots and assistants guides for the rest. And if you'd rather hand it over, I build AI features as client projects: send me what you're building and where it stands.
How do I integrate AI into my app?
Add one route on your server that receives the user's request, calls the AI provider with a key stored in your server's environment, checks the answer, and returns it. The browser only talks to your server. Then add a spending cap, a time limit and retries, pin a model version, and have a person approve anything the AI would send, charge or delete.
Can I call the OpenAI API directly from the browser?
No. Anything in the browser can be read by whoever opens the page, including your API key. OpenAI's official Node library blocks browser use by default, and Google says keys compiled into web or mobile apps can be extracted. Route every call through your own backend.
How much does it cost to add AI to an app?
Running it is often a few dollars a month. A feature that sends 2,000 tokens and gets 500 back, 10,000 times a month, costs about $4.50 on GPT-6 Luna or Claude Haiku 5.5 and about $450 on GPT-6 Astra, at October 2026 list prices. Batch processing halves that on OpenAI, Anthropic and Google.
How do I stop a surprise AI API bill?
Keep the key on your server, set a hard spending limit in the provider's console, and cap usage per user in your own app. Provider limits are not instant: OpenAI says you can overshoot slightly, and Google Cloud budget alerts do not stop spending on their own. A separate project or workspace for each app limits the damage of one leak.
Which AI API should I use for my app?
Start with the cheapest model that passes your own test questions, and move up only where it fails. OpenAI, Anthropic and Google all offer structured output, batch discounts and spending limits, so the choice usually comes down to price, quality on your task and how long each keeps a model before retiring it.
Will OpenAI or Anthropic train on my app's data?
Not by default. OpenAI has not trained on API data since March 1, 2023, and Anthropic does not train on commercial API inputs or outputs by default. Google's Gemini free tier is different: its terms say human reviewers may read input and output, so don't send personal data through it.
How do I add ChatGPT to my website?
You add the OpenAI API, not ChatGPT itself: a chat box on your page sends messages to a route on your server, which calls the API with your key and returns the reply. If the bot should answer from your own content, it also needs your documents, and most wrong answers come from that part, not from the model.
This post first appeared on tusharbhendarkar.com. I build and fix AI chatbots and RAG systems for small businesses. If something here is broken for you, email me at tusharbhendarkar44@gmail.com.