# I wasted $9,500 on LLM API bills in one month — here are the 7 mistakes that cost us

> Source: <https://dev.to/dijinlong/i-wasted-9500-on-llm-api-bills-in-one-month-here-are-the-7-mistakes-that-cost-us-1mam>
> Published: 2026-11-16 11:10:39+00:00

If you're building an AI product and your monthly API bill looks like this: $450 → $1,200 → $2,600 → $5,900 → $9,500 — you're not alone.

Six months ago, our AI writing assistant hit product-market fit. Users loved it. Our burn rate did not.

We started out like every new AI team: import OpenAI SDK, stick the API key in .env, call chat.completions.create(), ship it. That was January. By May, we were paying nearly ten thousand dollars a month for API calls, and when we finally sat down to look at the logs, 71% of that money was completely wasted.

No, OpenAI didn't overcharge us. No, we didn't get hacked. We were just making every classic mistake in the book — and none of them were obvious until we dug into the line items.

Here's exactly what we did wrong, how much each mistake cost us, and how we fixed it. After six weeks of work, we got our monthly bill from $9,580 down to $1,700 — without touching the user experience or changing a single prompt.

Mistake #1: Using GPT-4 for literally everything

Cost us: ~$4,200/month

This is the dumbest one, and we see every new AI team do it. When we launched, every single request — from "summarize this 2-sentence email" to "rewrite this bullet point" — was hitting GPT-4.

We thought "GPT-4 is the best, why use anything worse?" That's like using a Ferrari to pick up groceries. It works, but you're paying 50x more than you need to for a job a Honda Civic could do perfectly well.

The fix was obvious once we looked at the data:

83% of our requests were simple tasks: grammar checks, short rewrites, keyword extraction, formatting. These run perfectly on GPT-4o Mini or DeepSeek V3 for 1/50th the cost

15% were medium complexity: first drafts, paragraph rewrites, basic research. These go to Claude Sonnet or GPT-4o standard

2% actually needed top-tier reasoning: complex long-form writing, tricky logic, multi-step analysis. Only these hit GPT-4o / Opus

Just routing requests to the right model cut our bill in half in the first week. Not a single user noticed a difference.

Mistake #2: Infinite retries on 429 errors

Cost us: ~$1,800/month

When OpenAI returns a 429 "rate limit exceeded" error, the standard SDK example shows you how to add a retry. So we added retries. With exponential backoff. Maximum 10 attempts. That's what all the tutorials tell you to do.

What the tutorials don't tell you is what happens when the entire API is degraded for two hours. Our retry logic dutifully retried every failed request. Then retried the retries. Then retried those. We had requests that retried 17 times (the 10 limit wasn't actually being enforced correctly, oops). None of them succeeded. None of them were ever going to succeed. We just paid for thousands of calls that immediately failed.

The fix:

Hard cap retries at 2, not 10

Circuit breaker: if >20% of requests are failing in a 30-second window, stop sending new requests entirely for 60 seconds

Fail fast to the user instead of silently retrying forever — users would rather see "try again in a minute" than wait 2 minutes for a spinner that goes nowhere

Mistake #3: Unbounded context windows

Cost us: ~$1,200/month

We were doing RAG the lazy way: stuff every single document the user ever uploaded into the context window on every request. "Context is cheap!" we told ourselves. "GPT can handle 128k tokens, why not use it?"

Turns out context is not that cheap. When you're sending 40k tokens of context on every single turn just to generate a 200-token response, your cost per request explodes. And 90% of that context was completely irrelevant to the current question. We saw requests where we sent 60k tokens of old conversation history and document chunks that had nothing to do with what the user was asking right now.

Proper retrieval: only send the top 3 most relevant chunks for the current query, not every chunk ever

Trim conversation history: keep last 10 turns, summarize older ones

Hard cap context at 8k tokens for 95% of requests, only expand for specific long-form tasks

Mistake #4: Streaming tokens we never displayed

Cost us: ~$700/month

When you use streaming responses (which you should for UX), you pay for every token that gets generated — even if the user navigates away mid-response. We had a bug where we never aborted the stream when the user closed the tab or switched conversations. The model would keep generating 500, 1000, 2000 tokens that nobody ever saw. We were literally paying OpenAI to talk to an empty room.

Add an AbortController. Cancel the stream when the user leaves. That's it. That was $700/month. I'm still embarrassed about this one.

Mistake #5: No caching for repeated queries

Cost us: ~$600/month

We had enterprise customers with 500+ employees all using the same base prompt template for common tasks. We were generating the exact same response, from scratch, hundreds of times per day, every day.

Same prompt → same model → same temperature → same input = same output 99% of the time. And we were paying for it every single time.

The fix: simple semantic cache. Hash the prompt + model + temperature, store results for 24 hours. Cache hit rate for our enterprise tier hit 37% in week one. Same response time, zero quality loss, 37% cost cut on that segment.

Mistake #6: We had no observability at all

Cost us: Incalculable (it enabled every other mistake)

For the first four months, the only metric we had was the total number on the OpenAI invoice at the end of the month. We couldn't tell you:

Which model was costing us the most

Which endpoint had the highest token usage

How many retries were happening

Which users were generating the most cost

How many tokens we were sending vs receiving

What our error rate actually was

We were flying completely blind. You can't optimize what you don't measure.

We built a tiny middleware layer that logs every single request: timestamp, user ID, model, prompt tokens, completion tokens, latency, status code, cost. Total lines of code: ~200. It took an afternoon. And that's what let us find every single other problem on this list.

If you only do one thing after reading this: add logging for every API call. You will find things that horrify you.

Mistake #7: Locking ourselves into a single provider

Cost us: ~$1,100/month (and counting)

This one is less about obvious waste and more about silent ongoing cost. We built directly against the OpenAI SDK, with hardcoded model names and endpoints. When we wanted to try Claude or Gemini or any of the cheaper open source models, it would have required rewriting hundreds of lines of code across our entire codebase. So we didn't. We just kept paying OpenAI's premium price even when there were comparable models available for 1/10th the cost.

When we finally built a simple abstraction layer — one single function that takes a standard OpenAI-format request and routes it to whichever provider/model makes sense for that task — we unlocked the ability to:

Route by complexity (what Mistake #1 talks about)

Route by price (pick the cheapest available healthy endpoint for each model)

Route by availability (automatic failover if one provider is down or rate limited)

A/B test new models against our existing workloads without touching production code

This is where the real long-term savings came from, and it's the thing we should have built on day one. We now use a multi-provider setup (we built our own internal routing layer, eventually spun it out into LONG AI for other teams to use) and it's saved us far more than the engineering time it cost.

The results

After fixing all seven issues:

Monthly API cost: $9,580 → $1,700 (82% reduction)

Average response time: 2.8s → 1.9s (faster because lighter models and closer endpoints)

Error rate: 7.2% → 0.8% (circuit breakers and failover)

User satisfaction scores: no measurable change at all — literally nobody noticed anything different except the product got faster

The three things you can do tomorrow

You don't need to rebuild your entire architecture to get most of these gains. Do these three things this week:

Add basic logging for every API call — model, tokens, cost, status, user ID. Look at the first week of data and you will immediately find obvious waste.

Cap retries at 2 and add a simple circuit breaker — this alone will save you from 4-figure surprise bills during outages.

Route your simple requests to a cheaper model — you don't need GPT-4 to check if an email is spam or fix a typo. Start with 10% of traffic, A/B test it, you will be shocked at how few users can tell the difference.

If you're earlier in your journey than we were — don't make these mistakes. If you're already making them — it's fixable. None of these required brilliant engineering or expensive tools. We just had to stop treating API calls as a black box and actually look at what we were paying for.

What's the dumbest LLM API mistake your team has made? Drop it in the comments — I've got plenty more horror stories where these came from.

If you don't want to build all this routing, logging, caching and failover yourself, that's exactly what we built LONG AI for — one API key that gives you access to 100+ models across 20+ providers with automatic smart routing, real-time cost dashboards, and built-in failover. We use it in production for our own product, and it's what got our bill down to where it is now.
