My OpenAI bill was $400/month. After moving to self-hosted models on a single Hetzner GPU box, it's $80/month and falling. Here's the playbook.
#
The OpenAI bill
I was using GPT-4o for ~3M input tokens/day for a RAG app and ~500K output tokens/day. About $450/month. The killer was the output tokens.
#
What I tried
Option 1: Smaller OpenAI models (gpt-4o-mini) Worked for ~30% of queries. Quality drop too steep for the rest. Saved ~$100/month.
Option 2: Llama 3.1 70B on RunPod
Cost: $1.50/hour for an A100, mostly idle. About $300/month. Same quality as GPT-4o on most of my tasks. Breakthrough.
Option 3: Self-hosted on Hetzner Hetzner GPU servers (H100, A100) in Frankfurt. Latency ~10ms from EU, ~95ms from US. Worked for me.
#
What I actually run
Hardware: Hetzner GPU server, 1× A100 80GB on-demand. Effective cost ~$190/month (shared between workloads).
Software: vLLM serving Qwen 2.5 72B (Apache 2.0), all-MiniLM-L6-v2 for embeddings, LiteLLM as drop-in OpenAI-compatible gateway.
#
Cost breakdown
| | OpenAI | Self-hosted | | LLM inference | $400/mo | $190/mo | | Embeddings | $60/mo | $5/mo | | Vector DB | $30/mo | $0 | | Total | $490/mo | $195/mo |
Savings: ~$295/month = $3,540/year.
#
The catch
You need:
- Linux + Docker + Python comfort
- Willingness to debug CUDA driver issues
- Capacity to handle 4-second cold-start latency
If you have $500+/month OpenAI spend, the math works.
#
What I gave up
Peak latency : OpenAI p99 = 800ms. Mine = 2.5s. #
Model variety : Locked to one model. OpenAI gives easy access to GPT-4o, o1, o3-mini. #
API stability : vLLM crashes occasionally. OpenAI never. #
24/7 on-call : When vLLM goes down at 3am, I get paged.
#
What surprised me
Qwen 2.5 72B is competitive with GPT-4o on most coding/writing tasks. #
vLLM throughput is amazing — 50+ requests/sec on a single A100. #
Embeddings are dirt cheap to self-host — all-MiniLM-L6-v2 is 80 MB, runs anywhere. #
Hardware isn't the bottleneck . Model + tokenization is.
#
My migration playbook
Measure current bill . Know what's hurting. 2. Pick highest-volume task . Start there. 3. Stand up vLLM locally . Test with your prompts. 4. Add LiteLLM in front . Drop-in OpenAI replacement. 5. Move 10% of traffic . Compare outputs. 6. Move 100% if quality OK . Save the bill. 7. Set up monitoring . vLLM doesn't alert on crash.
#
When NOT to self-host
-
Under $200/month on OpenAI
-
Need o1-class reasoning
-
No GPU/LLM ops experience
-
Can't tolerate 24/7 on-call
#
Cost projection
If I grow to 2x volume:
- OpenAI: ~$980/month
- Self-hosted: needs 2× A100. ~$700/month
Savings drop as you scale. But at current scale, savings are real.
#
Final thought AI inference is now genuinely cheap to self-host if you have the volume to amortize setup. If you're paying $400+/month to OpenAI, it's worth investigating.