How I cut my AI inference bill by 80% - moving from OpenAI to self-hosted models

A developer cut monthly AI inference costs from roughly $490 to $195 by migrating a RAG application off OpenAI's GPT-4o to self-hosted models, serving Qwen 2.5 72B through vLLM on a single Hetzner A100 80GB GPU with LiteLLM as an OpenAI-compatible gateway. The setup handles about 3M input and 500K output tokens per day, with self-hosted embeddings via all-MiniLM-L6-v2, yielding roughly $3,540 in annual savings at the cost of higher p99 latency (2.5s versus 800ms), occasional vLLM crashes, and 24/7 on-call burden. The developer notes the math only works above roughly $500/month in API spend and that savings shrink at higher volume.

My OpenAI bill was $400/month. After moving to self-hosted models on a single Hetzner GPU box, it's $80/month and falling. Here's the playbook. The OpenAI bill I was using GPT-4o for ~3M input tokens/day for a RAG app and ~500K output tokens/day. About $450/month. The killer was the output tokens. What I tried Option 1: Smaller OpenAI models gpt-4o-mini Worked for ~30% of queries. Quality drop too steep for the rest. Saved ~$100/month. Option 2: Llama 3.1 70B on RunPod Cost: $1.50/hour for an A100, mostly idle. About $300/month. Same quality as GPT-4o on most of my tasks. Breakthrough. Option 3: Self-hosted on Hetzner Hetzner GPU servers H100, A100 in Frankfurt. Latency ~10ms from EU, ~95ms from US. Worked for me. What I actually run Hardware : Hetzner GPU server, 1× A100 80GB on-demand. Effective cost ~$190/month shared between workloads . Software : vLLM serving Qwen 2.5 72B Apache 2.0 , all-MiniLM-L6-v2 for embeddings, LiteLLM as drop-in OpenAI-compatible gateway. Cost breakdown | | OpenAI | Self-hosted | | LLM inference | $400/mo | $190/mo | | Embeddings | $60/mo | $5/mo | | Vector DB | $30/mo | $0 | | Total | $490/mo | $195/mo | Savings: ~$295/month = $3,540/year. The catch You need: - Linux + Docker + Python comfort - Willingness to debug CUDA driver issues - Capacity to handle 4-second cold-start latency If you have $500+/month OpenAI spend, the math works. What I gave up - Peak latency : OpenAI p99 = 800ms. Mine = 2.5s. - Model variety : Locked to one model. OpenAI gives easy access to GPT-4o, o1, o3-mini. - API stability : vLLM crashes occasionally. OpenAI never. - 24/7 on-call : When vLLM goes down at 3am, I get paged. What surprised me - Qwen 2.5 72B is competitive with GPT-4o on most coding/writing tasks. - vLLM throughput is amazing — 50+ requests/sec on a single A100. - Embeddings are dirt cheap to self-host — all-MiniLM-L6-v2 is 80 MB, runs anywhere. - Hardware isn't the bottleneck . Model loading + tokenization is. My migration playbook 1. Measure current bill . Know what's hurting. 2. Pick highest-volume task . Start there. 3. Stand up vLLM locally . Test with your prompts. 4. Add LiteLLM in front . Drop-in OpenAI replacement. 5. Move 10% of traffic . Compare outputs. 6. Move 100% if quality OK . Save the bill. 7. Set up monitoring . vLLM doesn't alert on crash. When NOT to self-host - Under $200/month on OpenAI - Need o1-class reasoning - No GPU/LLM ops experience - Can't tolerate 24/7 on-call Cost projection If I grow to 2x volume: - OpenAI: ~$980/month - Self-hosted: needs 2× A100. ~$700/month Savings drop as you scale. But at current scale, savings are real. Final thought AI inference is now genuinely cheap to self-host if you have the volume to amortize setup. If you're paying $400+/month to OpenAI, it's worth investigating.