# How I cut my AI inference bill by 80% - moving from OpenAI to self-hosted models

> Source: <https://dev.to/scsoi/how-i-cut-my-ai-inference-bill-by-80-moving-from-openai-to-self-hosted-models-16fc>
> Published: 2026-10-19 21:28:53+00:00

My OpenAI bill was $400/month. After moving to self-hosted models on a single Hetzner GPU box, it's $80/month and falling. Here's the playbook.

## 
  
  
  The OpenAI bill

I was using GPT-4o for ~3M input tokens/day for a RAG app and ~500K output tokens/day. About $450/month. The killer was the output tokens.

## 
  
  
  What I tried

### 
  
  
  Option 1: Smaller OpenAI models (gpt-4o-mini)

Worked for ~30% of queries. Quality drop too steep for the rest. Saved ~$100/month.

### 
  
  
  Option 2: Llama 3.1 70B on RunPod

Cost: $1.50/hour for an A100, mostly idle. About $300/month. Same quality as GPT-4o on most of my tasks. **Breakthrough.**

### 
  
  
  Option 3: Self-hosted on Hetzner

Hetzner GPU servers (H100, A100) in Frankfurt. Latency ~10ms from EU, ~95ms from US. Worked for me.

## 
  
  
  What I actually run

**Hardware**: Hetzner GPU server, 1× A100 80GB on-demand. Effective cost ~$190/month (shared between workloads).

**Software**: vLLM serving Qwen 2.5 72B (Apache 2.0), all-MiniLM-L6-v2 for embeddings, LiteLLM as drop-in OpenAI-compatible gateway.

## 
  
  
  Cost breakdown

|  | OpenAI | Self-hosted | 
| LLM inference | $400/mo | $190/mo | 
| Embeddings | $60/mo | $5/mo | 
| Vector DB | $30/mo | $0 | 
| Total | **$490/mo** | **$195/mo** | 

Savings: ~$295/month = $3,540/year.

## 
  
  
  The catch

You need:

- Linux + Docker + Python comfort
- Willingness to debug CUDA driver issues
- Capacity to handle 4-second cold-start latency

If you have $500+/month OpenAI spend, the math works.

## 
  
  
  What I gave up

- 
**Peak latency** : OpenAI p99 = 800ms. Mine = 2.5s.
- 
**Model variety** : Locked to one model. OpenAI gives easy access to GPT-4o, o1, o3-mini.
- 
**API stability** : vLLM crashes occasionally. OpenAI never.
- 
**24/7 on-call** : When vLLM goes down at 3am, I get paged.

## 
  
  
  What surprised me

- 
**Qwen 2.5 72B is competitive with GPT-4o** on most coding/writing tasks.
- 
**vLLM throughput is amazing** — 50+ requests/sec on a single A100.
- 
**Embeddings are dirt cheap to self-host** — all-MiniLM-L6-v2 is 80 MB, runs anywhere.
- 
**Hardware isn't the bottleneck** . Model loading + tokenization is.

## 
  
  
  My migration playbook

1. 
**Measure current bill** . Know what's hurting.
2. 
**Pick highest-volume task** . Start there.
3. 
**Stand up vLLM locally** . Test with your prompts.
4. 
**Add LiteLLM in front** . Drop-in OpenAI replacement.
5. 
**Move 10% of traffic** . Compare outputs.
6. 
**Move 100% if quality OK** . Save the bill.
7. 
**Set up monitoring** . vLLM doesn't alert on crash.

## 
  
  
  When NOT to self-host

- Under $200/month on OpenAI
- Need o1-class reasoning
- No GPU/LLM ops experience
- Can't tolerate 24/7 on-call

## 
  
  
  Cost projection

If I grow to 2x volume:

- OpenAI: ~$980/month
- Self-hosted: needs 2× A100. ~$700/month

Savings drop as you scale. But at current scale, savings are real.

## 
  
  
  Final thought

AI inference is now genuinely cheap to self-host if you have the volume to amortize setup. If you're paying $400+/month to OpenAI, it's worth investigating.