Your AI bill tripled last quarter. Your CTO forwarded you an article about companies saving 70% by switching to open models. Now someone is asking you to figure out what that would actually look like.
I spent the last few weeks digging into this. The numbers, the hardware, the real trade-offs. Here's what I found, with enough specifics that you can actually make a decision rather than just nodding along to another "open source is the future" think piece.
When someone says "open model" they mean an AI model where the weights (the learned parameters that make the model work) are publicly downloadable. You grab the file, run it on your hardware, and you don't pay anyone per request.
The big names right now: Meta's Llama 4, DeepSeek V4, Zhipu's GLM-5.2, Moonshot's Kimi K3, Alibaba's Qwen 3.5, and Google's Gemma 4.
These aren't toys. Some of them genuinely compete with the frontier models on real benchmarks. Chinese open models now handle over 30% of enterprise traffic on OpenRouter, up from 4.5% in early 2025. That's a massive shift in barely a year.
You want your team to use an open model. Here's the stack, from bottom to top.
A model is a giant file. We're talking anywhere from 4 GB (a small 7B model, quantized) to 1.5 TB (Kimi K3, full weights). That entire file needs to sit in GPU memory to run fast.
Why GPU memory specifically? Because generating each word in a response requires billions of multiply-and-add operations. GPUs do thousands of these in parallel. A CPU does them one at a time.
The practical difference: a 7B model on a CPU generates 2-5 tokens per second (painfully slow for interactive use). The same model on a GPU generates 30-80 tokens per second (feels instant). For one person on a CPU, it might be tolerable. For a team of 10 all hitting the same endpoint? Unusable. Requests queue up and everyone waits 30-60 seconds for responses.
Think of it like a highway. A CPU is a single lane with a high speed limit. A GPU is 4,000 lanes at a moderate speed. Language model inference is a traffic problem, not a speed problem. You need lanes, not a faster car.
Good news: the software stack is mature, open-source, and works today. No custom code required.
The setup: install vLLM, run vllm serve meta-llama/Llama-4-Maverick
, point Open WebUI at it, hand your team the URL. A day of work for someone comfortable with Linux. The vLLM API is OpenAI-compatible, which means any tool, extension, or script that works with the OpenAI API works here with zero code changes. Just swap the endpoint URL.
Download from Hugging Face with a single command. Models come in different quantization levels (compression trade-offs). A 4-bit quantized version is roughly 4x smaller than the full-precision version, with minor quality loss. For most team use cases, the quantized versions are the practical choice because they fit in less GPU memory.
This is where it gets real. I'm using AWS on-demand pricing as of August 2026.
For 10 people, you want a single inference server that everyone hits via API or web UI. The sweet spot model is **Llama 4 Maverick** (400B parameters, MoE architecture, but only ~17B active per request). It's a Meta model (US-origin, community license), strong all-rounder, and runs on a single node with 4 GPUs.
| Setup | AWS Instance | Monthly (business hours) | Monthly (24/7) |
|---|---|---|---|
| Budget (Qwen 3.5-27B) | g5.2xlarge (1x A10G) | ~$440 | ~$1,460 |
| Sweet spot (Maverick) | g5.12xlarge (4x A10G) | ~$1,250 | ~$4,100 |
| Always-on + reserved | g5.12xlarge (1-yr RI) | ~$2,900 | - |
| Frontier-class (Kimi K3 self-hosted) | p6-b300.48xlarge (8x B300) | ~$25,000-30,000 | ~$71,000+ |
The business-hours trick is the key cost saver. If your team works 10 hours a day on weekdays, you pay for ~220 hours/month instead of 730. Set up a Lambda or EventBridge scheduler to stop the instance at night and start it each morning. That single optimization cuts 70% off the bill.
Why 4 GPUs for Maverick? The model has 400B total parameters. Even though only 17B are active per request, all 400B must sit in memory. Each A10G has 24 GB of VRAM. Four of them give you 96 GB total, enough to hold the quantized Maverick model comfortably while leaving headroom for request batching.
At 500 people, the bottleneck is concurrent requests. If 10-15% of your company is hitting the model simultaneously, that's 50-75 concurrent requests. One server won't keep up. You need 3-4 replicas behind a load balancer.
| Setup | Cost/Month |
|---|---|
| Budget (business hours, reserved, some queuing at peak) | $3,500-5,000 |
| Standard (always-on, savings plan, good response times) | $9,000-12,000 |
| Premium (autoscaling, always fast, redundant) | $12,000-16,000 |
For comparison: 500 ChatGPT Enterprise seats would run ~$30,000/month. 500 people calling the Claude API at moderate usage (50 requests/person/day) would cost $6,000-12,000/month. The self-hosted route is competitive at this scale, and you get data sovereignty on top. Don't forget the hidden costs: someone (or a small team) needs to keep this running. Model updates, instance reboots, monitoring, scaling adjustments. At 500 users that's justified. At 10, it might be more hassle than it's worth.
(Pricing based on AWS on-demand rates, August 2026. Check the EC2 pricing page for current numbers.)
Here's the part that makes this conversation timely. The gap between open and closed models has collapsed. At the end of 2023, it was 17.5 percentage points on standard benchmarks. By mid-2026, it's single digits on most tasks and effectively zero on knowledge benchmarks.
The top contenders right now:
DeepSeek V4 Pro matches frontier on coding tasks with an 80.6 score on SWE-Bench Verified. MIT license. From a Chinese lab.
GLM-5.2 (Zhipu AI) beats Claude Opus on some coding benchmarks at 46% of the cost. MIT license. Fastest throughput of the top-tier open models.
Kimi K3 (Moonshot) is a 2.8 trillion parameter beast. Near-frontier quality. Available via API at $3/1M input tokens. Self-hosting it requires 8+ NVIDIA B300 GPUs and costs $70,000+/month, which makes zero sense for most organizations. Use the API instead.
That said, if you do want to self-host Kimi K3 on AWS, it's now documented. AWS published a step-by-step guide for deploying Kimi K3 on Amazon SageMaker HyperPod and Amazon EKS. The infrastructure: a p6-b300
instance (8x NVIDIA B300 Blackwell Ultra GPUs) using vLLM as the serving engine, with Flexible Training Plans or Capacity Blocks for GPU reservation. Enterprise-grade, not a weekend project, but at least the path is documented.
Llama 4 Maverick (Meta) is the US-origin workhorse. Roughly 90% of frontier quality on general tasks. The most practical model to self-host because it balances capability with reasonable hardware requirements.
Qwen 3.5 (Alibaba) has an Apache 2.0 license and the 27B model is surprisingly capable for coding and structured tasks. Runs on a single GPU. The budget pick.
Should you actually do this? Here's my framework.
Self-host if: Stick with API providers if:
The middle ground (what most teams should actually do): Route your traffic. Use open models for the 80% of tasks that don't need frontier quality: summarization, drafting, code completion, internal Q&A. Keep Claude or GPT for the 20% that does: complex reasoning, high-stakes decisions, nuanced analysis. This alone cuts your AI bill by 60-80% without sacrificing quality where it matters.
I'd be dishonest if I didn't mention this. Almost all the leading open models are coming out of Chinese labs. DeepSeek, Zhipu, Moonshot, Alibaba. They're outpacing Meta's Llama on most benchmarks.
Depending on your compliance posture, this might not matter. The weights are MIT-licensed, you self-host, no data leaves your infrastructure. Or it might be a hard blocker if your security team won't allow Chinese-origin model code on company infrastructure.
If you're in the latter camp, your practical options narrow to Llama 4 Maverick and whatever Google releases next as Gemma. Both are capable. Neither is the best open model available. That's the current state of things.
If you're running agentic AI workflows on self-hosted models, the security surface is different from API-hosted ones. I covered that in [The OWASP Agentic AI Top 10: What Builders on AWS Need to Know](https://blog.technodrone.cloud/2026/07/aws-intro-owasp-agentic-top10.html). Worth reading if you're planning to give these models tool access.
If this is new territory for you, here's the least risky way to start:
Total cost of this experiment: about $200.
The question isn't "open vs closed" anymore. That debate is over. The question is: which tasks go where? And if your architecture assumes your AI provider will always be there, you're running on hope. Self-hosting gives you a hedge. Whether that hedge is worth the operational cost depends on your team, your usage, and your risk tolerance.
But $200 to find out? That's not a bet. That's a rounding error.
I would be very interested to hear your thoughts or comments, so please feel free to ping me on Twitter or LinkedIn or leave me a comment below.