We recently launched a free tier powered by a self-hosted DeepSeek V4 Flash. We've always wanted a meaningful free tier, without the negative margins. Three months ago that felt impossible. Self-hosting the model isn't cheap, but it keeps our free-tier costs fixed.
For context on how we use it, camelAI's free tier (the model we call camelCode) runs on DeepSeek V4 Flash that we host ourselves on AWS GPU instances. Requests route through Cloudflare AI Gateway, which falls back to Azure's hosted DeepSeek if our machines go down, and paying users get scheduling priority over free users on the same hardware.
Our whole setup is open source, and everything below is in our repo, github.com/Vercantez/DeepSeek-V4-Flash-DSpark-RTX4.
Model and hardware, together #
We wanted the most powerful setup at the lowest price. Those are opposing goals. So, we had to decide the minimum quality model that worked for our product, and the hardware we could actually get.
DeepSeek V4 Flash is small. It's a mixture-of-experts model with 284B total parameters but only 13B active per token, which is what makes it servable on hardware a startup can afford. Initially it struggled on performance. But, I was able to optimize our harness around it and boost performance enough to pass our baseline eval suite (good enough for the free tier).
DeepSeek V4 Flash hits the sweet spot on size-versus-performance. If you want better performance, you need dramatically more expensive GPUs. If you go smaller, performance drops off fast. Artificial Analysis's intelligence vs cost-per-task chart puts Flash right on the Pareto frontier. It's strong enough that paying API prices for every free user would hurt, and efficient enough that self-hosting is doable.
For the hardware, we started with how much GPU memory the model needs, which comes down to precision. Model weights are numbers stored at some bit width. Models are usually trained in BF16, which is 16 bits per weight, and after training the weights get quantized to smaller formats so the model needs less GPU memory and less memory bandwidth to run. Flash's public checkpoint ships already quantized, with the bulk of its weights in MXFP4, an open standard 4-bit floating point format that keeps accuracy close to the 16-bit original at about a quarter of the memory. On disk that comes to roughly 150-160GB, where a full 16-bit version would be closer to 570GB. The quantized checkpoint was the only version we could ever hope to fit, and it still needed several GPUs.
The quantized checkpoint pointed us to NVIDIA. Blackwell, their current GPU generation, has tensor cores that run MXFP4 natively, so the 4-bit weights get full hardware speed, and NVIDIA has the best software support for open models in general. AMD was an option, but with less support it would have meant more work finding configs and fixing things ourselves. Within Blackwell, we didn't have the quota or the money for B200 machines, which left the RTX PRO 6000 as the most powerful card we could actually get. On AWS that's the G7e instance family. We use the g7e.24xlarge, which has four RTX PRO 6000s at 96GB each, 384GB in total, enough for the quantized weights with room left over for serving.
We started from a recipe #
We found an open source repo serving DeepSeek V4 Flash with DSpark on Blackwell GPUs, the same model and GPU generation as the machines we could get on AWS. Our repo adapts that work to the four-GPU box. It also carries a concurrency patch that multi-user speculative decoding needs, so it is not stock upstream vLLM. If you copy our setup, use the image tags in the repo.
If you're looking to host a model, chances are someone in the open source community has already served your desired model on hardware close to yours. Check Hugging Face and GitHub for reference recipes, which are usually just a long command for vLLM, the standard open source inference server, with flags for which model to load, at what precision, with what memory settings.
That search is also why we went with NVIDIA. Most recipes are written and tested on NVIDIA GPUs, so more of them worked without changes.
Weights live in regional S3 #
Spot instances restart more often than you'd expect, because AWS reclaims them whenever it needs the capacity. Every restart means getting roughly 150GB of weights onto a fresh box, so where the weights live matters. We don't bake them into the machine image. Instead, we keep an immutable release in an S3 bucket in the same region as the GPUs. On every boot, the instance copies the release to local NVMe, verifies a manifest, and starts vLLM.
The release holds the weights plus the compile caches that make startup tolerable. Same-region S3 delivers all of it quickly and reliably, and every replacement instance downloads the same known-good artifact. The local NVMe copy is just scratch space, so when an instance dies, nothing is lost and the next box fetches the same release. This beat the alternatives for us. Restoring from an EBS snapshot after a reclaim is slow while the snapshot hydrates, and baking everything into a machine image invites drift, where the caches in the image no longer match the container you actually shipped. Keeping the bucket in the same region matters more than it sounds, because a cross-region download turns a short instance replacement into a long outage.
Speculative decoding for speed #
For fast responses we needed speculative decoding. A small, fast draft model proposes the next few tokens, and the big model then verifies the whole draft in a single parallel pass instead of generating one token at a time. Most tokens are easy to predict, so most drafts get accepted and you generate multiple tokens per GPU cycle. When tokens are hard to predict, the speedup shrinks, so speed varies with the content.
For DeepSeek V4 we use DSpark, DeepSeek's own speculative decoding release. It's a lightweight draft module attached to the model's weights rather than a separate model. In our config it's one flag:
--speculative-config '{"method":"dspark","num_speculative_tokens":5}'
With DSpark on four RTX PRO 6000s, a single stream decodes at around 300 tokens per second. Under a realistic free-tier load, each user sees tens of tokens per second, while aggregate throughput climbs toward roughly 3,000 tokens per second as the batch fills. One box tops out around 64 active sequences, and past that requests queue. When we need more concurrency and more per-user speed at the same time, we add another box.
The KV cache mattered most #
In my opinion, this was the most important part. Every token in a conversation has key and value tensors that the model reuses on each request, and together these are the KV cache. In production it takes up most of your GPU memory, and if you ignore it, it becomes your bottleneck no matter how fast your GPUs are.
DeepSeek helped us here, and it's a real reason we picked it for self-hosting. the model and serving many long sessions are different problems. The mixture-of-experts design handles the first, but most of our traffic is the second, because agent sessions run long. A coding session accumulates tool results, file reads, and prior turns, and on most models the KV cache grows with every token of that history. A handful of long chats can fill the GPU while its compute sits mostly idle.
V4's attention design avoids keeping a full entry for every past token in every layer. Recent history is compressed and attended over sparsely, older history is compressed much more aggressively, and a short window of raw tokens keeps the latest detail sharp. We pay for the large weight file once, and after that, serving many long conversations costs much less than the parameter count suggests.
On our live workers, vLLM reports about 2.7 million tokens of GPU KV cache, roughly ten full sessions at our 256K cap:
GPU KV cache size: 2,712,968 tokens
Maximum concurrency for 262,144 tokens per request: 10.35x
Three things we did:
Cap the context window. Flash supports 1 million tokens of context, but long contexts degrade model performance and use more memory. At full 1M context, a couple of users would fill our entire cache and block everyone else, so we cap every user at 256K and compact conversations in the app at around 220K.Quantize the KV cache. Our production config uses the supported quantized cache path for DeepSeek's compressed attention, so each token of history costs less memory.Offload old caches. Recomputing the KV cache for a long conversation (called prefill) costs GPU time. But keeping every idle conversation in GPU memory blocks active requests. vLLM can offload KV caches to CPU memory and disk. These AWS boxes have lots of RAM sitting idle and fast NVMe disks, so colder caches move there instead of taking up GPU space. Off only helps if a user's next request lands on the machine that has their cache, so we run a sticky router in front of the workers and pin each conversation to one box.
Making it production ready #
Secure the endpoint. We keep the GPUs internal so only our backend path can reach them.Fair scheduling. vLLM lets us assign priorities to requests (--scheduling-policy priority
), so paying users go ahead of free users. We also deprioritize free users who have already used the service heavily, which keeps new users from waiting behind them.Spot recovery. When AWS reclaims an instance, our auto scaling group launches a replacement, the boot script pulls the S3 release onto local NVMe, and the router only sends traffic once the model endpoint reports healthy. This matters because an instance can show asInService
in AWS while the weights are still .Fallbacks. Spot instances are cheap because AWS can reclaim them at any time, so we needed a fallback for consistent uptime. We run behind Cloudflare AI Gateway, which handles this for us. When our spot capacity disappears, traffic goes to Azure's hosted DeepSeek instead. Azure is slower and more expensive than our box, but it accepts our Azure credits and it keeps the free tier up.
What it costs #
On-demand, a g7e.24xlarge runs about $12K a month. On spot, prices move around, but running 24/7 has cost us roughly $4K a month in Ohio and $6K in Oregon, and spot quotes dip lower than that.
The exact number matters less than how the cost behaves. Per-token pricing scales with usage. Give a million free users even $1 of tokens each and you're out a million dollars. A self-hosted box is a fixed cost at a given fleet size, and when more people use it, requests get batched together or queued, so the service gets slower instead of more expensive. That's the trade we wanted for a free tier. Slower responses for free users are an acceptable cost, but a bill that grows with fraud and abuse is not. The spend that does still scale with demand is the fallback, and eventually another box when we outgrow this one.
Spot plus cloud credits is what made the economics work for us. Our AWS credits pay for the spot instances and our Azure credits pay for the fallback, so while those last the free tier costs us close to nothing in actual cash while we grow.
What's next #
We're working on two things now.
First, we're evaluating Poolside Laguna S 2.1. It's smaller than DeepSeek V4 Flash, about 118B total and 8B active parameters versus 284B and 13B, so it should be easier to fit and cheaper to serve if the quality holds up on our agent workloads. We're running it through the same harness to see whether it does, and whether serving it is simpler on the hardware we already have.
Second, we're fine-tuning our self-hosted model using the evals we've built for camelAI. The free tier only works if the model is good at our product, not just good on public benchmarks. Those evals are how we decide what to train on, and how we'll judge whether a candidate like Laguna or a fine-tune of Flash is actually better for users.
TL;DR #
- We picked the model and hardware together: DeepSeek V4 Flash on 4x RTX PRO 6000 Blackwell (AWS g7e.24xlarge).
- We started from a community recipe and adapted it. Ours is here. - Weights live in same-region S3 and get copied to local NVMe on every boot.
- DSpark speculative decoding for speed.
- The KV cache mattered most. Flash's compressed attention is why we hold about 2.7M tokens of GPU KV cache, roughly ten sessions at our 256K cap. We cap context, quantize the cache, offload idle caches, and pin each conversation to one worker.
- Priority scheduling for paying users, and a gateway fallback for when spot capacity disappears.
- Cost: roughly $4-6K a month per box on spot versus about $12K on-demand, fixed at a given fleet size. More users means slower service, not a bigger bill.
- Next up, we're evaluating Laguna S 2.1 as a smaller model to serve and fine-tuning Flash on our own evals.