To force my own hand, I decided to build a tool that would make benchmarking impossible to ignore. I wanted a single prompt to fire at six different models simultaneously, streaming their responses side-by-side in columns, with real-time metrics for time-to-first-token (TTFT) and cost per run displayed right underneath.
The result is a 390-line Python script that serves as a practical tutorial for anyone wanting to run a real-world model shootout.
Implementation details and the OpenAI-compatible shortcut #
The integration part is actually the easiest bit. DigitalOcean’s inference endpoint is OpenAI-compatible, which makes the setup incredibly straightforward. If you are building an AI workflow and want to swap models without rewriting your entire logic, this is the standard you want to see.
client = OpenAI(
base_url="https://inference.do-ai.run/v1/",
api_key=os.environ["DIGITAL_OCEAN_MODEL_ACCESS_KEY"],
)
Every model I tested—Llama, DeepSeek, Mistral, Qwen, and OpenAI's open-weight line—runs through that exact same client. You just swap the model string. One quick heads-up for anyone trying this: make sure you are using a model access key from the Gradient AI Platform, not your standard API token from the main settings page.
Why I skipped the async event loop #
I wanted a true race where the columns fill up at the same time. The "correct" way to do this would be a server-side fan-out that multiplexes everything over a single connection, but I went with something much simpler. I had the browser open one EventSource
per model:
GET /stream?model=&prompt=
Six models, six independent connections. I used Flask in a synchronous way—no complex orchestration layer or heavy async setup. The entire streaming path is only about forty lines of code. It’s a beginner-friendly approach that works surprisingly well, though it led me straight into a debugging trap.
The Gunicorn bottleneck you didn't see coming #
I spent an hour chasing a bug I thought I understood, but I actually misdiagnosed the symptoms. I knew that Gunicorn’s default sync worker would struggle with long-lived streaming connections. I expected the page to just hang or time out. It didn't hang; instead, it created a very specific kind of latency pattern that looked like "slow" models.
When running with a single sync worker, the first tokens arrived in a staggered, sequential pattern:
mistral-3-14B: 1250 msopenai-gpt-oss-120b: 5326 msopenai-gpt-oss-20b: 7278 msdeepseek-3.2: 8347 msllama-4-maverick: 9147 ms
If you look at those numbers, the "latency" isn't actually the model's inference speed—it's a queue. Each stream was waiting for the previous one to finish before it could even start. It took over 10 seconds just to get the first tokens through because they were being processed one by one.
To fix this for a real-world deployment, you need to switch to threaded workers to allow for concurrency:
web: gunicorn --worker-class gthread --threads 16 --timeout 120 'app:create_app()'
Once I made that change, the results shifted dramatically. Four of the six first tokens landed within a 1.4-second window rather than marching across a ten-second timeline. This is a vital lesson for anyone building LLM agents or streaming interfaces: your infrastructure's concurrency model will often masquerade as model latency if you aren't careful.