Key highlights #
One hour from model access to a working Rust inference runtime, with no early access. #
1.59–1.63× the throughput of SGLang-Omni, which had day-zero support. #
38–60% lower time to first audio , with first speech in 15–24 milliseconds.
What happened #
Fish Audio released Drama 3 with day-zero SGLang-Omni support. Our Inference Engine Factory has generated more than 20 engines to date. Drama 3 combines three techniques that standard speech runtimes aren’t built for: FP8 mixture-of-experts, codebook multi-token prediction and linear attention. Each needs custom engineering to run fast. Tackling a new architecture on release day made Drama 3 one of our hardest tests yet.
Yet only one hour after we started, our self-improving agent swarms had a working Rust runtime that outperformed SGLang-Omni on every prompt tested.
Benchmark results #
Both runtimes used 2× B200: one GPU for generation, one for the codec. Each test used one speech stream, a 1-frame first chunk, and 20 warm trials per prompt.
We measured time to first audio, the delay before the first chunk of speech arrives, and throughput, how many audio frames the runtime generates per second. Lower latency and higher throughput are better.
| Benchmark | Runtime | Time to first audio ↓ Lower is better | Throughput ↑ Higher is better | RTF ↓ Lower is better |
|---|---|---|---|---|
| Hello | INT21 Rust | 15.11 ms | 223.4 frames/s | 0.0964 |
| Hello | SGLang-Omni | 37.79 ms | 136.9 frames/s | 0.1573 |
| Multiple sentences | INT21 Rust | 17.55 ms | 249.2 frames/s | 0.0864 |
| Multiple sentences | SGLang-Omni | 38.61 ms | 154.9 frames/s | 0.1390 |
| Voice reference | INT21 Rust | 23.58 ms | 231.0 frames/s | 0.0932 |
| Voice reference | SGLang-Omni | 38.03 ms | 145.4 frames/s | 0.1481 |
INT21 delivers 38–60% lower time to first audio and 1.59–1.63× the throughput across all three prompts. These are our B200 measurements, separate from SGLang’s published H200 results.
What the numbers mean #
Same GPUs, more speech. In single-stream use, 1.59–1.63× throughput means 37–39% less GPU time per second of generated audio. #
Faster first response. We deliver the first chunk of speech in 15–24 ms, versus about 38 ms for SGLang-Omni. In voice agents, that first-response delay is what users feel. #
Release-day support. New model architectures usually require weeks of hand-porting before they run efficiently. The agent swarms closed that gap in an hour. #
Built for deployment. Rust is compiled and memory-safe, with no Python interpreter in the serving path.
Faster inference means faster building #
When AI writes code, inference latency becomes development latency. Coding, debugging and revision loops all wait on model output. Faster inference lets builders test more ideas, fix problems sooner and ship faster. It also makes what they ship more responsive. Ultra-fast inference is a moat for building speed and user experience alike.
Steps to superintelligence #
The bottleneck in AI is shifting from what models can do to how fast infrastructure can keep up. Self-improving infrastructure removes this hurdle, as AI agents generate, test, measure, and improve the software that runs AI models.
At INT21, we’ve achieved a major milestone: when a new architecture shipped, we only needed one hour for it to run faster than the officially supported runtime. Each engine that the agent swarms build makes the next one faster.
This is what progress toward superintelligence looks like. To learn more, explore INT21’s Inference Engine Factory here, or contact us.