Training speed is the bottleneck on model iteration. SwarmOS, backed by GPT-6 Astra, generated a Rust/CUDA fully sharded data-parallel trainer for Qwen3.8-27B (the AI model released by Alibaba in August). Qwen3.8-27B is too large to fit on a single GPU, so the trainer distributes both the model parameters and training data across eight devices, with each GPU holding a slice of the model and training on a separate batch. The Rust runtime is built directly on the CUDA Driver API, with GPU kernels loaded from precompiled CUBIN binaries.
On eight NVIDIA B200 GPUs, it achieved over 11.5× higher throughput of the eager PyTorch FSDP2 baseline, and 21.5% higher throughput than SwarmOS-tuned PyTorch FSDP2. Tuning PyTorch closed most of the gap. The results demonstrate that agents can improve an existing framework as well as build a specialized trainer.
Key Takeaways #
Agents are building serious training infrastructure. SwarmOS generated a fully sharded trainer that fine-tunes the model’s entire 26.9-billion-parameter text backbone across eight GPUs. #
Specialization is the new advantage. A trainer optimized for one model, one objective, and one hardware configuration outperforms frameworks designed to handle a thousand use cases. This is part of Project Weaning: our effort to remove Python from every layer of the AI infrastructure stack and enable agents to develop, maintain, and continuously improve it end-to-end. #
Ownership enables optimization. Three design choices compound the trainer’s performance: operator-specialized CUDA kernels, elimination of the general autograd engine from the training path, and unified control over forward, backward and buffer lifetimes enables aggressive fusion.
The Results #
21.5% higher throughput than the tuned baseline; over 11.5× the original eager (with Fused AdamW and FlashAttention) baseline.
| Trainer | Steady-state throughput | Reported peak memory |
|---|---|---|
| SwarmOS-generated Rust/CUDA | 25,004 tokens/s | 76.92 GB |
| SwarmOS-tuned PyTorch FSDP2 | 20,582 tokens/s | 98.08 GB |
| Original eager PyTorch FSDP2 | 2,168 tokens/s | 95.76 GB |
Original matched run: eight NVIDIA B200s, one 32,768-token sequence per GPU, and no gradient accumulation: 262,144 input tokens per update. BF16 Weights and most of activations, FP32 for gradient accumulation, loss and norm. Rust and eager PyTorch completed 64 updates / 16.78 million tokens from the same checkpoint, using identical novel excerpts, input order, and optimizer settings.
Loss check: in that paired run, heldout loss improved from approximately 2.2026 to 2.1442 for Rust and 2.1445 for eager PyTorch. A heldout-loss result for the tuned PyTorch configuration has not been reported here.
Trading Generality for Speed #
The bottleneck on model scaling has been computational power—and the solution has been to add more GPUs solved the problem. But as frontier models become denser and more capable, and as hardware becomes more abundant, infrastructure efficiency emerged as a real constraint. A 27-billion-parameter model like Qwen3.8 can be iterated upon only as fast as the training loop completes.
PyTorch’s FSDP (Fully Sharded Data Parallel) is the industry standard for multi-GPU training. It abstracts away the complexity of sharding parameters, gradients, and optimizer states across devices. That abstraction buys simplicity—a researcher can write a training loop and scale it to hundreds of GPUs without rearchitecting. But abstraction has a cost. General-purpose frameworks carry overhead: autograd compilation, dynamic shape handling, multi-process synchronization, and the Python runtime itself. On eight B200 GPUs, that overhead compounds. Eager PyTorch FSDP2 achieves 2,168 tokens per second. The machine is capable of far more.
But what if our self-improving agent swarms can build and own a specialized trainer that trades generality for speed?
What Owning the Trainer Gives Us #
Specialized kernels and an explicit forward/backward schedule let us choose what to compute, retain, recompute, or fuse. That control extends across the system:
One process, multiple threads. GPU workers share host metadata and a token cache through a simpler control plane. #
Explicit reliability checks. Buffer ownership, collective ordering, and optimizer commits become concrete invariants to check. Broader reliability still needs validation. #
Fusion across the computation. Coordinate forward and backward work, reuse derivative buffers, and remove unnecessary intermediates. #
Memory, compute, and communication designed together. Choose sharding, activation retention, recomputation, and transfers as parts of one execution schedule. #
Ahead-of-time deployment. Compile the Rust trainer ahead of time and load precompiled CUBIN kernels through the CUDA Driver API, no runtime compilation decisions.
Evidence of Artificial General Intelligence #
INT21 operates with only two human employees, by design. All of our work on GPU kernels, inference engines, and training infrastructure is carried out by autonomous agents. We build the Swarm, not the infra. This trainer is concrete evidence that agents can own production systems and infrastructure that was previously the domain of specialized human engineers. As frontier models approach AGI capabilities, the infrastructure requirements will become more stringent. Training will need to scale, and inference will need to adapt to novel workloads in real time.
SwarmOS demonstrates that agent swarms can generate, validate, and continuously improve infrastructure systems that operate at scale. A native Rust/CUDA trainer, optimized for a specific model and hardware configuration, is one instance. Over time, this pattern of agents owning the infrastructure stack and continuing to self-improve, will become the foundation for AGI systems to operate on.
We believe AGI is coming, and we hope this work contributes to the infrastructure it will need. For now, this is an encouraging step toward agents building and improving the AI software stack.