I spent most of my career in managed-runtime languages like Java and Python, so coming to Rust presented a few new land mines for me. I'm now about six months into primarily writing code in Rust, or more accurately, causing code to be authored in Rust, and I think I'm finally getting to the far end of the mine field.
Here are some of the mines that I stepped on. Most of these are minor irritants if you're living in 2023 writing code a line at a time by hand, but they can absolutely ruin your day when multiplied by a dozen agents hitting them at once.
Mine #1: Cargo is a bad roommate #
Even Java programmers like me know that rustc is slow. But I didn't know that cargo defaults to just growing target/ larger and larger until you run out with ENOSPC.
Solution #1: MBX
Rustaceans have been using sccache and cargo-sweep for years, but recently Jeff Dickey created Mr. Boxington, aka mbx, which adds Rust-specific quality-of-life upgrades. It's not an exaggeration to say that if you're invoking rustc, you should be using mbx. Here are some of the things that mbx does for you:
- Cleans up
target/automatically. - Shares compiled dependencies across builds with reflinks, so you only ever need to build it once no matter how many concurrent worktrees or agents you have running. (But read the fine print around
share_workspace_root.) - Also shares completed crates with reflinks, but see below for the fine print.
- Also there's a server version that can throw (2) and (3) on AWS for entire teams to benefit from.
- (Unlike most build caches) plays nice with incremental builds.
- Gives you a single throat to choke when Cargo tries to run you out of space.
- Optionally throttles how much cpu and memory your builds can use to reduce contention and prevent OOM.
Mine #2: Rust builds are nightmarishly huge #
This is related to, but distinct from #1. The difference is that even if you have a perfectly groomed set of target directories with no wasted detritus from earlier builds, cargo's working set is absolutely massive. The Rust project I spend the most time in is Bifrost, which has a normal target footprint of 52GB. (I hope someone goes all Ward Cunningham on me and tells me how to cut that by a factor of five or ten, but as far as Fable and Astra can tell me, that's just the way things work here in Rust-land. Yes, we have line-tables-only enabled.)
And God help me if an agent decides to change feature or package sets, because those hash differently and cost up to another 52GB for every variant!
(Anyone who has coded with agents knows that they pay only a little more heed to directives like "don't change the damn feature sets" than they do to the evergreen "don't make any mistakes.")
Solution #2: ZFS
The only good thing about Rust binaries is that they are about 70% compressible with cpu-light zstd or lz4. So because apparently my purpose in life is to serve as a warning to others, I reformatted my mbx build volume three times:
First to XFS, because ext4 is stuck in the 90s and still doesn't have reflinks, which is what lets mbx share a single instance of Cargo artifacts. (Although there's some fine print where mbx can use hardlinks on ext4 if you're willing to live with some compromises.)
Then to Btrfs, because XFS doesn't support compression.
Finally (an hour after switching to Btrfs) to ZFS, because Btrfs is a broken sack of sadness for any workload with concurrent writes, which apparently is a scenario that Btrfs devs think is unusual enough to leave unaddressed for years.
Now my workstation's 3TB physical volume can hold an estimated 10TB of Cargo output, which with mbx keeping clean is enough for even an extremely excitable crew of agents to not be able to fill up.
- You might think, as I did: why do you need compression if mbx reflinks solve the space problem? The issue is that reflinks only help for exactly identical output, which is about 7% of my build outputs. Even full block-level deduplication with ZFS only gets me up to 11%, because the first inserted or removed byte in the binary means that everything afterwards is a unique block. So compression is really your only hope.
Mine #3: rustc is slow #
Yes, I did go into this one with my eyes open, but that doesn't make it less of a problem when I spend 4x longer waiting for Cargo than I do waiting for LLM inference. (That is my actual production ratio from my harness logs.) I don't really have a long story here, everyone knows it's slow, that's just the price you pay for zero-cost abstraction. Nightly gives you something like 10% back in exchange for unpredictable bugs in your compiler; no, thanks.
Solution #3a: mold
I'm told that the default rust-lld is a lot faster than in the bad old days, but mold is still 50%-100% faster. So you should use mold (which mbx has a convenient hook for), at least until wild gets incrementalism. But faster linkers don't solve the main problem, which is rustc itself.
Solution #3b: Mjolnir
Mjolnir is a control plane for coding harnesses that lets you move your session across profiles (including heterogeneously, from Claude to Codex to DeepSeek) and across machines ("targets"). So you can spread those hungry, hungry rustc-building sessions across all your machines, or burst into EC2.
Quantity has a quality of its own #
These problems are not new, but twenty agents running builds concurrently is a scenario that none of our infrastructure was designed for. I'm glad to see that a new generation of tooling is being created to solve the problem of working in Rust at agent scale.