We do many conversions of AI models from the reference PyTorch implementation to GGML and C++. The reason is GGML/C++ produces a relatively tiny package we can run almost anywhere. The performance usually matches or exceeds PyTorch with a fraction of the dependencies.
Personally I despise C++; can’t stand it. I don’t have to write it anymore though, LLMs do it. I sit above a layer of abstraction and only descend into the code to avoid worst case scenarios in much the same way I would randomly go read the code for a library I depended on before AI.
Someone asked if we had considered doing these ports in Rust. I am well aware of Rust, got excited about it in the past, but had not seriously considered it because GGML is written in C++.
In recent years I experimented by comparing Rust against Go and it was a slam dunk victory for Go on the project I was doing. Neither myself, LLMs or the person advocating for Rust seemed to understand how to solve basic problems without tripping over the type system.
However I do like to explore options and it had occurred to me already that an experiment between Rust and C++ could be interesting, so having someone ask for it is the push I needed to burn some tokens.
I in fact ended up doing a series of A/B experiments, none of which are very scientific, but I want to test realistic scenarios. Creating neat test environment tends to increase in difficulty the more realistic you try to make it.
The first experiment was to pit C++ with GGML Vs Rust with Burn. They both had to reach parity, in terms of correctness and performance against the PyTorch implementation of FLOAT (it animates portraits given voice audio to make it appear they are talking).
The final experiment was to create a pure CPU only implementation using SIMD libraries in C++ and Rust. Removing the confonders introduced by pitting GGML against Burn.
The original suggestion was to use GGML from unsafe blocks in Rust, but I thought that this might put Rust at a disadvantage, nullifying its safety properties. Rightly or wrongly I decided to go with Burn instead and I can say right now Burn is great. Except that is on CPU, it’s terrible on CPU. On GPU though it is competitive with GGML and PyTorch on inference and it does training too.
I have published Fable’s and ChatGPT’s analyses in the [rust-vs-cpp-analysis
repository](https://github.com/richiejp/rust-vs-cpp-analysis), but here I’ll give my opinions and highly condensed
version of events.
I asked Claude to create 3 independent sub-agents; one to do an analysis of the PyTorch implementation, create a parity harness to do layer by layer numerical comparisons and do some Rust research to level the playing field with C++.
You see, the thing is, the C++ agent would have a fair more prior art to go on, because we have done a bunch of these conversions and it helps to give access to those prior conversions. I’m not going to withold that info because I want results. The best I can do is do some extra work upfront to try and convert lessons learnt on C++ projects into lessons for a Rust proejct.
The other two subagents are the Rust and C++ implementors. They ran until they reached correctness and performance parity with the PyTorch implementation.
Initially I thought I had walked into a decisive result proving Rust’s greatness. The Rust agent reported about 50% of the token usage while achieving the same performance. However it turned out the Rust agent benefited from some work the C++ agent did to debug the parity harness.
The primary Fable agent also decided to do the GGML implementation in CUDA and the Rust one in Vulkan. Then blamed me for asking that GGML use CUDA and Burn use Vulkan. I don’t recall asking for that, but in any case it is a confounder, but not a big one as it turns out. Regardless of whether we use Vulkan or CUDA, fused kernels are required in GGML and something similar in Burn to get parity with compiled PyTorch.
After some iterations trying to get the Rust and C++ ports to be equivalent, I decided to do something more drastic as a final test.
I asked to create two new ports, both in new repositories, but using lessons learned from the previous attempts. This time the ports would be CPU only and use SIMD libraries. I demanded that they be statically compiled and allow cross compiling to ARM. Do zero allocations during computations and a bunch of other stuff.
Again it initally appeared that Rust had achieved a victory, but it turned out that the rust agent simply decided not to create a C API. Possibly because I didn’t explicitly ask for it, but of course there should always be a C API/ABI otherwise only other Rust code can consume the library.
The Rust port was also a bit slower, because the agent simply decided to stop with perf optimizations sooner. Maybe Claude doesn’t like optimizing Rust or more likely it’s just the random nature of LLMs coming through.
In any case after forcing the agent to complete the C API/ABI and get perf up to parity, the Rust advantage disappeared.
Finally I decided to do a couple of code reviews, one with Claude again and another with Codex. The argument in favor of Rust would predict that the Rust code would have less previously undiscovered bugs.
Both Rust and C++ were fuzzed from the beginning, but still the review found plenty of potential bugs in both. I asked both Claude and Codex if Rust’s and C++’s tools and features had been used to their fullest to prevent bugs and the answer was no. Not even close. The agent just didn’t decide to do it on the first round.
The agent had made some spurious uses of unsafe, but actually most of the uses seemed to be valid. Removing the unnecessary unsafe blocks wouldn’t have made a lot of difference to the analysis.
So essentially both languages have more intrinsic or extrinsic features available to combat bugs. Both agents failed to use all of those features and both agents ended using similar amounts of tokens and time.
From what I can gather there is no difference to an LLM between Rust and C++. The main difference between these lanauges is that Rust has some features built into the compiler whereas C++ relies on external tools. There are many problems with C++, but they get arbitraged with external tools like fuzzers, the address sanitzer, static anlysers and more ergonomic libraries. Rust is perhaps more prone to solving these issues in the language. In either case it’s not automatic that language features or external tools will solve the problems they can solve.
The surprising part is that solutions which are enforced by the language don’t really solve the original problem automatically. Let’s say you solve memory safety with a borrow checker, now you have a new problem which is working around the borrow checker. Except it is not really a new problem, you are still trying to solve the underlying problem, but with new constraints.
That isn’t to say the borrow checker can’t help solve the problem, it’s just that it requires effort to do so. The question is does it require less effort than testing the code with the address sanitizer on, using smart pointers, a GC or using a combination. Possibly all of these techniques are mathematically and computationally equivalent when carried out by an LLM, except that it makes a difference when the computation occurs; in production, test time or compile time.
Compile time and test time are practically the same it seems, so C++ and Rust are the same. They both move computation from runtime to development.
I suppose the fundamental problem is that you have to model the real world task the software is trying to orchestrate. You have to express this model in the programming language’s primitives. The real world, being as varied as it is, means that any general purpose programming language lacks primitives that abstract or verify an arbitrary model.
Even if the language does contain primitives to help with a large part of your model. You have to know how to use those features, which is not free. Especially when you view it through the lens of agentic codimg. You can put a token count on finding the correct primitives to express the model in.
Intuitively I think one language is better, but I haven’t been able to disprove the null hypothesis. That’s most likely because there are things that matter a lot more than whether you write your loops over arrays in C++ or Rust.
Now both these languages are created for humans and there is the question of what happens if you systematically optimize a language to reduce token usage while completing practical tasks. It could be a deadend where the language always collapses into degenerate solutions or it could lead to something much better.
My feeling is you’d hit diminishing returns quite quickly because the language would need to expand to encode solutions for an increasing number of problems, making recall the bottleneck. However the initial gains could be large.