How pgrust was built: four attempts to rewrite Postgres in Rust with AI

Developers spent $100k and four attempts to rewrite PostgreSQL in Rust using AI, resulting in pgrust, a 1.8M-line idiomatic Rust codebase. The project aims to enable safer AI-driven feature additions to Postgres, with a fifth version already outperforming Postgres in transactional workloads and matching Clickhouse in analytical queries.

tl;dr After four attempts and dropping $100k, we succeeded in using Opus to rewrite Postgres in Rust. We defined a number of skills and were able to build a repeatable process to rewrite Postgres files into Rust. After running up to 40 concurrent subagents for days, the end result is the 1.8M lines of idiomatic rust code that make up pgrust. Over the past three months, I’ve been working with my friend Jason Seibel on building pgrust. The idea behind pgrust is we want to show that by leveraging AI, it’s possible to can build a much better version of Postgres. Unfortunately Postgres is written in C and AI still makes a lot of mistakes. For this reason, we started by first rewriting Postgres in Rust. We believe that by having Postgres in Rust it will become a lot easier for AI to safely add new features to Postgres. This thesis is starting to be proven true. Currently in development we have a new, 5th version of pgrust that is faster than Postgres for transactional workloads and is as fast as Clickhouse when it comes to analytical style queries. For context, Clickhouse is hundreds of times faster than Postgres when it comes to analytical queries. Let us walk you through each of our attempts and what we learned from each approach. We figure for those interested in completely rewriting legacy systems, this post will serve as a useful guide as how to go about such a rewrite of your own. Attempt 1 – pgrust-og The first version of pgrust, pgrust-og was initially was going well. It got very close to completion, but ended up having fundamental issues that made it unsalvageable. What I consider to be the root issue is we approached pgrust-og by porting one feature at a time for contrast in later versions I worked on porting one file at a time . There were roughly three different phases to the pgrust-og lifecycle: - Foundation - Features - Last Mile Foundation Initially I worked on laying out the core foundation of the database. Postgres has a couple of key systems that everything else relies on. To name a few: - The query parser – The parser converts your SQL query into a representation Postgres can understand what data you’re asking for - The query planner – The query planner takes the output from the parser and produces a query plan. A specific way for Postgres to retrieve the data you’re looking for - The executor – Once Postgres has a plan for how to fetch your data, the executor is the one that actually fetches it Along with these three, there are maybe 15 other core subsystems of Postgres. Over a couple of days I built a new database in Rust that architecturally had the same subsystems as Postgres. This step seemingly went very smooth. For each component, I already had a pretty good sense of how it should work. I gave Codex 5.4 access to the Postgres codebase and would ask it to come up with a plan for rewriting X subsystem of Postgres into pgrust-og. I would then review the plan and make sure it covered everything I wanted and would give Codex feedback on the plan where it fell short. From there I had codex execute the plan. Given these were the core systems of Postgres, I read the code thoroughly to make sure I understood how they worked. This approach worked pretty well and after a couple of days I had knocked out pretty much all the core pieces. Features, Features, Features Once I had the core scaffolding in place, I started working on adding all the features that Postgres supports. The easiest way to do this was to look at the Postgres regression tests and look at all the features they were testing. Now the thing about Postgres is it has a surprising number of features. When it comes to features, Postgres looks almost like a programming language’s standard library. Postgres has: - Over 300 different types including two different json types - Nearly 500 different SQL keywords - More than 3000 different builtin functions That’s a lot to implement Initially I was following the same approach I did when working on the foundation phase. The “problem” I was running into was that Codex was too good at implementing these features. For any individual feature, Codex is usually able to one-shot it. If you want to try it yourself, ask Codex to write a JSON parser for you. The problem I was running into was I would ask Codex to come up with a plan and then wait hours for it to implement the plan. I spent most of my time “programming” being idle This is when I realized that it might be possible to run multiple agents at a time. I decided it was time for a new approach. Instead of working on one feature at a time, I would try multiple. The next day I installed Conductor a tool that let’s you manage multiple agent sessions simultaneously. You can see the moment I started using Conductor in the git history: Surprisingly, this approach worked Because each feature was relatively independent of each other, multiple Codex agents could work on them independently. I eventually scaled up this approach and started spinning up as many as 20 agent sessions each. Usually I would look at which test files were left to work on and then spin up a session for each file. From there I would see what the current failures were and have Codex work on completing that file. There were two issues with this approach. First I started burning through my Codex quota. I had to get multiple Codex accounts and switch between them in order to stay on the subscription plan. Second, I started maxing out CPU on my computer. Rust builds end up being somewhat cpu intensive and when you run enough builds, even my M4 Macbook Pro couldn’t handle them all. I ended up setting up GitHub actions, just so I could run tests off of my computer. Last Mile Over the course of about four weeks I was able to get 96% of the Postgres tests passing. That’s when progress started to slow down and things started to grind to a halt. The big obstacle came when I tried to get all the Postgres planner tests to work. The postgres planner is the most complicated part of Postgres. It’s 50k lines of very nuanced logic that figures out the optimal way to fetch the data your SQL query is asking for. In short, it evaluates every possible way to fetch your rows out of the table, every possible way to perform the joins, and applies a advanced cost model to figure out the best way to plan the query. While pgrust-og had a planner and had all the functionality that postgres does, the way the planner was integrated into the rest of the codebase was subtly different. This led to an “organ transplant failure”. To give an example of one of the failures, let me show you the differences in how pgrust-og and pgrust would represent a plan for the query: SELECT name FROM users WHERE age 30 . In Postgres this ends up being represented as a single plan node: SeqScan { .scan.scanrelid = 1, .plan.targetlist = name , .plan.qual = age 30 , } While in Rust, it ended up being represented as three plan nodes: Projection { .columns = name , .input = Filter { .predicate = age 30 , .input = SeqScan { .table = users, }, }, } On top of that in Postgres the tableis referred to by an index for the query while in Rust it’s referred to by a string. While these look like small differences, they end up being massive. In C lots of functions will take a sequential scan node with a filter. In rust, those functions would sometimes need to take a sequential scan, sometimes take a filter node, and other times need to take a projection node. Fix just this one issue would be difficult as it would require rewriting everywhere where a seqscan, filter, or projection node is used. In total, dozens of slight differences like this made it infeasible to get all the tests passing with pgrust-og. After struggling to get the planner to work I realized just how different the internals of pgrust-og were versus Postgres. If there were subtle differences in how the planner worked, who knew how many subtle differences there were in things like replication and backups. I realized it would be impossible to get pgrust-og into a production ready state. That’s when I decided it was time for a new approach. Enter c2rust. cost: 8x codex accounts for 1 month = $1600 Attempt 2 – c2rust For our next attempt we wanted to take a different approach. Instead of trying to go feature by feature, we wanted to see if there was some way we could transpile the Postgres code from C to Rust. That way each Rust code would match up line by line with equivalent Postgres code. It would be much easier to audit the code and show that I was doing things exactly the same way Postgres was doing. c2rust is a open source project that will transpile your C code to unsafe rust. While that technically would be “Postgres rewritten in Rust” it wouldn’t help me further my goal of making it easy to add new features to Postgres. The idea was that we would use c2rust to unsafe rust and from there gradually rewrite the unsafe rust code into safe rust. After toying with c2rust for two days, Jason managed to get it to completely transpile the Postgres C code to 5.3M lines of unsafe Rust. That resulting code completely passed the Postgres test suite and passed 100% of the Postgres regressions suite and had similar performance to Postgres. That’s not even the coolest part. The c2rust version of pgrust was ABI compatible with Postgres. That meant things like Postgres extensions would be compatible with Postgres. We didn’t even think that was possible At that point, we thought the hard part was over. Now all we had to do was have Codex refactor the code into idiomatic Rust. It turns out the hard part was not over. When we tried to refactor the unsafe rust code, we found it would take hours to convert a single function. To show you why let me show you what some of the output of c2rust looks like: pub unsafe extern "C" fn makeAlias mut aliasname: const ::core::ffi::c char, mut colnames: mut List, - mut Alias { let mut a: mut Alias = newNode ::core::mem::size of::<Alias as size t, T Alias as mut Alias; a .aliasname = pstrdup aliasname ; a .colnames = colnames; return a; } This looks closer to idiomatic C code than it does to idiomatic rust Every value is represented as a pointer which more or less means all the code has to be unsafe. In order to convert a single function from unsafe rust to safe rust you had to change all the types the function accepted to be more idiomatic Rust types, change all the callers of the function, and change all the callees of the function. That meant changing one function could necessitate changing 100s of functions to make it idiomatic. We tried a couple of different approaches. Could we define two versions of every function one that’s safe, one that’s unsafe? Could we make the internal functions of a package safe and leave the external functions unsafe? We were never able to get any of these incremental approaches to work that well. That’s when I had a new idea – what if instead of trying to rewrite the c2rust code incrementally, I instead created a new codebase and piece by piece, rewrote the c2rust code from unsafe Rust to idiomatic Rust. That led to the creation of pgrust-idiomatic. Attempt 3 – pgrust-idiomatic The output of c2rust was ~1000 individual Rust crates, with roughly one C file mapping to one Rust crate. The idea for pgrust-idiomatic was to rewrite the generated c2rust code, crate by crate. While I had considered doing a similar approach with the original Postgres code, I had brushed it off because it had seemed too high risk. If I’m rewriting it crate by crate, the code won’t work until I rewrite all of the crates. At the same time, Jarred Sumner had approached the infamous rewrite of Bun from Zig to Rust by going file by file. I figured if Jarred was successful with it, maybe I could be successful with it to. What I started doing is asking my coding agent to identify a crate we could port next, have it port the crate, and then have it audit the results. Note that although I was primarily focusing on the c2rust code, Codex had access to the original Postgres code as well. This was the beginning of what people now call a “loop”, albeit a manual one. As I was going through this process, I was constantly giving Codex feedback. The most common shape of feedback was pushing Codex to eliminate uses of unsafe. Once the loop started to run smoothly and work consistently, I had Codex encode all the feedback I gave it into skills. I ran prompts like this: can you write a skill in this project $find-next-crate which will find the next crate we should work on – it should find a crate we have not ported and has no dependencies on crates we have not ported This time I approached the project a little bit differently. This time, I set up my “agentic infrastructure”. I started out by defining my Agents.md and defining some skills to start the project. I knew that if I could get Codex to be good at rewriting one crate into idiomatic Rust, I could massively parallelize that and quickly get to completion. I defined three skills: find-next-crate – This was a skill that explained to codex how to determine which crates we were yet to port and we had already ported all the dependencies of those crates port-crate – This would take one crate and rewriting it from C to Rust audit-crate – This would compare the new pgrust-idiomatic crate against both the c2rust code and the original C code and make sure things lined up. It would also make sure the code was idiomatic Rust and there weren’t any major issues with it. It would come back with any findings and if there were issues, I would dig in Initially I tested the skills on a coupleof crates. I gave Codex some feedback and have it update the skills. Once the skills started working consistently, I started to scale up the process. I started using Conductor to port the crates in mass. Initially this was working great I was making consistent progress. That was until I hit the Postgres regex implementation. Here’s a paraphrased part of my session with Codex: Me:Here’s the entire regex crate – about 16,000 lines of machine-translated C. Rewrite it into idiomatic Rust. Model: five minutes and 185,000 tokens later Done. I’ve rewritten 11 functions. Me:…11? How many functions are there? Model:228. Me:Keep going until all the code has been ported. Model: 334,000 tokens later I’ve rewritten 11 functions. I don’t know what it was, but GPT-5.5 was really struggling to port the regex code. There were some other approaches I tried, suchas porting function by function or using a cheaper model and then having a more expensive model clean the code up, but I wasn’t making any real progress. I spent about two days trying different approaches to rewrite the regular expression code when Anthropic released something new – Claude dynamic workflows. If you aren’t aware, Claude dynamic workflows are a way for you to run a massive amounts of subagents in a structured way. They’re kind of like map reduce, but with coding agents. They are what Jarred Sumner used to rewrite Bun from Zig to Rust. To give a simple example, here’s what it looks like when you ask Claude: Can you spin up a dynamic workflow of 10 agents each of which write a document on how to implement a different part of a database? Then can you have 10 agents review each of those documents I decided to try dynamic workflows on the regex code. I turned on ultracode in Claude Code which has Claude try to use dynamic workflows and asked to it port over the regular expression code. It spun up 4 subagents, one for each file in the Regex crate, spun up a couple of reviewer agents, a couple of fixer agents, and finally an integrator agent. I was watching the code being written and it seemed like Claude was making real progress. After 30 minutes, Claude had completed porting the regex code. I dug through the code and it appeared as if all the code had been ported. I asked Claude how many functions were in the Rust regex code and it confirmed that it had ported all 228 functions. I was astonished. A task that I was struggling with for days had suddenly been taking down by dynamic workflows. Now there was just one problem with dynamic workflows – token usage. Because you’re running dozens of subagents at a time, you burn through the 5h usage limits in minutes. I was churning through the usage limits so fast that it was infeasible for me to have enough subscription accounts. If I wanted to use dynamic workflows for the rest of the project, it would cost an order of magnitude more than the Claude Code subscriptions. Now I may be a little crazy, or maybe very crazy, but I thought if I could complete pgrust for under $100k, it would be worth it. I know that sounds absurd, but when comparing to the cost of Postgres, Postgres probably cost on the order of hundreds of millions of dollars in R&D effort to create. I thought if I could create a new database that was compatible with Postgres for under $100k, that would be worth it. So going forward, I used dynamic workflows to port a bunch of crates in batch. I could go to codex and ask it for what files was it currently possible to port and then I would spin up a dynamic workflow to port all of them at once. At any given time, I would have 3-6 different dynamic workflows porting different parts of the Postgres code. I often had over 100 subagents running at a time working on porting a large number of files from unsafe Rust to idiomatic Rust a time. And it was working I was moving many times faster than I was before and from what I could tell it was all working reasonably well. Now you remember how at the beginning of the section I said that doing a file by file approach was really risky. Yeah, well… I ended up running into a major issue. To explain, let me cover one major challenge with migrating from C to Rust that I haven’t covered yet – circular dependencies. In C, you’re allowed to have circular dependencies. On the other hand, Rust crates forbid the creation of circular dependencies. This had created a pretty big challenge for me. The Postgres source code has a ton of circular dependencies. To show you just how tightly integrated the Postgres codebase is, here’s what the dependency relationships between the different Postgres subsystems looks like: A red line means the both subsystems is bi-directional and any other color means the relationship is one way. It’s safe to say that there’s no easy way to just break the circular dependency. Up until this point, I had been handling circular dependencies with a form of dependency injection Claude calls “seams”. The idea was when crate A wanted to call crate B but crate B wanted to call crate A, we would create a seam that would break the circular dependency. Specifically crate A would define a global function variable that represented the function it wanted to call in crate B. Later when starting up pgrust, that function pointer would be updated to the respective function. This meant B could depend on A without A depending on B and both crates could call functions in the other respective crate. To give an example, let’s say we have the following C code / ---- executor.c ---- / include "executor.h" include "storage.h" void executor run { storage read ; } void executor report progress { } / ---- storage.c ---- / include "storage.h" include "executor.h" int storage read { executor report progress ; } In C, this is perfectly valid. In Rust on the other hand it would look something like this with seams: use std::sync::OnceLock; allow non upper case globals static report progress: OnceLock<fn ; = OnceLock::new ; pub fn register report progress f: fn { report progress.set f .expect "seam registered twice" ; } pub fn storage read - i32 { report progress.get .expect "seam not registered" ; 0 } pub fn executor report progress {} pub fn init { storage::register report progress executor report progress ; } pub fn executor run { storage::storage read ; } // main.rs fn main { executor::init ; executor::executor run ; } This had appeared to work fine. As far as I could tell all the code was ported, it compiled, and all I had to do was initialize all the seams. When I told Claude to wire up the seams, for some reason it struggled and was unable to do so. When I dug in deeper, I found a major issue: in many cases the type signature of the seam diverged from the type signature of the function definition. For example, Postgres has a Relation type that is defined as: typedef RelationData Relation The obvious way to translate Relation into Rust it as &RelationData. Unfortunately for me, Opus had sometimes translated Relation into &RelationData and other times it defined Relation as: pub type Relation = usize; and other times Opus translated it as: pub type Relation = oid I don’t quite now why Opus sometimes translated it one way and sometimes translated it another way, but now I was stuck with it. Large parts of the pgrust-idiomatic would use one definition of Relation and other parts would use a different definition. In total, there were around ten thousand seams that diverged from their implementation. Solving this problem was just as hard as solving the c2rust refactoring problem. In order to fix one seam, I had to completely remove one of the incorrect type definitions. I tried for several days to save pgrust-idiomatic, but it was just too token intensive to fix all the seams. It was around this time that Claude Fable was released. I tried throwing Fable at the problem but it was to no avail. Even Fable couldn’t recover pgrust-idiomatic. I started questioning what do I do? Is there any way to save pgrust-idiomatic? Does it just make sense to start over again? Given I had what appeared to be a new superior model, I wanted to see just how capable Fable would be at building it’s own version of pgrust. That resulted in pgrust-fabled. cost: Lots of Claude dynamic workflows – $50k Attempt 4: pgrust-fabled Spoiler: pgrust-fabled was not built by Fable With Fable in hand, I created a new codebase and got to it. This time I approached things a little bit differently. This time I gave Fable all the relevant context to the project and gave it access to all the code across all the previous versions of pgrust. I also gave it access to some tech debt logs I was keeping track of while building previous versions so it would know exactly what issues it needed to avoid. Based on my lesson from building pgrust-idiomatic, I changed two things. First I changed the skill for finding the next crate to prioritize implementing seams. That way if there were any seam divergences, they would be detected quickly. Second, I updated the audit skill and gave it many more things to look for. In particular, seams that had diverged. Once I had laid out the groundwork, I kicked off the build. I had proven out Fable was working with a couple of crates and then started doing a number of dynamic workflows to port code. I got maybe 50k lines of code in, before Anthropic shut off Fable. I was stuck with Opus again. This time, I kept going with Opus. I regularly ran the audit skill and was consistently making sure all the seams were wired. I would periodically find an issue, but I made sure to nip it in the bud as soon as I discovered it. There were some minor hiccups along the way, but everything kept going smoothly. It kept going smoothly all the way until I ported all the code over. By that point it took a little bit of work to wire all the seams together, some additional work to fix the bugs, and pgrust-fabled was able to pass 100% of the Postgres regression suite I had finally accomplished my goal of getting pgrust into idiomatic Rust. This is the version of pgrust that’s available publicly. Cost: Lots of Claude dynamic workflows + Fable – $50k What’s next? The big issue with pgrust-fabled is it’s really slow. It’s about 8x slower than Postgres. As it turns out, when you try to write the most idiomatic Rust possible, something has to give. That’s why we’re currently working on the next version of pgrust we’re internally calling pgrust-fast. So far we’ve put in $200k into pgrust-fast on top of the $100k we put into pgrust-idiomatic. pgrust-fast has been completely built by Fable and was built with speed in mind. We’ve had Fable attempt to optimize every individual bit of code of pgrust-fast. On top of that we’ve completely revamped a number of Postgres components including: - A new vectorized executor - A new JIT compiler that replaces LLVM and cuts latency of JIT compilation from 50ms to 5μs - A scheduler that dynamically adjusts the prioritization of queries. Long running queries get throttled allowing for faster queries to run with less contention Altogether the optimizations we’ve made have made pgrust-fast over 50% faster than Postgres on trasactional workloads and about as fast as Clickhouse for analytical ones. Keep an eye out because we plan on releasing pgrust-fast within the next two weeks.