For this benchmark, I asked the models to design a basic Unix-like file system including the superblock, inode table, and data blocks, and then implement a function to read a file given its path.
The Architectural Phase #
When it comes to the initial design, Claude 3.5 Sonnet was significantly more methodical. It didn't just list the components; it explained the offset calculations for the disk image. GPT-4o provided a correct high-level overview but skipped the actual math required to map a file ID to a physical sector on a virtual disk, which is where most implementation bugs happen.
Claude 3.5 Sonnet: Precise block mapping, clear separation of metadata and data, and a logical layout for the inode table.GPT-4o: Strong conceptual understanding but lacked the granular detail needed for a direct C implementation.Surprisingly efficient code, though it tended to oversimplify the directory structure to save tokens.DeepSeek-V3:Great at explaining the theory, but the actual structural layout was slightly inconsistent across different prompts.Gemini1.5 Pro:
Implementation and Code Quality #
The real test was the C implementation. I focused on the read_file
logic, which requires traversing the directory tree and resolving inodes.
Claude 3.5 Sonnet produced code that was almost compile-ready. It handled the pointer arithmetic correctly and included necessary bounds checking to prevent buffer overflows—something often missed by AI. GPT-4o's code was clean but had a logic error in how it handled the indirect blocks, which would have led to a kernel panic in a real scenario.
Here is the general structure Claude suggested for the inode definition, which I found to be the most robust:
struct inode {
uint32_t mode;
uint32_t size;
uint32_t blocks[12]; // Direct blocks
uint32_t indirect_block;
uint32_t uid;
uint32_t gid;
};
Final Verdict on AI Workflow #
If you are using these as a practical tutorial for systems programming, Claude is currently the gold standard for low-level logic. It treats the problem like a systems engineer rather than a general-purpose coder. GPT-4o is still fantastic for brainstorming the high-level AI workflow, but for the actual deployment of a system-level project, the precision of Sonnet saves a lot of debugging time.
For anyone starting a deep dive into OS internals, I'd suggest using Claude to draft the memory map and then using GPT-4o to generate the documentation or test cases. This hybrid approach maximizes the strengths of both models.
Next Remembrane lets you run agent memory from a single SQLite file →