{"slug": "reading-the-code-is-the-hard-part", "title": "Reading the code is the hard part", "summary": "Rewind VM, a deterministic virtual machine, found a race condition in `git notes` in which two concurrent `git notes add` commands on different commits can lose one note, with 15 of 32 Rewind schedules violating an exit-zero property and 25 of 30 runs losing a note on a normal Linux machine. The tool's developer reports that `rewind check` narrows each failure to a single step, but reading the code to understand the failure remains the hard part, and pairing deterministic traces with AI agents that read the code produced fixes for the bugs found. An upstream patch for the `git notes` bug was submitted to the Git mailing list.", "body_md": "I have been stress testing [Rewind VM](https://rewindvm.dev), a deterministic VM, to work through the kinks, bugs and [dog-fooding](https://en.wikipedia.org/wiki/Eating_your_own_dog_food) the user-experience. Even having developed it myself, I continue to be wild-eye amazed at the magic of what it can do.\n\nDespite the magic, the ease of claiming to have a bug “reproduced deterministically”, and the ease of narrowing it down to a single step, the hard part is still reading the code and understanding why it failed. Rewind can tell you what went wrong, but it cannot read the code.\n\nThis is a particular area that is ripe for AI assistance. Agents can already read the code and often find bugs. Unsurprisingly, when given a wealth of reliable data about what went wrong, they can pinpoint the bug in a way that is much more efficient than reading the code alone. The combination of deterministic runs and AI agents is incredibly powerful. 🪄\n\nFinding the bugs in Rewind is easy. `rewind check` replays a test, script, or binary under many different thread schedules until one fails, then narrows the failure down to the single step. That leaves you, though, with a step number and potentially a large codebase, one you are unfamiliar with. The hard part is reading the code and understanding why it failed.\n\nRewind can tell you what went wrong, but it cannot read the code for you. Rewind does however give you, *or an agent*, a lot of information about what went wrong. I pointed [Rewind VM](https://rewindvm.dev) at a cornicopia of tools, and unsurprisingly, it found bugs…*many bugs*. The best part is that it found them in a way that is reproducible and debuggable, which has led to the agents also easily providing the fixes as well.<sup>1</sup>1You can find all the bugs on the [bugs page](https://rewindvm.dev/bugs.html). \n\n**Aside:**\nNew to Rewind? Start with [a flaky build you only catch once](https://fzakaria.com/2026/10/03/rewind-vm-a-flaky-build-you-only-catch-once) and [dafuq is a deterministic VM?](https://fzakaria.com/2026/10/24/dafuq-is-a-deterministic-vm)\n\nThis is a small example of a bug that Rewind found in git, and how I used Rewind to find the bug and the line of code that caused it. The bug is in `git notes`, which attaches a note to a commit without changing the commit. The bug is that two notes, on two different commits, at the same time, can lose one of the notes.<sup>2</sup>2I submitted an [upstream patch](https://lore.kernel.org/git/20261028-fz-notes-concurrent-write-v1-1-3d513e1fdbb7@gmail.com/T/#u) for the bug as well. \n\n## [§](#the-bug)The bug\n\n[`git notes`](https://git-scm.com/docs/git-notes) attaches a note to a commit without changing the commit.\n\nWe can start with an idea for a simple test<sup>3</sup>3This test demonstrates the race condition in `git notes` when adding notes concurrently. We can easily build this as a Nix derivation using [`runCommand`](https://nixos.org/manual/nix/stable/#sec-runcommand). : create two notes, on two different commits, at the same time concurrently.\n\n```\ngit notes add -m \"note 1\" HEAD~0 &\ngit notes add -m \"note 2\" HEAD~1 &\nwait\n\n# both commands exited 0, so both notes should be there\ngit notes show HEAD~0 && git notes show HEAD~1\nbash\n$ rewind check --all --schedules 32 --property exit-zero .#notes2\nschedule   1: exited:1             3167 steps    run e5e5a1ecd0195f8f  violates exit-zero\nschedule   2: exited:1             3312 steps    run ea6889e56564e374  violates exit-zero\n...\n15 of 32 other schedules violated a property\nstep 2682 decides it: a request to switch threads there makes the run violate exit-zero\n```\n\nWhen executed by Rewind, fifteen of thirty-two schedules lose a git-note, yet every `git notes add` still exits successfully.<sup>4</sup>4On a normal Linux machine, 25 of 30 runs lose one too, so this is not a Rewind-onlybug. \n\nNow what?\n\n## [§](#ask-the-runs-not-the-code)Ask the runs, not the code\n\nA run in Rewind is a full trace of the build: every process, every file it wrote, and any file’s contents at any step, that you replay to any point in time. **That is a lot of data**. Rewind does make it a little simpler to comb through the data though that help narrow the culprit.\n\nNone of those three commands knows anything about git, locks or races. They work on any program.\n\n### [§](#what-ended-differently)What ended differently?\n\n`rewind diff --outputs` compares the files the two runs left behind. This is like [diffoscope](https://diffoscope.org/) but on steroids 💉: it can compare any two runs, and it can compare the files at any step in the run, not just the final outputs.<sup>5</sup>5I have a [lengthier post](https://fzakaria.com/2026/10/23/who-wrote-this-byte) about how Rewind can help make your build bit-for-bit reproducible. \n\n``` bash\n$ rewind diff --outputs 94c59fdf987c4944 ad0de5aec5a0be2d\n3 of the 30 files the runs left behind differ:\n  /build/r/.git/logs/refs/notes/commits: 36 of 465 bytes differ, first at byte 351\n  /build/r/.git/refs/notes/commits: 36 of 41 bytes differ, first at byte 0  /nix/store/014fz1lj7hmk5h3rzcaw0w4576md3bg9-git-notes2: only in the right run\n```\n\nOur clue is that `.git/refs/notes/commits` is the ref that points at the latest notes commit, and it points at different commits in the two runs.\n\n### [§](#who-wrote-it-and-what-did-it-read)Who wrote it, and what did it read?\n\n`rewind blame` takes two runs and a file:\n\n``` bash\n$ rewind blame 94c59fdf987c4944 ad0de5aec5a0be2d /build/r/.git/refs/notes/commits\ncommits differs at byte 0. git notes (pid 56) wrote that byte at step\n2,751 in run 94c59fdf and at step 2,717 in run ad0de5ae.\n...\nLikely cause: git notes (pid 56) read /build/r/.git/refs/notes/commitsbefore git notes (pid 55) changed it at step 2,721 in run 94c59fdf,and after that change in run ad0de5ae. It read an older version ofthe file in run 94c59fdf.\n```\n\nIn this example, Rewind has given us pretty close to the smoking gun. The second `git notes add` read the notes before the first one had saved its own, then saved its version over the top: a read-modify-write race.\n\nWe can visualize this exact step in the Rewind GUI at the deciding step: both `git notes add` processes are alive (top box), and the first is halfway through saving, with its lock file written but not yet renamed into place (bottom box).\n\n### [§](#which-line-wrote-it)Which line wrote it?\n\nIf we have access to debug symbols, which Nixpkgs provides,  `rewind where` shows where in git’s own code a process was at a step. Here is pid 56 when it wrote the ref:\n\n``` bash\n$ rewind where 94c59fdf987c4944 2754 --pid 56 --frames 8\nprocess 56 (git), thread 56, at step 2754\n#1 rename_tempfile (tempfile.c:347)\ncalled from #2 commit_lock_file_to (/build/source/lockfile.h:364)\ncalled from #3 commit_lock_file (lockfile.c:358)\ncalled from #4 commit_ref (refs/files-backend.c:1889)\ncalled from #5 files_transaction_finish (refs/files-backend.c:3381)\ncalled from #6 ref_transaction_commit (refs.c:2806)\ncalled from #7 refs_update_ref (refs.c:1592)\ncalled from #8 commit_notes (notes-utils.c:64)\n```\n\nWe can then look at the code at a very specific line, and try to understand why it failed. \nIn this case, [`notes-utils.c:64`](https://github.com/git/git/blob/8d1b215c01512b9658d4525c54ff9b9245f9a7e8/notes-utils.c#L64) updates the ref without checking that it still points where it did when the command loaded the notes with a lock.\n\nThree questions, three commands, and I read one line of git. 🤓\n\n## [§](#who-reads-all-of-this)Who reads all of this?\n\nThe git notes failure is 2,936 steps and 1,937 events in one run, and `rewind check` made 33 runs. Going through one run is a lot of work, and going through all thirty-three is unrealistic.\n\nRecently, a friend told me AI has been really useful for parsing the [CAN bus](https://en.wikipedia.org/wiki/CAN_bus) of cars. The CAN bus is a firehose of messages from the car from every sensor, every button, every switch, and every light. LLMs have made managing the firehose of data easier.\n\nI see the same emergence in Rewind. The run is also a firehose of events, and similarly, LLMs can help parse the firehose of events and find the bugs. Telling an LLM to investigate a bug with Rewind has been very effective. Narrowing the bug down to a single step and source lines, is especially helpful for an LLM as it keeps the context small and focused. At that point, the LLM can read the code and understand what went wrong, and very likely suggest a fix.\n\nLLMs have a lot of stochasticity built into them. I have found that pairing them with more deterministic technologies such as [Nix](https://nixos.org/) and Rewind makes them more effective. I’m very optimistic about the\nleveraging the combination of deterministic runs and LLMs to find and fix bugs in large codebases.\n\nLLMs induce many of these bugs but now they can also fix them. 🙃", "url": "https://wpnews.pro/news/reading-the-code-is-the-hard-part", "canonical_source": "https://fzakaria.com/2026/10/28/reading-the-code-is-the-hard-part", "published_at": "2026-10-28 17:00:00+00:00", "updated_at": "2026-11-12 05:23:03.525457+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools"], "entities": ["Rewind VM", "git notes", "Git", "Nix"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/reading-the-code-is-the-hard-part", "markdown": "https://wpnews.pro/news/reading-the-code-is-the-hard-part.md", "text": "https://wpnews.pro/news/reading-the-code-is-the-hard-part.txt", "jsonld": "https://wpnews.pro/news/reading-the-code-is-the-hard-part.jsonld"}}