{"slug": "i-came-i-prompted-i-left-part-1-building-a-hypervisor-for-the-macbook-neo", "title": "I Came, I Prompted, I Left Part 1: Building a Hypervisor for the MacBook Neo", "summary": "A developer building a custom hypervisor for the MacBook Neo (A18 Pro) has achieved a milestone in porting Linux to Apple Silicon in weeks instead of months, using mostly unattended LLM loops to reverse engineer Apple's proprietary security monitor (SPTM). The hypervisor traces macOS hardware interactions to avoid copyright infringement, following the approach of the Asahi Linux project's m1n1 hypervisor.", "body_md": "# I Came, I Prompted, I Left Part 1: Building a Custom Hypervisor for the MacBook Neo\n\n**TL;DR:** To reverse engineer modern Apple Silicon Macs for the purpose of\nporting Linux, I built a custom hypervisor for the MacBook Neo and documented\nthe internals of Apple’s proprietary security monitor (SPTM). I have\nsuccessfully used this hypervisor to achieve a milestone that usually takes\nmonths in a few weeks. This was all enabled by (mostly) unattended LLM loops.\n\nContext: My goal is to port Linux to modern Apple Silicon SoCs, starting with\nthe MacBook Neo (A18 Pro). To do this, I need a way to understand how to\ninteract with the hardware *without infringing on Apple’s copyright*. The Asahi\nLinux project already has a solution for this problem: run macOS in a\nhypervisor (codenamed m1n1), watch how macOS interacts with the hardware, then\ncopy what macOS does without looking at Apple code<sup>[1](#fn:1)</sup>. For example, this is a\nsnippet of a MMIO trace showing PCIe init:\n\n```\n# [cpu3] [0xfffffe000bbf2338] MMIO: W.4   0x48c000004 (apcie[1], offset 0x04) = 0x0\n# [cpu3] [0xfffffe000bbf2068] MMIO: R.4   0x48f020000 (apcie[2], offset 0x20000) = 0xf7c030d0\n# [cpu3] [0xfffffe000bbf2338] MMIO: W.4   0x48f020000 (apcie[2], offset 0x20000) = 0xf3c030d0\n# [cpu3] [0xfffffe000bbf2068] MMIO: R.4   0x48f010000 (apcie[2], offset 0x10000) = 0x33000070\n# [cpu3] [0xfffffe000bbf2338] MMIO: W.4   0x48f010000 (apcie[2], offset 0x10000) = 0x23000070\n# [cpu8] [0xfffffe000bbf2068] MMIO: R.4   0x48f010010 (apcie[2], offset 0x10010) = 0x300c03\n# [cpu8] [0xfffffe000bbf2338] MMIO: W.4   0x48f010010 (apcie[2], offset 0x10010) = 0x310003\n# [cpu8] [0xfffffe000bbf2068] MMIO: R.4   0x48f010014 (apcie[2], offset 0x10014) = 0x300c03\n# [cpu8] [0xfffffe000bbf2338] MMIO: W.4   0x48f010014 (apcie[2], offset 0x10014) = 0x4000c03\n# [cpu8] [0xfffffe000bbf2068] MMIO: R.4   0x48f014000 (apcie[2], offset 0x14000) = 0x33000070\n# [cpu8] [0xfffffe000bbf2338] MMIO: W.4   0x48f014000 (apcie[2], offset 0x14000) = 0x23000070\n# [cpu8] [0xfffffe000bbf2068] MMIO: R.4   0x48f014010 (apcie[2], offset 0x14010) = 0x300c03\n# [cpu8] [0xfffffe000bbf2338] MMIO: W.4   0x48f014010 (apcie[2], offset 0x14010) = 0x310003\n# [cpu8] [0xfffffe000bbf2068] MMIO: R.4   0x48f014014 (apcie[2], offset 0x14014) = 0x300c03\n# [cpu8] [0xfffffe000bbf2338] MMIO: W.4   0x48f014014 (apcie[2], offset 0x14014) = 0x4000c03\n# [cpu8] [0xfffffe000bbf2068] MMIO: R.4   0x48f018000 (apcie[2], offset 0x18000) = 0x33000070\n# [cpu8] [0xfffffe000bbf2338] MMIO: W.4   0x48f018000 (apcie[2], offset 0x18000) = 0x23000070\n# [cpu8] [0xfffffe000bbf2068] MMIO: R.4   0x48f018010 (apcie[2], offset 0x18010) = 0x300c03\n# [cpu8] [0xfffffe000bbf2338] MMIO: W.4   0x48f018010 (apcie[2], offset 0x18010) = 0x310003\n# [cpu8] [0xfffffe000bbf2068] MMIO: R.4   0x48f018014 (apcie[2], offset 0x18014) = 0x300c03\n# [cpu8] [0xfffffe000bbf2338] MMIO: W.4   0x48f018014 (apcie[2], offset 0x18014) = 0x4000c03\n# [cpu8] [0xfffffe000bbf2068] MMIO: R.4   0x48f004000 (apcie[2], offset 0x4000) = 0x80300000\n# [cpu8] [0xfffffe000bbf2068] MMIO: R.4   0x48f020000 (apcie[2], offset 0x20000) = 0xf3c030d0\n# [cpu8] [0xfffffe000bbf2338] MMIO: W.4   0x48f020000 (apcie[2], offset 0x20000) = 0xf3c030d1\n# [cpu8] [0xfffffe000bbf2068] MMIO: R.4   0x48f020000 (apcie[2], offset 0x20000) = 0xf3c030d5\n# [cpu8] [0xfffffe000bbf2338] MMIO: W.4   0x48f020000 (apcie[2], offset 0x20000) = 0xf3c030d7\n# [cpu8] [0xfffffe000bbf2068] MMIO: R.4   0x48f020000 (apcie[2], offset 0x20000) = 0xf3c030df\n# [cpu8] [0xfffffe000bbf2338] MMIO: W.4   0x48f020000 (apcie[2], offset 0x20000) = 0xf3c030cf\n# [cpu8] [0xfffffe000bbf2068] MMIO: R.4   0x48f020004 (apcie[2], offset 0x20004) = 0x20\n# [cpu8] [0xfffffe000bbf2338] MMIO: W.4   0x48f020004 (apcie[2], offset 0x20004) = 0x21\n```\n\nWhile this trace looks opaque and intimidating at first, it actually contains lots of valuable information of how PCIe works on this platform. The existing Asahi Linux project, and drivers I have built/will build, are built from traces just like this. As a bonus, m1n1 also gives us a Python shell that we can use to probe a live macOS guest.\n\nThe problem is that the m1n1 hypervisor doesn’t work on M4 SoCs and above (including A18 Pro). That means my goal is simple: to boot to a fully functional macOS desktop under m1n1 to allow us to reverse engineer modern Apple Silicon.\n\nAs for why this problem hasn’t already been solved, a normal ARM system looks something like this:\n\nIt’s fairly easy to just stick a hypervisor under this; indeed, that’s what Asahi Linux already does for M1-M3, and the process works well. The problem is that on M4+, Apple changed things, so now it looks like this:\n\nOh boy, what happened here. We can see a lot of new components, chiefly,\nthere’s a whole separate privilege level (GL, Guarded Level) that runs\nalongside the regular ARM exception levels. Inside GL there’s *SPTM*, the\nSecure Page Table Monitor, Apple’s custom secure kernel that manages pagetables\nfor XNU, and *ExclaveOS*, essentially a full L4 kernel and userspace running\nalongside macOS. The entire GL side is completely closed off; I cannot run code\nin that context.\n\nThere are other problems too:\n\n- Many implementation defined (impdef) system registers come locked; I cannot write to them and thus cannot replicate their hardware side effects\n- Custom Guarded Execution Features (GXF) instructions that only run in GL mode used in SPTM\n\nIn order to virtualize macOS, all of these problems need solutions. We can\nmostly ignore Exclaves because there exist fallback paths at the cost of some\nhardware functionality, but SPTM is the key blocker <sup>[2](#fn:2)</sup>. SPTM implements a\n*lot* of functionality, allocating and managing many data structures shared\nwith macOS, and exposing a number of endpoints used by macOS to manage its\npagetables and that of the coprocessors.\n\n## Architectural Design\n\nThe primary thought in Asahi Linux on how to deal with this is to virtualize\nSPTM as a second guest under the existing m1n1 hypervisor, then intercept the\n`genter` calls to SPTM, forward them to the real SPTM binary, and proxy the\nresult back without ever having to look at Apple code. This is copyright\nfriendly, and removes the need to understand all the complicated stuff SPTM\ndoes.\n\nI was skeptical of this approach, and from what I’ve learned, I’m glad I didn’t\nfollow it. At a high level, SPTM is a GL2 binary that relies on a *lot* of\nApple proprietary hardware features that are disabled for us. The approach I\nthought of was conceptually simpler, at the cost of more development: just\nemulate SPTM in m1n1, because it makes use of the same memory and page table\nprimitives already in m1n1, without the Apple proprietary requirements.\nIn essence, I decided to make a tradeoff and do something I was sure would work\nat the cost of development time.\n\nThere’s only one, very large problem: to emulate SPTM, I have to know internal\ndetails of how it works, which currently no one outside of Apple knows<sup>[3](#fn:3)</sup>. I\ndecide to jump on the grenade, pull out Claude Code, and disassemble the SPTM\nbinary. This means that any code I write around SPTM is tainted and can’t be\nused by others. However, following a clean room process, I can write\ndocumentation on SPTM internals and then other people can use said\ndocumentation to build a clean room implementation.\n\n## The Setup\n\nSolving all of these problems would be a very difficult endeavor on its own, requiring significant fluency in assembly and a lot of rote work/head banging, so enter the unattended LLM loop. Starting the project, I was concerned that LLMs would be unable to work in this environment, being such a niche, low-level task, but I came to learn they were more than capable.\n\nThis was my setup:\n\nThe Mac Mini was the host; my agent was running on the device and had access to\n`macvdmtool`<sup>[4](#fn:4)</sup> which allowed it to reboot the device remotely. The MacBook\nNeo was the device being debugged, connected via its debug port to the Mac\nMini. The workflow we settled on after a few days was, essentially:\n\n1. Reboot the device with `macvdmtool reboot serial` .\n2. Invoke the hypervisor entrypoint `python3 proxyclient/tools/sptm_hv.py` with whatever arguments were required, including a timeout.\n3. Once the timeout triggered, capture as much debugging information as we could and interact with the live Python shell our hypervisor presents to figure out what went wrong.\n4. *Log the test details and result to `EXPERIMENT_LOG.md`*\n5. Write a fix to test\n6. grep the experiment log to make sure it hadn’t been tried before\n7. `b step1`\n\nThese instructions lived in my `CLAUDE.md` (later `CODEX.md` when Claude\ncatastrophically failed one too many times).\n\nIt turned out that step 4 was critically important, otherwise an ugly cycle would happen where an agent would try something, it wouldn’t work, compact the context window, try more things, and then lose that it had tried the initial thing and enter a loop of repeating known bad experiments (usually with some minor, superficial variation that the LLM overstated).\n\nIn theory this could also live in the git history, but I found that a simple “just grep the experiment log” worked better. As it stands, the experiment log is 120k lines, but grepping it is a simpler process than wading through commits.\n\nI would check in roughly once a day to see where the LLM was and guide its experiments, otherwise the LLM would spend large amounts of time exploring paths that would very likely not work.\n\nI bootstrapped the hypervisor development with the LLM (Claude to start) and then left the LLM to its own devices. In hindsight, this was a mistake and I should have scaffolded out the problem more than I had. All I did was ensure Claude could interact with the shell (Claude can’t write to a PTY, so I had Claude write a minimal bridge to allow it to talk to the shell), and then patch XNU (the macOS kernel) to allow us to catch its calls to SPTM. From there, Claude was on its own.\n\nThroughout the rest of this post, I will refer to the LLM and I as “we” or “our”, reflecting the nature of the workflow.\n\nFull development of the hypervisor took almost exactly two months, and of those two months about 95% of our time were spent on a relatively small number of issues. Our progress was measured by how far in macOS boot we made it, and every time we solved an issue, it got us further in boot and closer to a working macOS system under our hypervisor:\n\n## `tst` Hang\n\nOne of the first things that happens on any platform is interrupt\nconfiguration, which is critical for the rest of the system to work. Thus, the\nfirst major issue we hit was XNU was hanging on a `tst` instruction as it was\nconfiguring interrupts:\n\n```\n0xfffffe000b57b30c: bl   0xfffffe000b58aee8\n0xfffffe000b57b310: adrp x8, 0xfffffe000c282000\n0xfffffe000b57b314: ldr  w8, [x8, #0x310]       ; kdebug_enable\n0xfffffe000b57b318: tst  w8, #0xfffffff7        ; <-- hang here\n0xfffffe000b57b31c: b.ne 0xfffffe000b57b364\n...\n0xfffffe000b57b338: b    0xfffffe000b582374     ; interrupt-enable tail\n```\n\nNeither Claude nor I had any explanation for this, because if you read the ARM\nArchitecture Reference Manual (lovingly referred to as the ARM ARM), there is\nno scenario where `tst` can hang. Our first thought was that maybe there was\nsome weird interrupt stuff going on. Claude spent a while ping-ponging around\nwith different ideas, mostly involving trying to figure out if we were\naccidentally preventing interrupt delivery by clobbering the interrupt mask. We\nwere clobbering the interrupt mask, but fixing this behavior did not solve the\nproblem.\n\nAt this point Claude gave up (who trains a model to give up?????). Despite days of iteration, Claude had failed to isolate where the problem came from, or even come up with a plausible hypothesis of the root cause. Rather, it just oscillated between various hallucinated ideas, every time claiming it had definitely solved the problem before, in fact, not solving the problem.\n\nI swapped in Codex, because I had already been getting very tired of Claude’s antics. Codex was able to figure out, in order: this was inside of a critical section (ie, XNU was blocking interrupts deliberately); there was an interrupt storm; and when we checked the interrupt state there was always an interrupt pending. The interrupt storm repeatedly triggered our interrupt handler, starving XNU of CPU time, causing the seemingly impossible hang. After identifying the root cause, Codex implemented a fix to our interrupt handling, which moved us past the issue.\n\nI liked Codex’s approach to debugging the problem; each step felt like it logically followed the last and it was able to continually make progress until it identified and fixed the issue. This ran contrary to Claude’s failure to converge and its wild, nonsensical debugging steps. Codex’s monotonically increasing progress is something I came to really appreciate over the course of this project. It also proved to me that Codex had what it took to function in this environment and to really help as a pair programmer.\n\n## Ni6 PIO Gateway Panic\n\nAfter configuring interrupts, XNU tries to initialize a bunch of devices. One of these is the PIO Gateway (we’re not quite sure what PIO stands for, it might mean programmed IO, the DMA alternative). It is an opaque fabric that we cannot introspect; the only information we had on what the failure was came from the panic text which was, brace yourself:\n\n```\nNi6 PIOGW 0 ADDR_DECODE error (0x00000002) cmd:0x14\nvalid/type/tid:0x1/0x1/0x1a\nsrcdieid/srcparentid/srcnodeid/srcdeviceid:0x0/0x19/0xb/0x1\ndstdieid/dstparentid/dstnodeid/dstdeviceid:0x0/0x8/0x3e/0x0\naddrhi/addrl:0x0/0x0\ncmd/size/vc/subch:0x14/0x3f/0x2/0x1\nuser/userext:0x4/0x0\nafixrxliu/afitxliu/afcrxliu/afctxliu:0x0/0x0/0x0/0x0\ncpgerrcode:0x0\n@AppleH17PPlatformErrorHandler.cpp:1688\n```\n\nIf you decode the src and dst, it works out to a special read between two subsystems:\n\n``` php\nSOC_DPE:MSG:1 -> PCPM:APCIE_SYS_GP:0\n```\n\nThe former subsystem is described in an Apple patent\n[US9195291B2](https://patents.google.com/patent/US9195291B2/en), and the latter\ntells us the problem has something to do with PCIe, but we’re really not sure\nwhat the panic means beyond this.\n\nCodex spun on this problem chasing down many leads, none successfully, as the opaque fabric made real debugging nearly impossible.\n\nI had to step in and, using my own intuition, prompt Codex to take another look at a specific part of our SPTM emulator. The idea was that we know everything normally works, so anything that breaks has to be a fidelity bug introduced either by our virtualization or our emulation (this isn’t quite right– macOS wasn’t at the time known to work in the boot mode we use– but it was still the underlying assumption).\n\nIt turns out that a significant amount of our SPTM emulator was incorrect. Fixing our emulator resolved this bug and significantly progressed boot, far enough that XNU tried to mount the root volume from disk.\n\nWe hit a further issue trying to mount the root volume, which was traced down to a very similar root cause as PIOGW (and which we resolved similarly).\n\nThis was a process failure on my part because the errors we hit should either have been avoided or quickly fixed if I had stronger oversight over what my LLM was doing. I always told my LLM to copy what real SPTM does, and Codex always told me it was doing this (which was a lie). What I should have done in hindsight was to figure out, for myself, what SPTM did, and then give Codex a much more detailed guide on what to build rather than just say “copy SPTM” without further details.\n\nAfter successfully mounting root, we successfully completed kernel boot and made it to userspace!\n\n## SEP Timeouts\n\nIn userspace, I measured our progress by the number of live processes. A full macOS system has roughly 170 processes at boot, so the closer we were to that the closer we were to a working system.\n\nWe were sitting at around 10 userspace processes when macOS tried to configure the Secure Enclave Processor (SEP). The SEP is Apple’s root of trust for the platform and provides many runtime features, such as remote attestation. Well, during this configuration, the SEP was timing out as indicated by the error code:\n\n``` php\nAppleSEPGenericTransfer::transact: -> -536870186\nAppleTrustedAccessoryManager::analyticsUpdateDailyDataHandler\n    ... value = 0xffffffffe00002d6\n```\n\nThis was one of the worst problems to debug, because the SEP is a black box: its firmware is encrypted; its crashlogs are encrypted; and many of its capabilities are tightly integrated with hardware. Any failure just causes it to not respond and gives us no debugging information.\n\nAfter some days of debugging, the solution we came to was to disable the thing that was timing out, TrustedAccessory, since nothing depended on it and we don’t care about it (TrustedAccessory is what allows things like external Apple keyboards with fingerprint readers to work, which is clearly unimportant).\n\nI had to manually tell Codex to disable TrustedAccessory, because earlier I had\ninstructed it to never disable functionality under any circumstances (giving\nCodex *any* latitude to turn things off on its own led it to aggressively\ndisable functionality in the name of making progress). This was one of most\ndisappointing things about Codex. If I wanted to give Codex freedom to turn\nsomething off after debugging, I would usually tell it to set a timer, and if\nat the end of the timer only then did it have permission to turn off *that\nspecific* feature.\n\nDisabling TrustedAccessory worked, but there was a second SEP related problem:\nwe couldn’t mount user data because the SEP was refusing to decrypt it.\nObviously, we couldn’t just turn this off. After some frustrating debugging, I\nfound out that this problem was *self-inflicted*. In the previous step, Codex\nhad correctly disabled TrustedAccessory, but also disabled `xars` (the SEP anti\nreplay endpoint), `AppleEpochManager`, and `AppleMobileApNonceNVRAM` (two\nuserspace daemons involved in replay protection) so the SEP would, correctly,\nrefuse to service the decryption request. I had not noticed Codex’s mistake\nbecause it was two lines of code in a sea of changes.\n\nUnlike the previous error, I don’t have a great idea for how to prevent problems like this. I have to give the LLM freedom to try things and can’t look over every line of code since this was a mostly unattended workflow.\n\nAfter we solved both SEP issues, we were able to decrypt and mount the volume:\n\n``` php\ndiscovery-sks=present\nIOSEPBiometricService::sksQueueInit -> err:0x0\nAppleSEPKeyStore: negotiated to ipc header ... negotiated:v2\nAppleSEPKeyStore: Sending notification ... unlocked\napfs_keybag_load_class_keys: successfully loaded volume class keys\ndisk4s1 mount-complete volume custom-macos - Data\n```\n\nThis brought us from roughly 10 userspace processes to 50.\n\n## AppleSunriseWLAN dext Timeouts\n\nThe Neo’s wifi driver is a\n[dext](https://developer.apple.com/documentation/kernel/implementing_drivers_system_extensions_and_kexts),\nor a userspace driver running as part of DriverKit, and is initialized in\nanother userspace boot task. It was timing out:\n\n```\nwlanConfigWifiFunc: FW_START CMD send, waiting for RSP\nwlanConfigWifiFunc: wait event timeout\nwlanProbe: probe failed, reason:3\nAppleSunriseWLAN::setPowerState(..., 0 -> 3) timed out after 22773 ms\nIOPCIDevice::ClientCrashed_Impl() for client AppleSunriseWLAN\n```\n\nFixing this is complicated because it’s a userspace process– generally, any instrumentation would trap to EL1, where XNU was running, rather than EL2 where our code was running. However, Codex was able to debug userspace issues using only raw memory reads and writes. Among other things, I saw Codex manually walk the userspace page tables and make sense of the raw pages and register values. It recognized high level data structures serialized in memory and tracked their life cycles across multiple processes, such as it being able to trace Mach port communication.\n\nUltimately, Codex bisected the problem until it came to what appeared to be a\nclearly incorrect conclusion: that the dext was *not* timing out despite the\nerror message explicitly specifying a timeout. Except Codex was correct. What\nwas happening was that the dext used the proprietary Apple impdef timer. Due to\nthe locked sysreg issue, this timer was misconfigured, and the dext thought\ntime was moving 41.66667x (1 GHz / 24 MHz) faster than it actually was. The\nsolution to this had nothing to do with wifi– we had to set a commpage bit to\ntell userspace to use the ARM generic timer.\n\nThis is very impressive given the difficulty of debugging and the unintuitive nature of the problem. That Codex was able to come to such a strange, but correct, conclusion, was and is very impressive to me. I doubt I would have figured it out even given significantly more time.\n\nThis fix brought us from 50 processes to 150, indicating that almost all of userspace was functional.\n\n## WindowServer exit\n\nThe next major userspace gate was WindowServer, one of the final processes spawned before the whole system works. Right after WindowServer spawned, it would immediately die. After three rounds of this, the system would fail to boot. Unlike the wifi issue, Codex wasn’t making progress, requiring my intervention. Userspace was alive, so I figured I could just boot into macOS regularly, and look at the crashlog. Indeed, this made the problem obvious: the first thing I saw after booting macOS was a crash report:\n\n```\nWindowServer — EXC_CRASH (SIGABRT)\nAbort trap: 6\n\nabort\nsil_mgr_instance(void*, unsigned int)\nCA::WindowServer::IOMFBDisplay::update_power_state_locked(bool)\nWS::Displays::SLCADisplay::set_ca_display_enabled(bool)\nWSInitialize\n```\n\nwhich matched up incredibly well with a log file we had from WindowServer:\n\n```\nWindowServer (SILManager):\nCannot read directory: … Code=260\n“The folder ‘SILManagerAssets.framework’ doesn’t exist.”\nNSFilePath=/System/Cryptexes/ExclaveOS/.../\nPrivateFrameworks/SILManagerAssets.framework/\n… fnfErr: File not found\n\nLoading assets from fallback path:\n…/SILManagerComponent.framework/secureindicatorassets/\n\nCannot read directory: … Code=260\n“The folder ‘secureindicatorassets’ doesn’t exist.”\n… fnfErr: File not found\n\nNo manifest files found in …/secureindicatorassets/\n```\n\nThe red box on screen that tells you when the camera is recording is what was breaking boot because that functionality depends on Exclaves, which aren’t supported. The solution was to disable the Exclave functionality:\n\n```\ndefaults write \"$VOL/Library/Preferences/com.apple.coreanimation\" CAEnableMedina -bool false\n```\n\nI think that if I had given Codex more time, it would have identified this failure eventually, though fixing it would have been extremely difficult if not impossible (requiring a persistent write to disk); human intervention was definitely the right call here.\n\n## UNEXP_RT_HIT_DIR type-24 issue\n\nAt this point I was very happy, because WindowServer spawning means that the verbose boot log finally disappeared and was replaced with the familiar macOS progress bar, except it wasn’t filled at all before macOS would (nondeterministically) panic with this error message:\n\n```\nAMCC0 PLANE0 UNEXP_RT_HIT_DIR error: INTSTS1 0x00000100\nADDR 0x101760f5cf0\nCMD/SIZE 0x5(ncrdincri)/0xf\nAID/TID 0x1a4d8/0x81 DSID:0\nsource=PMGR_SOC_OCLA:PMS_FPWM1:24\n```\n\nThis is a message from the internal SoC fabric that’s telling us that there was a cache directory entry on a page that isn’t supposed to have one. When we looked at the page corresponding to that address, we found a few interesting things: its type was 24 (types are an SPTM concept that essentially let SPTM understand what a page is being used for), the type indicating it stored coprocessor page tables, that the page was, correctly, already marked as noncoherent (NC) in the page tables, and that we did the required TLB/cache flushes for that state change.\n\nIt turns out that any time SPTM changes the type of a page to or from type 24,\nit runs a custom instruction (either `0x00201401` which is a global flush or\n`0x00201328` which takes a page argument) that we cannot execute. We’re not\nsure exactly what this flush does (because we can’t run it) but it might be\nresponsible for fixing whatever caused the panic.\n\nWe came up with a workaround: anytime macOS would retype a page to type 24, we would throw that page away, and substitute our own page, from a memory region that is marked NC forever, sidestepping the problem.\n\nCodex was the one that suggested the fix overnight while unattended, and I woke up to the very pleasant surprise of the progress bar going from not full at all to about 40 percent full.\n\nThe final issues were a collection of issues involving memory maps, cache\ncoherency, and a bug our SPTM emulation; solving these yielded a working macOS\nsystem under our hypervisor! [5](#fn:5)\n\n## Bonus: Exclaves\n\nRemember those Exclaves from earlier that I just turned off? Well, my shiny new\nMacBook Air M5 came and I decided to get Exclaves working. Unlike SPTM,\nExclaves are essentially a fully standard ARM kernel and userspace, so my\nthought here was to virtualize them alongside XNU as a second guest. That means\nall I had to do was implement `ringgate`, essentially a context switch protocol\nfor XNU to send messages to Exclaves and get the results back.\n\nThis was remarkably easy and took under two days to fully support: MTE,\nExclaves, the M5, and a new firmware version (27.0 Beta 3 v2). This SPTM\nimplementation is tainted, and so far has not been replicated, but the\ndocumentation is public<sup>[6](#fn:6)</sup> if anyone wants to take a crack at it.\n\n## Some Lessons I Learned the Hard Way\n\n**Make sure you understand the problem before you put an agent on it**\n\nHad I started this project by understanding what SPTM does and how it works for a day before jumping in, I think I could have saved a month of development time. Given the full development time was only two months, this would have been very significant. I ended up having to pay this cost anyways while writing documentation, so it was a net loss all around.\n\n**Agents need guidance on process**\n\nThere were a few key process improvements I had to make that massively accelerated progress. The most important one (aside from the experiment log) was to tell Codex every few days to simplify its experiments. Otherwise, the hypervisor codebase would balloon to tens of thousands of lines, with hundreds of envvars, most of which were diagnostic but some were load bearing. This would become unmaintainable and slow progress, even for an LLM.\n\nI also noticed that Codex had a very specific bad taste in experiments: if it\nwanted to read a value, instead of just using the hypervisor shell and reading\nthe value, it would usually append the read call to the primary hypervisor\npath, gate it behind an env var, and then reboot and run the hypervisor, which\nadded codebase bloat and 4-5 minutes of time for what could have been a single\n`read32()`. This lived in my `CODEX.md` which also contained fixes for a bunch\nof smaller but still annoying misbehaviors<sup>[7](#fn:7)</sup>.\n\n**Just don’t use Claude Code**\n\nI personally know many people who love Claude Code– I don’t. I think it’s a horrible model that is strictly worse than GPT for everything other than sycophancy, lying, and generating large amounts of slop. I used it occasionally (usually when Codex ran out of usage) and it consistently failed to produce any useful output.\n\n## Deliverables\n\nI manually rewrote the hypervisor changes into a clean patchset that others can\nuse, but it’s useless without an SPTM emulator, which I cannot contribute for\ncopyright reasons. Instead, I wrote documentation on SPTM, which is to my\nknowledge the most detailed public document on SPTM internals. Writing this\ndocumentation took more active effort from me than the entire development\nprocess, because while I had LLMs write the first draft, I had to extensively\nreview everything because LLMs are terrible at writing documentation (it had\nmany inaccuracies and smelled like slop). I’m quite happy with the final\nresult, and Niklas Sheth was able to write a [clean room SPTM\nemulator](https://github.com/ADevWithAnIdea/m1n1/pull/1) based on it.\n\n- Code: [https://github.com/ADevWithAnIdea/m1n1/](https://github.com/ADevWithAnIdea/m1n1)\n- Documentation: [https://github.com/ADevWithAnIdea/apple-silicon-re/tree/master/docs/sptm](https://github.com/ADevWithAnIdea/apple-silicon-re/tree/master/docs/sptm)\n\nIt became obvious during the doc writing phase how full of hacks my tainted emulator was. Suffice to say, I much prefer the clean room version!\n\n## What Comes Next\n\nAs a dev tool, this hypervisor doesn’t mean much to end users– what’s\ninteresting is what you can do with it. Well, we are told that writing a GPU\ndriver is a process that takes years. I’m not sure if I agree. My agents have\nbeen busy reverse engineering the A18 Pro and M5 GPUs. For the A18 Pro, I\nalready have a working prototype of a kernel driver (built using traces from my\nhypervisor)<sup>[8](#fn:8)</sup> and full details of the userspace (via blind probing)<sup>[9](#fn:9)</sup>, all\nbuilt in about two weeks. Let’s see how long it takes to get into your hands.\nIn the meantime, enjoy the beauty of a triangle, rendered using an entirely\nclean-room RE’d userspace and kernel driver.\n\nFunny story about this– I asked Codex for a single triangle for this blog post, thinking it would be a single render. Instead it drew a triangle on a black background using 47 different draw commands. I suppose that’s a good sign for our GPU driver.\n\nAs an aside, Niklas has also been reverse engineering the DCP; he has a working driver that supports all the functionality required on the Mac Mini (but is missing some of the laptop bits). We are looking forward to seeing how we can contribute to the open OS ecosystem on Apple Silicon Macs.\n\n**Edit 9/8:** I took so long to publish this blog post that we’ve made\n*significant* progress on the drivers since then. I have promoted my kernel\ndriver prototype into a fully functional Rust driver on the Neo, and built a\n(very thoroughly tested) prototype kernel driver for the M5. Niklas has fleshed\nout signifiant parts of the GPU userspace stack, finished his kernel driver\nprototype, and built a synchronous Rust driver for Linux on the M4. Here is\nBalatro and Quake running on Linux using his GPU stack with working\ncompositing:\n\nI’ll have more to say on this in the coming weeks.\n\nIf you’re interested in being a part of this, we invite you to join our\n[Discord Server](https://discord.gg/2UeuP9x632). Feel free to come by to\ndiscuss ideas, chat, or just hang out!\n\n## Acknowledgements\n\n- [Niklas Sheth](https://x.com/niklassheth) for writing a clean room implementation of my SPTM emulator and his excellent work on the DCP and GPU\n- [Yureka Lilian](https://fedi.yuka.dev) for significant help with my dev setup\n- The authors of [*Modern iOS Security Features*](https://arxiv.org/pdf/2510.09272) for providing valuable insight into how SPTM works\n- The [Asahi Linux](https://asahilinux.org) project for laying the groundwork that made this work possible\n\n## Footnotes\n\nThis prototype driver contains basically all firmware ABI details we ever need and is very comprehensive, containing full compute, render (partial renders, depth/stencil buffers, etc), fence, and multi context support.\n\nThis issue is still present on the MacBook Neo; on the M4 mac mini, Niklas fixed the issue during the development of the clean room emulator. His best guess of the root cause is that m1n1 has a WB mapping of all guest RAM, and then an access to one of the problematic pages may be speculatively executed, but we’re ultimately unsure of the root cause or why the Neo hits the problem while the M4 Mac Mini does not.\n\n1. [Hardware Reverse Engineering with the m1n1 Hypervisor](https://asahilinux.org/2021/08/progress-report-august-2021/#hardware-reverse-engineering-with-the-m1n1-hypervisor)[↩︎](#fnref:1)\n2. My best guess is that the kernelcache has fallbacks for booting without Exclaves for DFU restore purposes. DFU restore requires booting a full kernelcache and sending all of the components over USB, but in DFU boot Exclaves are not booted (probably because they’re not necessary). Userspace has support for no Exclaves because userspace is the same in macOS images built to run under a hypervisor and in bare metal images, so it has to have fallbacks for when Exclaves are not available. [↩︎](#fnref:2)\n3. The Modern iOS Security paper does not contain enough detail to actually implement the SPTM contract, though it was very helpful in other ways. [↩︎](#fnref:3)\n4. [AsahiLinux/macvdmtool](https://github.com/AsahiLinux/macvdmtool)[↩︎](#fnref:4)\n5. There was one issue I didn’t solve, for some reason shortly after logging in (or just sitting for long enough on the lock screen) the system would freeze with the same `UNEXP_RT_HIT_DIR` error as in type 24, except there were no\ncoprocessor page tables this time.[↩︎](#fnref:5)\n6. [https://github.com/ADevWithAnIdea/apple-silicon-re/blob/master/docs/sptm/exclaves.md](https://github.com/ADevWithAnIdea/apple-silicon-re/blob/master/docs/sptm/exclaves.md)[↩︎](#fnref:6)\n7. A brief list: setting long timeouts then continuing to wait after they were hit (saying the guest was just slow), focusing on timing/sequencing issues when there were much more obvious logic bugs, hallucinating the function of sysregs or other Apple specific concepts. [↩︎](#fnref:7)\n8. [https://github.com/ADevWithAnIdea/m1n1/commit/ffe58c7802376b111f67b3cf45d80a1945db1f90](https://github.com/ADevWithAnIdea/m1n1/commit/ffe58c7802376b111f67b3cf45d80a1945db1f90)[↩︎](#fnref:8)\n9. [https://github.com/adevwithanidea/agx-re/](https://github.com/adevwithanidea/agx-re/)[↩︎](#fnref:9)", "url": "https://wpnews.pro/news/i-came-i-prompted-i-left-part-1-building-a-hypervisor-for-the-macbook-neo", "canonical_source": "https://codyho.dev/blog/hypervisor-macbook-neo/", "published_at": "2026-09-09 18:49:09+00:00", "updated_at": "2026-09-09 19:10:46.152089+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-tools", "ai-research"], "entities": ["Apple", "MacBook Neo", "A18 Pro", "Asahi Linux", "m1n1", "SPTM"], "alternates": {"html": "https://wpnews.pro/news/i-came-i-prompted-i-left-part-1-building-a-hypervisor-for-the-macbook-neo", "markdown": "https://wpnews.pro/news/i-came-i-prompted-i-left-part-1-building-a-hypervisor-for-the-macbook-neo.md", "text": "https://wpnews.pro/news/i-came-i-prompted-i-left-part-1-building-a-hypervisor-for-the-macbook-neo.txt", "jsonld": "https://wpnews.pro/news/i-came-i-prompted-i-left-part-1-building-a-hypervisor-for-the-macbook-neo.jsonld"}}