Running Local AI on Nvidia DGX Spark Without CUDA A research-stage toolchain called CUDA++ now compiles GPU programs to native machine code for Nvidia's GB10 Grace Blackwell chip (compute capability sm_121, PCI id 10de:2e12) inside the DGX Spark, running them without a CUDA toolkit, nvcc, PTX, libcuda or cuBLAS at build or run time, according to the project's aien-dev/CUDApp repository. On 2026-10-21 the Qwen3.6-35B-A3B mixture-of-experts model (about 3B active parameters per token) was served on the Spark by the C GPU stack the repository was extracted from, with only the C library, maths library, libgcc_s and the loader mapped and zero CUDA libraries, per a receipt in the Omega repository. The toolchain supports only the GB10 today, requires Nvidia's open kernel module version 580.173.02 and its signed GSP firmware, does not compile CUDA C++ or PTX, and is licensed Apache-2.0 WITH LLVM-exception for first-party material. CUDA++ for the NVIDIA DGX Spark GB10, sm 121 . It is the GPU half of the Omega runtime. First-party CUDA++ material in this revision is licensed under Apache-2.0 WITH LLVM-exception. You write a GPU program, CUDA++ turns it into the GB10's own machine code and runs it on the Spark's GPU with no CUDA user-space software at build or run time: no CUDA toolkit, no nvcc , no PTX, no libcuda , no cuBLAS. NVIDIA's compiler and disassembler were used offline as references while the encoding was worked out how I did it how-i-did-it . NVIDIA's open kernel module 580.173.02 and the GPU's signed GSP firmware are still in the path. It's a research-stage toolchain and runtime, not a drop-in replacement for all of CUDA. Who this is for, and what hardware Supported The NVIDIA DGX Spark. Its GB10 Grace Blackwell chip GPU compute capability sm 121, PCI id 10de:2e12 is the only GPU CUDA++ has machine code for. Other GB10 systems Not run. CUDA++ recognises the GPU by its PCI id, not by the system maker, so another GB10 system with the same kernel module is the same target to the code. Nobody has run CUDA++ on one. You need Linux on the Spark's ARM CPU aarch64 . NVIDIA's open kernel module, version 580.173.02: make doctor and make hello-gpu refuse another version, and /dev/nvidiactl , /dev/nvidia0 and /dev/nvidia-uvm must exist. NVIDIA's signed GSP firmware, which that module loads into the GPU. git , make and a C compiler. No CUDA toolkit. Not supported Other NVIDIA GPUs: PLANNED, not supported. A target row for the RTX 5070 sm 120 exists on Omega's branch rig/gb20x aien-dev/omega-mv, src/omega bw target.h , built from public sources and never run on that chip portability https://github.com/aien-dev/CUDApp/blob/main/docs/runtime/PORTABILITY.md . Every other GPU, NVIDIA or not AMD, Intel, Apple, Qualcomm, Arm, Imagination and the rest : PLANNED targets, not supported. CUDA++ has machine code, a driver client and receipts for exactly one GPU today, the GB10; the multi-backend split is docs/ARCHITECTURE.md https://github.com/aien-dev/CUDApp/blob/main/docs/ARCHITECTURE.md PROPOSED , and no capability claim about any other GPU is public until a kernel runs on it with a receipt. Unmodified CUDA applications: not supported; a CUDA++ program calls CUDA++'s C kernel builder, and CUDA++ does not compile CUDA C++ or PTX. Licensing status: This revision uses Apache-2.0 WITH LLVM-exception for first-party material. For the historical MIT version, use 232332422bfdc31c8918085b50f13c1e18ab702e https://github.com/aien-dev/CUDApp/tree/232332422bfdc31c8918085b50f13c1e18ab702e . The old MIT grant and third-party notices are preserved. See the transition and provenance audit https://github.com/aien-dev/CUDApp/blob/main/docs/LICENSING TRANSITION.md . It runs on the NVIDIA GB10 the DGX Spark, compute capability sm 121 today. On 2026-10-21 the model Aien serves on the Spark, Qwen3.6-35B-A3B a mixture-of-experts model with about 3B parameters active per token , was served by the C GPU stack this repository was extracted from the live build used the Omega repository's copy of these sources . The live server had only the C library, the maths library, libgcc s and the loader mapped, and zero CUDA libraries receipt in Omega: evidence/WHAT-OMEGA-IS-20261021/live-serve-no-cuda.txt . That stack is mostly C. Two pieces of it now run as code written in the Omega language: the sm 121 instruction encoder om/encoder/bw enc sm121.om and the cubin packer om/cubin/cubin elf.om , compiled by the pinned Omega compiler and linked through an omega-bridge binding. Their C bodies are out of the default build and kept only as differential oracles; the .om encoder is byte-identical to the C one on every kernel the test suite builds evidence https://github.com/aien-dev/CUDApp/blob/main/evidence/CUDAPP-OM-ENCODER/README.md , migration table in CUDApp 79 . The rest kernel builders, register allocation, runtime is still C, and Omega programs compiled to the GPU by the Omega compiler itself is Omega bar C1, still BUILDING. CUDA++ keeps growing to cover the jobs of NVIDIA's user-space software on the Spark, and then do things CUDA does not: - NVIDIA's open kernel modules nvidia.ko and nvidia-uvm.ko stay. GB10 support belongs in the upstream Linux drivers Nova and nouveau, not in a third driver; Aien's own driver branch L332, aienux 441 is a research harness, not a product; - my own disassembler replaces nvdisasm ; - the GSP firmware stays NVIDIA's too, for a physical reason: the GPU checks its signature and boots nothing else; - beyond CUDA: owning the whole path from program to chip, so one submission per generated word, whole-model kernels that stay resident on the GPU, and zero-copy use of the memory the CPU and GPU share. CUDA++ is not affiliated with or endorsed by NVIDIA; CUDA is NVIDIA's trademark. New to CUDA++? Start with the first-run and independent verification guide https://github.com/aien-dev/CUDApp/blob/main/docs/START HERE.md . It separates host-only tests from real GB10 execution, documents the current support boundary, and explains how to interpret receipts. See contributing https://github.com/aien-dev/CUDApp/blob/main/CONTRIBUTING.md , security reporting https://github.com/aien-dev/CUDApp/blob/main/SECURITY.md , and the readiness roadmap https://github.com/aien-dev/CUDApp/blob/main/ROADMAP.md before relying on it in a deployment. On this page: the stack the-stack · how I did it how-i-did-it · the machine code what-the-machine-code-looks-like · the driver, part by part the-driver-part-by-part · one launch, start to finish one-launch-start-to-finish · CUDA, piece by piece cuda-piece-by-piece · speed speed-against-the-ceiling · run your first kernel run-your-first-kernel Between a CUDA program and the GPU sit four closed NVIDIA layers: the compiler, the maths libraries, the runtime and the driver library. CUDA++ does their jobs with its own code, at research stage. Under them are NVIDIA's kernel module, whose source NVIDIA publishes, and the firmware inside the GPU. CUDA++ talks to the kernel module directly, and it stays; GB10 support goes upstream to Nova and nouveau. I built it alone on one DGX Spark in under four weeks, directing AI coding agents. NVIDIA publishes the source of its Linux kernel module, but not its compiler, its driver library or the GPU's machine code. So I skipped every closed layer: 1. No libcuda. I wrote my own client for the open kernel module's ioctls, from its MIT headers: client, device, address space, memory, command ring, compute object, doorbell, completion. The driver, part by part the-driver-part-by-part shows every piece. 2. No nvcc, no PTX. The GPU's machine code is undocumented, so I reverse engineered it: compiled small probe kernels of my own once with NVIDIA's compiler, read the machine code back with NVIDIA's disassembler, compared kernels that differ in one detail to find which bits mean what, cross-checked against Mesa's open NVIDIA compiler, and believed nothing until the chip ran it and gave the right answer. Arithmetic the documents do not pin down the tensor cores was measured on the chip until a model matched every output bit. 3. My own encoder, runtime and libraries on top: 59 instruction kinds, a register allocator, launch descriptors, command lists, matrix multiply, attention. I'm not the only one mapping this instruction set in the open, and I credit the others: Mesa's NAK compiler has an SM120 encoder tested against nvdisasm ; cubit https://github.com/kacper-daftcode/cubit is an MIT assembler and disassembler for SM120 and SM121a; blackwell-isa https://github.com/kacper-daftcode/blackwell-isa publishes an sm121a encoding table; nvidia-sass-document https://github.com/cicuvc/nvidia-sass-document documents SASS encodings; tinygrad's NV backend talks to the same open kernel module interface. Cross-checking CUDA++ against them is PROPOSED. No NVIDIA binary is in this repository. One listing printed by NVIDIA's nvdisasm is stored as a test pin: the decode of CUDA++'s own trace-emitter words PROVENANCE.md section 6 . The whole method, step by step with the files and receipts: docs/HOW I DID IT.md https://github.com/aien-dev/CUDApp/blob/main/docs/HOW I DID IT.md . Every GB10 instruction is 16 bytes. The low bits say what to do and to which registers; the top 23 bits tell the warp scheduler how long to wait before the next instruction and which earlier results to wait for. Here is the add from the first kernel, every bit accounted for: Those scheduling bits are what a compiler normally hides. Slow instructions memory loads, constant loads, special registers set one of six scoreboards; a later instruction that needs the result waits on it. Here is the whole first kernel with its waits drawn in: Getting one of these bits wrong does not fail loudly: the GPU reads a register before the load has landed and returns a wrong number. That is why every encoding has golden words in the tests and a chip run behind it. The figures on this page are drawn from the repository's own data by tools/figures/make figures.py ; the two above decode the words in src/omega blackwell encoder.c . libcuda is closed, but everything it does to the hardware goes through ioctls on three device files, and NVIDIA's open kernel module defines every one of those calls in its public headers. I copied 67 of those headers unchanged third party/nvidia-open-580.173.02/ , MIT and wrote my own client against them. This is the whole of it: | Part | Where | What it does | |---|---|---| | Open | nvrm open in nvrm/nvrm.c | Opens /dev/nvidiactl and /dev/nvidia-uvm , finds the GPU NV ESC CARD INFO , allocates the root client, device and subdevice, reads the class list to pick the Blackwell classes, reads the SM version and GPU UUID, maps the 64 KiB usermode page that holds the doorbell, and creates an address space that the unified-memory driver manages. | | Memory | nvrm alloc , nvrm free | One system-memory object per buffer. The GB10 has no separate video memory, so each buffer is mapped at the same address for the CPU and the GPU: a pointer means the same thing on both sides and nothing is copied. Anything the CPU polls is allocated GPU-uncached nvrm alloc gpu uncached . | | Channel | nvrm channel | Channel group, context share, a command ring of 1,024 entries with its USERD control page, and the Blackwell compute object. Asks for the channel's work-submit token, registers the channel with the unified-memory driver and switches scheduling on. | | Submit | nvrm enqueue , nvrm ring | Writes one 8-byte ring entry pushbuffer address and length , advances GPPut with a full memory barrier, then rings the doorbell: a single 32-bit store of the token. No system call is made to launch work. | | Wait | m16/m16 gpu wait.c | The one completion wait: polls a word the GPU writes, with a hard deadline that is never extended and a stall deadline that restarts only on progress. | | Qualification | m16/m16 requalify.c , m16/m16 concurrent.c | Proofs on the chip. Requalify queues work without ringing the doorbell and checks nothing happens for 100 ms, rings it, checks the marker turns to 0x16c0ffee , and reads /proc/self/maps to prove libcuda was never loaded. Concurrent runs two channels and checks neither can fire the other. | | Platform contract | src/omega gpu platform.h | What the runtime may ask of the machine: typed handles instead of file descriptors, explicit rights on every buffer, no silent fallback. The Linux client above is one implementation; AIEN's own driver plugs in at the same seam. | | Session, command lists, engine | src/omega gpu session.c , src/omega gpu cmdlist.c , src/omega gpu engine.c | One device per process, kept open; many launches recorded into one submission; a job runner that returns success or exactly one named failure, and refuses all further GPU work if it cannot tell whether the GPU finished. | What the headers do not say, I learned on the chip and wrote down as rules, each with a receipt or an issue behind it: - A word the CPU polls must be GPU-uncached. Otherwise the GPU keeps the new value in its L2 cache and the CPU never sees it change. This stalled the work once, on 2026-10-02; the platform now refuses such a buffer. - New code needs a shader-cache invalidate before it runs. The GPU's instruction cache does not see CPU writes; after a code address was reused, the chip ran the old instructions. Every launch that uploads code now starts with INVALIDATE SHADER CACHES receipt FB1-CUT1B-4345406 in Omega . - Dependent launches need a wait-for-idle between them. Without one, two grids on one channel may overlap. The order of the launch commands follows Mesa's NVK driver. - Blackwell's control page reports no read pointer. The client counts completed ring entries itself and never overruns entries that have not been seen to finish. - Creating the compute object needs one free, physically contiguous 2 MiB block for its context buffer. On a busy host it is briefly missing, so that one allocation is retried, bounded omega 523 . Under the client: NVIDIA's open kernel module nvidia.ko , nvidia-uvm.ko and the GSP firmware inside the GPU. Both stay: GB10 support goes upstream to Nova and nouveau, Aien's own driver branch L332 is a research harness only, and the firmware is NVIDIA's because the chip checks its signature. Every object and call in order, with line references: docs/DRIVER.md https://github.com/aien-dev/CUDApp/blob/main/docs/DRIVER.md . The CPU does five things, all of them plain memory writes except the last, which is a single store to a GPU register. The GPU then does the rest on its own: it fetches the ring entry, reads the commands, reads the launch descriptor the QMD: grid size, block size, registers, shared memory, code address , runs the grid, and writes two completion markers with a cache flush between them. Results are already in memory the CPU can read. Recording many launches into one submission src/omega gpu cmdlist.c is how a whole generated word goes to the GPU at once. | CUDA piece | Its job | CUDA++ replacement | Status | |---|---|---|---| | CUDA C++ language | How you write a kernel | The kernel builder C calls . Omega's languages reaching the GPU is Omega bar C1, built on this toolchain | PARTIAL: C calls only | | nvcc, NVVM, PTX, ptxas | Compile the kernel to machine code | My own sm 121 encoder and register allocator | PASS: every GPU kernel this stack runs is encoded by it | | libcuda driver API | Talk to the kernel driver | nvrm client, ioctls straight to the open kernel module | PARTIAL: implemented; M16 historical summary is unauditable here see below | | libcudart runtime API | Device, memory, launch, streams | Session, command lists, job engine | PARTIAL: one device, one process; no streams or events yet; a timeout ends GPU use for the process | | cuBLAS | Matrix multiply | Matmul, tensor-core GEMM, FP8 and BF16 libraries | PARTIAL: shaped for the models AIEN runs; no any-shape GEMM yet | | cuDNN, FlashAttention | Attention and network layers | Decode and prefill attention | PARTIAL: the fused fast paths are model-shaped; since runtime API 0.6.0 a general fallback omega gpu attention general , EXPERIMENTAL takes head dim 1..256, any MHA/GQA/MQA ratio, causal or not, sliding windows and any length, chip-exercised on GB10 evidence/ATTN-GENERAL-20261027-4d51f91f/ . It is a scalar f32 kernel without tensor cores, and has no ALiBi, softcap or dropout | | CUB, Thrust | Reductions, scans, sorts | Frozen-order reductions evidence/E1-CLOSURE/ | PARTIAL: no general scan or sort | | cuFFT, cuRAND | FFT, random numbers | none yet | MISSING | | cuobjdump, nvdisasm | Read machine code back | My own sm 121 disassembler | BUILDING: on Omega branch c6/disasm 19eebe0e it agrees with nvdisasm on 331 kernels; not in CUDA++ yet. Some host tests here still call nvdisasm as a cross-check when it is installed | | Nsight | Measure | Ceiling probes: 115.14 TFLOPS tensor-core peak, 195.1 GB/s reads evidence/OWN30B-T3-TC-PEAK-BW-5faa0f70-20261013T2236Z/ | PARTIAL: probes, no single tool | | CUDA samples, docs | Your first kernel | examples/hello gpu , docs/runtime/ | PARTIAL: one example | | nvidia.ko , nvidia-uvm.ko | Kernel driver | stays NVIDIA's open kernel module; GB10 support goes upstream to Nova and nouveau L332 is a research harness | stays | | GSP firmware | Runs inside the GPU | stays NVIDIA's signature-checked by the chip | not replaceable | PASS means a receipt exists on a named commit. Status words come from Omega's docs/OMEGA.md , section 2. ROADMAP.md lists every NVIDIA piece on the Spark with its receipt, and the bars CUDA++ must pass before it says "replaces CUDA". The imported evidence/m16-blackwell-native-path-requalification-receipt.json is an unauditable historical summary , not qualification of this checkout. It names implementation commit a2c0d7fbc03f7ce91fcb6389312a0ee8630322cd , but its evidence/m16-requalification raw logs and superseded receipt are absent from this repository. Its historical PASS and durability fields are preserved unchanged, not endorsed as current evidence. No immutable durable source for those missing logs has been established. Current native qualification is NOT RUN ; it needs retained raw logs from a new authorized GPU run, not a host-only check. Every speed number is held against what this machine can physically do, measured by my own probe kernels. Generating a word of text reads each active weight once and does about one operation per byte, far left of the ridge, so its speed is set by memory, not by the tensor cores. The head-to-head measurement against llama.cpp and vLLM on this Spark is Omega bar P0 AIEN L394 and has not been published. Until it is, this repository claims no speed comparison with CUDA, llama.cpp, vLLM or TensorRT. Its predeclared protocol is in aien-dev/gpu-head-to-head https://github.com/aien-dev/gpu-head-to-head , which pins this repository by commit; the raw logs and results will be published there. Chip-only times GPU time per token and end-to-end times through the HTTP server, prompt included are different measurements and are always labelled separately; figures other projects publish are theirs, not re-measured here. | Part | What it does | Where | |---|---|---| | GPU toolchain | Writes sm 121 machine code SASS directly: a kernel builder with 59 instruction kinds, a register allocator, the instruction encoder, launch descriptors QMD | src/omega blackwell codegen .c , src/omega blackwell encoder.c , src/omega blackwell qmd.c | | GPU runtime | Opens the device, allocates memory, uploads code, launches kernels, records command lists, runs jobs. The job of CUDA's runtime and driver API | src/omega gpu session. , omega gpu cmdlist. , omega gpu engine. , omega gpu platform | | Libraries | Matrix multiply and matrix-vector BF16, FP8, 4-bit, ternary , attention for decode and prefill, elementwise, mixture-of-experts routing. The job of cuBLAS and cuDNN | src/omega gpu api. | | Driver client | Talks to NVIDIA's open kernel module through its ioctls; replaces libcuda | nvrm/ , m16/ | | NVIDIA interface headers | The open kernel module's public interface definitions MIT, unchanged | third party/nvidia-open-580.173.02/ | | Runtime API | The written, versioned API and its manifest check | docs/runtime/API.md , src/omega runtime api.h | | Figures | Every picture on this page, drawn from the repository's own data | tools/figures/make figures.py , docs/img/ | I read what developers write about CUDA and its libraries on GitHub, Reddit, NVIDIA's forums, the PyTorch forum, Hacker News and Stack Overflow, and ranked the problems by how many people they hurt GitHub issue counts times how badly they hurt . The full research, with 97 sources and every quote checked word for word against its page, is docs/PAIN POINTS.md . How CUDA++ itself was built, reverse engineering included, is docs/HOW I DID IT.md . Each row below gives the pain in a developer's own words, how CUDA++ fixes it, and the command that shows it, with the output it printed on my DGX Spark and the receipt it came from. A row says FIXED only when that command and receipt are on this branch. Where CUDA is still better, docs/PAIN POINTS.md section 4 says so. 1. Out of memory, and on the DGX Spark a frozen machine 26,390 GitHub issues say "CUDA out of memory" The pain. "The Spark ships with zero swap. On unified memory a spike goes straight to a global OOM that hard hangs the whole box. No ssh, no ping, needs a power cycle." dgx-spark-laguna FINDINGS.md https://github.com/sudoingX/dgx-spark-laguna/blob/main/FINDINGS.md How CUDA++ fixes it. On the GB10 the CPU and GPU share one memory, and every CUDA++ buffer is part of it. CUDA++ has no caching allocator, so what a process holds is what it asked for. Before any allocation reaches the driver, CUDA++ checks how much memory the machine has free. If the buffer would leave less than a reserve 4 GiB by default, CUDAPP MEM RESERVE MIB to change it, 0 to turn it off , the allocation is refused with a sentence instead of freezing the box. The command that shows it host test of the runtime on a simulated driver, run on the Spark : bash $ make build/test omega gpu platform linux && ./build/test omega gpu platform linux ... refusal: dma alloc: refused 2.0 MiB: 4.0 GiB of memory is free and CUDA++ keeps 4.0 GiB free CUDAPP MEM RESERVE MIB=4096 , because the CPU and GPU share it and running it out freezes the machine. Free buffers or use a smaller batch. session: dma alloc: refused 2.0 MiB: 4.0 GiB of memory is free and CUDA++ keeps 4.0 GiB free CUDAPP MEM RESERVE MIB=4096 , because the CPU and GPU share it and running it out freezes the machine. Free buffers or use a smaller batch. PASS D1 MEMORY RESERVE REFUSES BEFORE THE DRIVER test omega gpu platform linux reference only, simulated driver : 9 passed, 0 failed Receipts: evidence/dx/d1-memory/platform-test-spark.txt , and evidence/dx/d1-memory/negative-control.txt with the check removed, the test fails . FIXED in the runtime, shown on the host. Left: the same refusal on the chip itself, inside a GPU window docs/issues/D1-memory.md . The pain. "I don't know a right solution that works for all setups, happy to hear suggestions." FlashAttention's maintainer, flash-attention 246 https://github.com/Dao-AILab/flash-attention/issues/246 How CUDA++ fixes it. There is no toolkit version to match: compiler, runtime and libraries are one repository at one commit, built with make and a C compiler. The one outside version is NVIDIA's kernel module, whose public headers CUDA++ is built against. make doctor checks it and everything else a first run needs. It reads /proc , /sys and /dev only and opens no device. Every problem gets a line saying what is wrong, why and the command that fixes it. The doctor wrapper refuses a module of another version; make hello-gpu runs that wrapper first. Direct execution of binaries and applications linking the library have no driver-version or PCI-ID admission guard . The diagnostic is not a sandbox or current-commit qualification; callers must perform the compatibility preflight themselves and have an authorized GPU window. The command that shows it: The following excerpts are historical doctor output; current output explicitly labels the M16 receipt as a historical summary with raw evidence unavailable. bash $ make doctor cudapp doctor: is this machine ready for CUDA++? info system: Linux 7.0.0-1019-nvidia aarch64 ok NVIDIA kernel module: open module 580.173.02 CUDA++ speaks 580.173.02 ok device files: /dev/nvidiactl /dev/nvidia0 /dev/nvidia-uvm can be opened by this user ok GPU 000f:01:00.0: NVIDIA GB10 sm 121 , qualified receipt: evidence/m16-blackwell-native-path-requalification-receipt.json ok memory: 121.7 GiB shared by the CPU and the GPU, 61.3 GiB available now info CUDA toolkit: not needed and not used by CUDA++ one is installed at /usr/local/cuda/bin/nvcc; that is fine info compiler: this doctor was built by 13.3.0; CUDA++ needs only make and a C compiler READY. Next: make hello-gpu On a machine with the wrong module one of the nine fake machines CUDAPP DOCTOR SHOW=1 sh tests/doctor/run.sh build/cudapp-doctor builds : FAIL NVIDIA kernel module: version 575.64.05 is loaded; CUDA++ speaks the 580.173.02 interface Why: CUDA++ builds its driver calls from the 580.173.02 open-module headers third party/nvidia-open-580.173.02 . The layout of those calls can change between module versions, and a wrong layout fails in strange ways, so CUDA++ refuses instead of guessing. This is the only version CUDA++ asks you to match: no toolkit, no framework build, no extension build. Do this: install the 580.173.02 open kernel module, or port the client to your version CONTRIBUTING.md, "Port a driver version": copy the new headers, rebuild, run make test and make hello-gpu . Receipts: evidence/dx/d2-doctor/make-doctor-spark.txt , evidence/dx/d2-doctor/fake-machines.txt , evidence/dx/d2-doctor/negative-control.txt . FIXED. The pain. "In our experience the python stack traces for these types of exceptions are basically always incorrect and pretty worthless." vLLM's maintainers, vLLM blog https://vllm.ai/blog/2025-08-11-cuda-debugging How CUDA++ fixes it. A CUDA++ launch waits for its own completion by default, so an error comes back from the call that caused it. And the same kernel program runs on CUDA++'s host warp simulator with one flag, where every global access is checked against the buffers the launch was given. A bad access stops the run at that launch and names the kernel, the launch, the instruction, the block and thread, the address, the buffer it missed and by how much, and what to do next. The command that shows it --bug=past-end hands the last launch one element too many, the classic off-by-one : bash $ build/hello gpu --sim --bug=past-end FAULT at launch 8 of 8 of kernel vecadd simulator : global load outside every buffer launch: grid 1025 x 1 blocks of 128 threads, n = 131073, first element 917504 instruction: 11 of 19, LDG.E 4-byte load thread: block 1024, thread 0 element 1048576 of the whole job address: 0xeba6fbc6f040 4 bytes , 0 bytes past the end of buffer a = 0xeba6fb86f040, 0xeba6fbc6f040 4194304 bytes simulator: MEM at pc 11: global access outside the declared buffers next: launch 8 was given n = 131073, and buffer a holds 131072 elements from this launch's first element. Fix the launch or the buffer, run hello gpu --sim until it passes, then run hello gpu on the GPU. Receipts: evidence/dx/d3-faults/fault-report-spark.txt , evidence/dx/d3-faults/test-hello-sim.txt , evidence/dx/d3-faults/negative-control.txt with bounds checking off, the fault checks fail . FIXED on the simulator. Left: on the chip a faulting kernel still shows up as a launch that never completes, and the process then refuses further GPU work. Reading the GPU's own fault record docs/issues/D3-chip-faults.md and recovering without a restart Omega bar C5, ledger L386 are open. The pain. "For reference I just compiled it on a 2018 Dell laptop Quadro P1000 GPU and the build time was about 15 hours." flash-attention 1038 https://github.com/Dao-AILab/flash-attention/issues/1038 How CUDA++ fixes it. No per-architecture copies and no template-heavy device compiler: CUDA++ generates each kernel's machine code when the program builds it row 9 , so the library is plain C. The command that shows it on the Spark, 20 cores, with other builds running: load average 31.31 : bash $ git clone -b dx/pain-points