Why Local LLMs Don't Need C++ or Python: Building a 15MB Native AOT Inference Engine in .NET 10

A developer built Glacier.Inference, a ~15 MB single-file Native AOT inference engine written in pure C# on .NET 10 that runs GGUF models directly across NVIDIA, AMD, and Intel GPUs without C++ binaries or the CUDA runtime. The engine bypasses cudart64.dll and cublas64.dll by dispatching precompiled cubins straight to NVIDIA SMs via P/Invoke to nvcuda.dll, and fuses GPU-side argmax reduction so only a 4-byte token ID crosses the bus per step instead of 608 KB of logits. It also includes a Direct3D 12 compute path to avoid Windows TDR driver resets on laptops where integrated graphics drive the display.

The conventional consensus across AI engineering is simple: high-performance local LLM execution belongs exclusively to C++ runtimes, multi-gigabyte CUDA toolkits, and bindings over llama.cpp or vLLM . When orchestrating local models from managed languages like C , typical implementations rely on interop wrappers over unmanaged native binaries cudart64.dll , cublas64.dll , or libllama . This introduces DLL distribution overhead, host-to-device PCIe bandwidth bottlenecks during token sampling, and severe desktop instability when running compute on display-bound integrated GPUs. By stripping out runtime layers and interacting directly with driver interfaces, managed runtimes can match and outpace conventional native daemons. Glacier.Inference runs direct memory-mapped GGUF models in pure C .NET 10 across NVIDIA, AMD, and Intel silicon without external C++ binaries. ┌────────────────────────────────────────────────────────────────────────┐ │ Glacier.Inference Core │ │ ├─ MemoryMappedFile Zero-Copy GGUF Reader Sub-30ms cold mapping │ │ ├─ Pure C Bare-Metal SASS Engine Direct P/Invoke nvcuda.dll │ │ ├─ Bare-Metal Direct3D 12 Compute HLSL Wave32 via Vortice.D3D12 │ │ ├─ SpeculativeEngine N-gram Prompt Lookup & Batched Verification │ │ ├─ Fused In-VRAM GPU Argmax Reduction Warp-shuffle, 4-byte transfer │ │ └─ Adaptive Unmanaged KV-Cache FP16 / FP8 Dynamic Ring Buffer │ └────────────────────────────────────────────────────────────────────────┘ Typical CUDA execution routes instructions through cudart64.dll and cublas64.dll . Glacier completely circumvents the CUDA runtime layer. It communicates directly with the base Windows GPU driver nvcuda.dll via low-level P/Invoke, dispatching precompiled fatbinaries straight to the Streaming Multiprocessors SMs . // Direct driver context and module dispatch without cudart64.dll DllImport "nvcuda.dll", EntryPoint = "cuLaunchKernel" public static unsafe extern CUresult LaunchKernel CUfunction f, uint gridDimX, uint gridDimY, uint gridDimZ, uint blockDimX, uint blockDimY, uint blockDimZ, uint sharedMemBytes, CUstream hStream, void kernelParams, void extra ; By compiling modular .cuh routines into an embedded universal cubin containing dedicated slices sm 75 , sm 80 , sm 86 , sm 89 , sm 90 , the engine verifies zero DRAM stack spills STACK: 0 via cuobjdump . Registers house all dequantization multipliers and accumulators without hitting DRAM stack frames. The result is a self-contained ~15 MB single-file Native AOT executable replacing a 4.5 GB toolchain. In standard architectures, evaluating greedy token selection involves transferring logit tensors back over PCIe to the host CPU: $$\text{Logit Transfer Per Step} = 152\text{K vocab} \times 4\text{ bytes} \approx 608\text{ KB}$$ At 45 tokens per second, transferring 608 KB back and forth over the host interface introduces micro-stalls and PCIe latency. Glacier fuses the final linear projection and reduction directly on the device with a 512-thread warp-shuffle kernel: // GPU-side warp reduction eliminating CPU transfer overhead numthreads 512, 1, 1 void ArgmaxReduction uint3 tid : SV DispatchThreadID, uint3 lid : SV GroupThreadID { // 512-thread tree reduction within registers across 152k logits // Emits exactly 1 int32 winning token index into device memory } Instead of copying 608 KB across the bus every token, Glacier transfers exactly 4 bytes one int32 token ID , dropping reduction latency to ~3.2 µs. Deploying local inference on consumer hardware often fails on laptops where AMD RDNA or Intel Arc graphics double as the primary display adapter. Long compute dispatches trigger Windows Timeout Detection and Recovery TDR , resetting the display driver. Glacier implements a pure Direct3D 12 compute engine using Vortice.D3D12 and HLSL Wave32 compute shaders: Autoregressive transformer generation is inherently memory-bandwidth bound. Every generated token requires streaming the entire model weight footprint through the compute core. On a 128-bit GDDR6 memory bus running at 256 GB/s, reading a 4.68 GB model sets a strict theoretical wall clock limit: $$\text{Max Serial Rate} = \frac{256\text{ GB/s}}{4.68\text{ GB}} \approx 54.7\text{ tokens/sec}$$ Glacier integrates batched speculative verification VerifyBatch using zero-cost prompt suffix lookup PromptLookupDraftProvider : js using var target = new InferenceSession "models/Qwen2.5-7B-Instruct-Q4 K M.gguf" ; using var engine = new SpeculativeEngine target ; var options = new SpeculativeOptions { MaxDraftTokens = 4, // Propose 4 candidate tokens in <1 µs MaxTokens = 256 }; var result = await engine.GenerateAsync "Explain quicksort in C ", options ; Instead of reading 4.68 GB from VRAM $K$ times for $K$ tokens, the candidate sequence is verified against the transformer in a single batch pass. The weights are pulled through the memory bus only once , accelerating generation rates to 72–104+ tokens/sec on an RTX 4060 laptop GPU. The following runs compare Glacier against an Ollama local daemon on identical hardware. Model: DeepSeek-R1-Distill-Qwen-7B-Q4 K M.gguf 4.68 GB | Metric | Glacier.Inference .NET 10 | Local Daemon Go + C++ CUDA | Margin | |---|---|---|---| | Runtime Architecture | Pure C Native AOT | C++ / cuBLAS / libllama | Zero external DLLs | | Binary Footprint | ~15 MB Single File | ~4.5 GB Toolkit + Engine | 300x smaller | | Cold Start to First Token | 1.50 s | 3.50+ s | 2.3x faster | | Serial Generation | 41.92 tok/s 208 GB/s | 43.20 tok/s 216 GB/s | Within 3% of cuBLAS | | Speculative Generation | 72.5 – 104.8 tok/s | N/A Serial decode | Up to 2.4x faster | | Sampling Overhead | ~3.2 µs In-VRAM | ~800 µs DtoH transfer | 250x reduction | Model: Qwen3-30B-A3B-Instruct-Q3 K L.gguf 13.58 GB MoE, 3B Active | Execution Pipeline | Memory Model | Generation Rate | Turnaround Time | |---|---|---|---| | Direct3D 12 Compute HLSL Wave32 | Unified LPDDR5X Direct | 21.68 tok/s | 1.54 s | | Host CPU SIMD 24T AVX-512 | System Memory | 0.89 tok/s | 18.20 s | Running a 30B parameter Mixture-of-Experts architecture in pure C directly on an integrated APU delivers 21.68 tok/s , beating multi-threaded AVX-512 CPU execution by 24.4x . Managed languages don't have to surrender low-level compute workloads to external runtime stacks. By combining MemoryMappedFile zero-allocation weight access, Direct3D 12 compute pipelines, direct driver P/Invokes, and in-VRAM warp reductions, pure .NET 10 delivers bare-metal throughput while keeping deployment to a single, portable binary. The complete code, benchmarks, and standalone CLI binaries are available on GitHub: Glacier.Inference https://github.com/ian-cowley/Glacier.Inference .