{"slug": "why-local-llms-don-t-need-c-or-python-building-a-15mb-native-aot-inference-in-10", "title": "Why Local LLMs Don't Need C++ or Python: Building a 15MB Native AOT Inference Engine in .NET 10", "summary": "A developer built Glacier.Inference, a ~15 MB single-file Native AOT inference engine written in pure C# on .NET 10 that runs GGUF models directly across NVIDIA, AMD, and Intel GPUs without C++ binaries or the CUDA runtime. The engine bypasses cudart64.dll and cublas64.dll by dispatching precompiled cubins straight to NVIDIA SMs via P/Invoke to nvcuda.dll, and fuses GPU-side argmax reduction so only a 4-byte token ID crosses the bus per step instead of 608 KB of logits. It also includes a Direct3D 12 compute path to avoid Windows TDR driver resets on laptops where integrated graphics drive the display.", "body_md": "The conventional consensus across AI engineering is simple: high-performance local LLM execution belongs exclusively to C++ runtimes, multi-gigabyte CUDA toolkits, and bindings over `llama.cpp` or `vLLM`.\n\nWhen orchestrating local models from managed languages like C#, typical implementations rely on interop wrappers over unmanaged native binaries (`cudart64.dll`, `cublas64.dll`, or `libllama`). This introduces DLL distribution overhead, host-to-device PCIe bandwidth bottlenecks during token sampling, and severe desktop instability when running compute on display-bound integrated GPUs.\n\nBy stripping out runtime layers and interacting directly with driver interfaces, managed runtimes can match and outpace conventional native daemons. **Glacier.Inference** runs direct memory-mapped GGUF models in pure C# .NET 10 across NVIDIA, AMD, and Intel silicon without external C++ binaries.\n\n```\n┌────────────────────────────────────────────────────────────────────────┐\n│                        Glacier.Inference Core                          │\n│  ├─ MemoryMappedFile Zero-Copy GGUF Reader (Sub-30ms cold mapping)     │\n│  ├─ Pure C# Bare-Metal SASS Engine (Direct P/Invoke nvcuda.dll)        │\n│  ├─ Bare-Metal Direct3D 12 Compute (HLSL Wave32 via Vortice.D3D12)     │\n│  ├─ SpeculativeEngine (N-gram Prompt Lookup & Batched Verification)    │\n│  ├─ Fused In-VRAM GPU Argmax Reduction (Warp-shuffle, 4-byte transfer) │\n│  └─ Adaptive Unmanaged KV-Cache (FP16 / FP8 Dynamic Ring Buffer)       │\n└────────────────────────────────────────────────────────────────────────┘\n```\n\nTypical CUDA execution routes instructions through `cudart64.dll` and `cublas64.dll`. Glacier completely circumvents the CUDA runtime layer. It communicates directly with the base Windows GPU driver (`nvcuda.dll`) via low-level P/Invoke, dispatching precompiled fatbinaries straight to the Streaming Multiprocessors (SMs).\n\n```\n// Direct driver context and module dispatch without cudart64.dll\n[DllImport(\"nvcuda.dll\", EntryPoint = \"cuLaunchKernel\")]\npublic static unsafe extern CUresult LaunchKernel(\n    CUfunction f,\n    uint gridDimX, uint gridDimY, uint gridDimZ,\n    uint blockDimX, uint blockDimY, uint blockDimZ,\n    uint sharedMemBytes,\n    CUstream hStream,\n    void** kernelParams,\n    void** extra);\n```\n\nBy compiling modular `.cuh` routines into an embedded universal `cubin` containing dedicated slices (`sm_75`, `sm_80`, `sm_86`, `sm_89`, `sm_90`), the engine verifies zero DRAM stack spills (` STACK: 0` via `cuobjdump`). Registers house all dequantization multipliers and accumulators without hitting DRAM stack frames.\n\nThe result is a self-contained ~15 MB single-file Native AOT executable replacing a 4.5 GB toolchain.\n\nIn standard architectures, evaluating greedy token selection involves transferring logit tensors back over PCIe to the host CPU:\n\n$$\\text{Logit Transfer Per Step} = 152\\text{K vocab} \\times 4\\text{ bytes} \\approx 608\\text{ KB}$$\n\nAt 45 tokens per second, transferring 608 KB back and forth over the host interface introduces micro-stalls and PCIe latency. Glacier fuses the final linear projection and reduction directly on the device with a 512-thread warp-shuffle kernel:\n\n```\n// GPU-side warp reduction eliminating CPU transfer overhead\n[numthreads(512, 1, 1)]\nvoid ArgmaxReduction(uint3 tid : SV_DispatchThreadID, uint3 lid : SV_GroupThreadID) {\n    // 512-thread tree reduction within registers across 152k logits\n    // Emits exactly 1 int32 winning token index into device memory\n}\n```\n\nInstead of copying 608 KB across the bus every token, Glacier transfers **exactly 4 bytes (one `int32` token ID)**, dropping reduction latency to ~3.2 µs.\n\nDeploying local inference on consumer hardware often fails on laptops where AMD RDNA or Intel Arc graphics double as the primary display adapter. Long compute dispatches trigger Windows Timeout Detection and Recovery (TDR), resetting the display driver.\n\nGlacier implements a pure Direct3D 12 compute engine using `Vortice.D3D12` and HLSL Wave32 compute shaders:\n\nAutoregressive transformer generation is inherently memory-bandwidth bound. Every generated token requires streaming the entire model weight footprint through the compute core.\n\nOn a 128-bit GDDR6 memory bus running at 256 GB/s, reading a 4.68 GB model sets a strict theoretical wall clock limit:\n\n$$\\text{Max Serial Rate} = \\frac{256\\text{ GB/s}}{4.68\\text{ GB}} \\approx 54.7\\text{ tokens/sec}$$\n\nGlacier integrates batched speculative verification (`VerifyBatch`) using zero-cost prompt suffix lookup (` PromptLookupDraftProvider`):\n\n``` js\nusing var target = new InferenceSession(\"models/Qwen2.5-7B-Instruct-Q4_K_M.gguf\");\nusing var engine = new SpeculativeEngine(target);\n\nvar options = new SpeculativeOptions\n{\n    MaxDraftTokens = 4, // Propose 4 candidate tokens in <1 µs\n    MaxTokens = 256\n};\n\nvar result = await engine.GenerateAsync(\"Explain quicksort in C#\", options);\n```\n\nInstead of reading 4.68 GB from VRAM $K$ times for $K$ tokens, the candidate sequence is verified against the transformer in a single batch pass. The weights are pulled through the memory bus **only once**, accelerating generation rates to **72–104+ tokens/sec** on an RTX 4060 laptop GPU.\n\nThe following runs compare Glacier against an Ollama local daemon on identical hardware.\n\n*Model: `DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf` (4.68 GB)*\n\n| Metric | Glacier.Inference (.NET 10) | Local Daemon (Go + C++ CUDA) | Margin | \n|---|---|---|---|\n| **Runtime Architecture** | **Pure C# (Native AOT)** | C++ / cuBLAS / libllama | Zero external DLLs | \n| **Binary Footprint** | **~15 MB Single File** | ~4.5 GB Toolkit + Engine | **300x smaller** | \n| **Cold Start to First Token** | **1.50 s** | 3.50+ s | **2.3x faster** | \n| **Serial Generation** | **41.92 tok/s** (208 GB/s) | 43.20 tok/s (216 GB/s) | Within 3% of cuBLAS | \n| **Speculative Generation** | **72.5 – 104.8 tok/s** | N/A (Serial decode) | **Up to 2.4x faster** | \n| **Sampling Overhead** | **~3.2 µs (In-VRAM)** | ~800 µs (DtoH transfer) | **250x reduction** | \n\n*Model: `Qwen3-30B-A3B-Instruct-Q3_K_L.gguf` (13.58 GB MoE, 3B Active)*\n\n| Execution Pipeline | Memory Model | Generation Rate | Turnaround Time | \n|---|---|---|---|\n| **Direct3D 12 Compute (HLSL Wave32)** | **Unified LPDDR5X (Direct)** | **21.68 tok/s** | **1.54 s** | \n| **Host CPU SIMD (24T AVX-512)** | System Memory | 0.89 tok/s | 18.20 s | \n\nRunning a 30B parameter Mixture-of-Experts architecture in pure C# directly on an integrated APU delivers **21.68 tok/s**, beating multi-threaded AVX-512 CPU execution by **24.4x**.\n\nManaged languages don't have to surrender low-level compute workloads to external runtime stacks. By combining `MemoryMappedFile` zero-allocation weight access, Direct3D 12 compute pipelines, direct driver P/Invokes, and in-VRAM warp reductions, pure .NET 10 delivers bare-metal throughput while keeping deployment to a single, portable binary.\n\nThe complete code, benchmarks, and standalone CLI binaries are available on GitHub: [Glacier.Inference](https://github.com/ian-cowley/Glacier.Inference).", "url": "https://wpnews.pro/news/why-local-llms-don-t-need-c-or-python-building-a-15mb-native-aot-inference-in-10", "canonical_source": "https://dev.to/iancowley/why-local-llms-dont-need-c-or-python-building-a-15mb-native-aot-inference-engine-in-net-10-1m2d", "published_at": "2026-09-13 10:27:57+00:00", "updated_at": "2026-09-13 10:34:35.466055+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "developer-tools", "mlops"], "entities": ["Glacier.Inference", ".NET 10", "C#", "NVIDIA", "AMD", "Intel", "CUDA", "Direct3D 12"], "alternates": {"html": "https://wpnews.pro/news/why-local-llms-don-t-need-c-or-python-building-a-15mb-native-aot-inference-in-10", "markdown": "https://wpnews.pro/news/why-local-llms-don-t-need-c-or-python-building-a-15mb-native-aot-inference-in-10.md", "text": "https://wpnews.pro/news/why-local-llms-don-t-need-c-or-python-building-a-15mb-native-aot-inference-in-10.txt", "jsonld": "https://wpnews.pro/news/why-local-llms-don-t-need-c-or-python-building-a-15mb-native-aot-inference-in-10.jsonld"}}