# Building an LLM System from Scratch in Pure Python & NumPy: Architecture, Invariants, and Clean Code

> Source: <https://dev.to/ducnguyen-creator/building-an-llm-system-from-scratch-in-pure-python-numpy-architecture-invariants-and-clean-code-5a8c>
> Published: 2026-07-11 00:56:51+00:00

Hi everyone,

I wanted to share the design invariants and architecture of **Draco AI**, a full-stack, hardware-agnostic LLM system built completely from scratch in pure Python and NumPy. The core philosophy behind this project is strict code quality, mathematical correctness, and clean architecture—completely eliminating heavy framework dependencies like PyTorch or HuggingFace.

⚠️

100% Fully Open-Source:To clarify, this isnota half-baked open-source release where only the inference wrapper is public. Draco AI is 100% open-source across its entire execution path, including BOTH deep decoupled layers: the execution mechanics of theInference Engine(`modeling/`

) and the cognitive/reasoning paths of theThinking Engine(`thinking_engine/`

).

The baseline includes a robust pytest validation suite covering GQA, MLA, hybrid attention, Medusa heads, and speculative tree decoding execution paths.

`Y = (X @ pos_mask.T - X @ neg_mask.T) * scale`

) without floating-point multiplications on the core execution path.`HealthMonitor`

tracking NaN/Inf propagation, saturation, and adversarial expert collapse. It signals a `DynamicPrecisionManager`

for advisory dtype escalation/de-escalation based on overflow EMA while strictly respecting a hard VRAM budget.To prevent silent corruptions common in speculative systems, Draco AI enforces strict execution invariants:

`runtime → layers → ops → kernels`

. No upward execution calls are permitted. `device.py`

acts as the single source of truth for hardware capabilities.`rope_offset`

is captured exactly `SnapshotStack`

. It restores both the `KVCache`

slab and the `EngramCache`

pointers to their exact states `add_noise=False`

on replay to eliminate stochastic routing drift.I’d highly appreciate your insights regarding the mathematical fusion choices for uncertainty assessment, the decoupled architecture, or the pure-NumPy tensor layout. Let's discuss in the comments below!