# From API to GPU, Week 1: Understanding NVIDIA DGX Spark Environment > Source: > Published: 2026-07-11 20:57:28+00:00 I've used AI through APIs for years — `POST` a prompt, get tokens back, ship the feature. I have never once deployed a model myself. No PyTorch, no GPU memory math, no idea what actually happens between my HTTP request and the text that comes back. This series is me closing that gap on purpose, one week at a time, on an NVIDIA DGX Spark. I'm a software engineer and technical program manager. I'm comfortable with Linux, Python, Docker, Kubernetes, and APIs. I'm a complete beginner at machine learning. So Week 1 is deliberately unglamorous: before running any model, I want to *know the machine* — what CPU and GPU it has, how its memory works, and what the NVIDIA software stack underneath is actually made of. Every claim below is backed by a real command and its real output, so you can run the same thing on your own box and compare. *From API to GPU* is a 32-week journey from **AI-API consumer** to **local LLM systems architect** — running, optimizing, and eventually fine-tuning models on local hardware, documenting each week as a hands-on lab plus a blog post. The full week-by-week plan lives in the roadmap1, and every week's runnable code lands in the companion GitHub repo2. If you have a similar machine, you can follow along and reproduce every result. The plan runs in eight phases, with a parallel CUDA track starting around week 5: | Phase | Weeks | Focus | |---|---|---| | 1 | 1–4 | Comfortable running local models | | 2 | 5–8 | Enough ML to understand inference | | 3 | 9–13 | Transformers and terminology | | 4 | 14–16 | Quantization and model formats | | 5 | 17–20 | Inference engineering | | 6 | 21–24 | Production model services | | 7 | 25–28 | RAG and application integration | | 8 | 29–32 | Fine-tuning | **The goal of this first post** is narrow on purpose: stand up and *understand* the environment. By the end you'll be able to inventory a DGX Spark, read what `nvidia-smi` tells you, explain how its unified memory differs from a normal GPU, untangle the NVIDIA driver/CUDA-runtime/toolkit layers, and prove the GPU is usable from PyTorch with a measured CPU-vs-GPU speedup. No model yet — that's week 2. This is the foundation everything else builds on. All the commands in this post are packaged as a runnable script in the companion repo under `week-01-environment/` 3 — clone it, set up an SSH alias `spark` that reaches your DGX Spark, and run `./inventory.sh` to reproduce everything here. The repo holds the commands and scripts; this post holds the explanations, so neither repeats the other. Here's the whole inventory script, so you can see exactly what it runs without cloning anything. It just wraps each command in an SSH call to the Spark and prints a labelled section: ``` bash #!/usr/bin/env bash # Week 1 — DGX Spark machine inventory. Prereq: an SSH alias `spark`. set -euo pipefail SPARK_HOST="${SPARK_HOST:-spark}" run() { echo "=== {% katex inline %}1 ==="; ssh "{% endkatex %}SPARK_HOST" "$2" 2>&1 || true; echo; } run "uname" 'uname -a' run "os-release" 'cat /etc/os-release' run "lscpu" 'lscpu' run "memory" 'free -h' run "storage" 'lsblk' run "gpu" 'nvidia-smi' run "cuda-nvcc" 'nvcc --version || /usr/local/cuda/bin/nvcc --version' run "cuda-dirs" 'ls -d /usr/local/cuda*' run "python" 'python3 --version' run "docker" 'docker version' ``` The rest of this post walks through the interesting parts of that output one section at a time. The setup is two machines. A MacBook Pro is the *control* machine — for writing, editing, and opening SSH sessions. An NVIDIA DGX Spark is the *workhorse* — where every model and every GPU command actually runs. I reach it over SSH as `spark` . A theme for this whole series is that I verify facts with a command instead of assuming them — even the obvious ones. So rather than start by *stating* what my machines are, I'll show them. First the control machine: ``` # On the control MacBook uname -m && sysctl -n machdep.cpu.brand_string x86_64 Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz ``` An **Intel x86_64** Mac. This matters more than it looks, because it's a different architecture from the Spark — so I want it on the record, not assumed. Now the Spark: ``` ssh spark 'uname -a' Linux spark-66b9 6.17.0-1026-nvidia ... aarch64 aarch64 aarch64 GNU/Linux ``` The Spark is **aarch64** — ARM64, the same CPU family as phones and Apple Silicon, but a different architecture from my Intel Mac. | Machine | Role | Architecture | Verified by | |---|---|---|---| | MacBook Pro | control / authoring | x86_64 (Intel i5) | `uname -m` | | DGX Spark | model + GPU work | aarch64 (ARM64) | `uname -a` | This is not trivia. Because the two machines are **different architectures**, a Python wheel or Docker image built for my Intel Mac will not necessarily run on the ARM64 Spark. That is exactly why all the real work in this series happens over `ssh spark` , on the box itself — and why "it works on my laptop" means nothing here. The Spark's CPU is a 20-core ARM chip: ``` ssh spark 'lscpu' Architecture: aarch64 CPU(s): 20 Model name: Cortex-X925 (10 performance cores) Model name: Cortex-A725 (10 efficiency cores) ``` Twenty cores sounds like a lot, but for running a model the CPU is mostly a traffic director: it runs the OS, my Python, and the data loading. The heavy lifting happens on the GPU. Here's the intuition that finally made "why a GPU" click for me. A neural-network layer boils down to one operation repeated endlessly: multiply a big grid of numbers (the model's **weights**, W ) by a list of numbers (the input, x ) and add them up — a **matrix multiplication**. One output value is: A model with billions of parameters does *billions* of these multiply-adds for a single token. The magic property is that they're **independent**: computing y1 doesn't need y2 . In terms I already know, that's *embarrassingly parallel* — like the map phase of a MapReduce where no shard waits on another. That's the whole reason a GPU wins: | Aspect | CPU (the 20 ARM cores) | GPU (the NVIDIA GB10) | |---|---|---| | Parallel workers | a few strong cores | thousands of small cores | | Good at | branching logic, one-at-a-time | the same math on huge data at once | | Analogy | a few expert chefs | a stadium of line cooks | A CPU is a handful of very smart workers doing tasks in sequence. A GPU is thousands of simpler workers all doing the *identical* multiply-add on different numbers simultaneously. Since a model is nothing but that identical operation repeated, the GPU is the right tool. One caveat I'm carrying forward: those cores are useless if you can't *feed* them numbers fast enough, so **memory bandwidth** — not raw compute — usually limits how fast a model runs. I'll test the CPU-vs-GPU speed difference directly with a matmul benchmark once PyTorch is installed; my prediction is a 10x–50x GPU speedup, with a slow first GPU run due to one-time warmup. `nvidia-smi` : my new `top` for the GPU `nvidia-smi` is the command I'll run every day from now on. It's the GPU equivalent of `top` or `docker stats` . Here's the real output, lightly trimmed: ``` ssh spark 'nvidia-smi' NVIDIA-SMI 580.159.03 Driver Version: 580.159.03 CUDA Version: 13.0 GPU 0: NVIDIA GB10 Persistence-M: On Temp Perf Pwr:Usage/Cap Memory-Usage GPU-Util Compute M. 35C P8 4W / N/A Not Supported 0% Default ``` Field by field, and why each one will matter later: | Field | Value | Meaning | |---|---|---| | Driver Version | 580.159.03 | kernel driver talking to the GPU | | CUDA Version | 13.0 | max CUDA the driver supports | | GPU name | NVIDIA GB10 | the device (Grace-Blackwell) | | Temp | 35C | die temperature (heat → throttling) | | Perf | P8 | clock state, P0 = max … P8 = idle | | Pwr:Usage/Cap | 4W / N/A | current vs max power draw | | Memory-Usage | Not Supported | would show VRAM used/total — see below | | GPU-Util | 0% | % of last sample the GPU was busy | The bottom of the output also has a process table listing every PID holding the GPU — right now just Xorg, GNOME, and Firefox using it for the desktop. That's my first stop whenever I hit "out of memory": find the offender, like `lsof` on a stuck port. For Week 1 I'm using `nvidia-smi` purely as an **inventory** tool — what GPU, what driver, what max CUDA, who's using it. The deeper use (streaming monitors, reading utilization and memory bandwidth to decide if a workload is compute-bound or memory-bound) is a profiling skill I'm deliberately saving for the CUDA track around Week 5, so I don't tangle the two learning tracks. The one field that stopped me was `Memory-Usage: Not Supported` . On a normal PC with a discrete GPU, that column is how you answer "did my model fit? how much VRAM is left?" On the Spark it's blank — and that's not a bug, it's the whole point of the machine. A traditional GPU has its own separate memory (VRAM), physically distinct from system RAM. The DGX Spark's GB10 is a Grace-Blackwell superchip that fuses the ARM CPU and the GPU onto one package and gives them **one shared memory pool**: ``` ssh spark 'free -h' total used free available Mem: 121Gi 6.2Gi 67Gi 115Gi ``` | Traditional discrete GPU | DGX Spark (GB10) | |---|---| | System RAM (e.g. 64 GB) + separate VRAM (e.g. 24 GB) | one shared 121 GiB pool | | Copy data RAM → VRAM over PCIe | CPU and GPU read the same memory | | "Will it fit?" limited by VRAM (24 GB) | limited by total RAM (121 GB) | This is **unified memory**. `nvidia-smi` reports "Not Supported" for GPU memory because there is no separate VRAM to report — the GPU's memory *is* the system's 121 GiB. In infra terms, a discrete GPU pays a "copy tax" moving weights across the PCIe bus, like shuffling data between two services with separate caches; unified memory removes that hop, like two services sharing one in-memory cache. The practical consequence for me: on the Spark, the ceiling on model size isn't a stingy 24 GB of VRAM — it's 121 GB. But I have to track model memory differently, via `free -h` or PyTorch's own counters, not the `nvidia-smi` memory column. The trade-off (which I'll measure later) is that shared memory usually has lower peak bandwidth than a top-end discrete card's dedicated VRAM, so the Spark trades some raw speed for the ability to fit much larger models. The most confusing part of the NVIDIA stack for a newcomer is that "CUDA" isn't one thing — it's three separate layers, installed and versioned independently. Mapping each to infrastructure I already understand finally made it stick: | Layer | What it is | Infra analogy | Who needs it | |---|---|---|---| | NVIDIA driver | kernel module that talks to the GPU | a device driver | everyone using the GPU | | CUDA runtime (libcudart) | shared libs an app calls to run GPU work | the `.so` libs you link | anyone running GPU programs | | CUDA toolkit | the `nvcc` compiler, headers, profilers | gcc + headers + build tools | only people compiling CUDA | The insight that unblocked me: **you can run GPU code without the toolkit.** PyTorch ships its own copy of the CUDA runtime inside its wheel. So to run models I need the *driver* (system-level) plus a *CUDA-enabled PyTorch* (which brings its own runtime). I do **not** need `nvcc` — that's only for compiling custom CUDA kernels, a much-later CUDA-track activity. With that lens, two clues from the inventory make sense. First, the header line `CUDA Version: 13.0` is the **maximum** CUDA the driver supports — a ceiling, not what's installed. That's the number that matters this week: when I install PyTorch, I must pick a CUDA build ≤ 13.0 so the driver can run it. Second, the alarming-looking one: ``` ssh spark 'nvcc --version' bash: nvcc: command not found ``` This looks broken but isn't. It only means the toolkit's compiler isn't on my `PATH` . The driver and runtime clearly work — `nvidia-smi` talks to the GPU. And the toolkit is physically installed: ``` ssh spark 'ls -d /usr/local/cuda*' /usr/local/cuda /usr/local/cuda-13 /usr/local/cuda-13.0 ``` So `nvcc` exists; it's just at `/usr/local/cuda/bin/nvcc` , not on `PATH` . Rather than assert that, I confirmed it by calling the full path: ``` ssh spark '/usr/local/cuda/bin/nvcc --version' Cuda compilation tools, release 13.0, V13.0.88 ``` The toolkit is version **13.0.88**, matching the driver's CUDA 13.0 ceiling — a healthy, consistent stack. Nothing is broken; `command not found` was a `PATH` issue, not a missing install. If I ever want `nvcc` on `PATH` , it's one line: `export PATH=/usr/local/cuda/bin:$PATH` . But to *run* models, I never need it. PyTorch is the Python library I'll use to talk to the GPU. Before installing it, one habit worth keeping: never install into the system Python. I use a **virtual environment** (venv) — an isolated per-project Python with its own packages, exactly like a per-service dependency sandbox so one project's libraries can't break another's. On the Spark that's why the earlier `python3 -c "import torch"` failed: the system Python genuinely has no torch, and I want to keep it that way. If you're following along, first check whether you already have PyTorch — many setups ship with it: ``` python ssh spark 'python3 -c "import torch, sys; print(torch.__version__)" \ || echo "no torch in this Python"' ``` I didn't, so I made a clean venv and installed torch into it: ``` ssh spark 'python3 -m venv ~/venvs/w1' ssh spark '~/venvs/w1/bin/python -m pip install --upgrade pip' ssh spark '~/venvs/w1/bin/python -m pip install torch numpy' ``` The install is packaged as `setup.sh` in the companion repo, which just runs the three steps above against `requirements.txt` : ``` bash #!/usr/bin/env bash set -euo pipefail VENV="{% katex inline %}{VENV:-{% endkatex %}HOME/venvs/w1}" HERE="{% katex inline %}(cd "{% endkatex %}(dirname "$0")" && pwd)" python3 -m venv "$VENV" "$VENV/bin/python" -m pip install --upgrade pip "{% katex inline %}VENV/bin/python" -m pip install -r "{% endkatex %}HERE/requirements.txt" ``` The interesting part is what pip pulled in. Remember the stack tops out at CUDA 13.0 — and without me specifying any special index, PyPI served an **ARM64 + CUDA 13** build automatically: ``` Successfully installed torch-2.13.0 nvidia-cuda-runtime-13.0.96 nvidia-cudnn-cu13-9.20.0.48 nvidia-cublas-13.1.1.3 ... (aarch64 wheels) ``` This is the payoff of the "PyTorch ships its own CUDA runtime" point from earlier: I did **not** install the CUDA toolkit or touch `nvcc` . Torch brought its own CUDA 13 runtime libraries (`libcudart` , `cuDNN` , `cuBLAS` ) as ordinary Python wheels, matched to the ARM64 architecture and the driver's CUDA 13 ceiling. The driver was the only piece I needed pre-installed. Now the moment this whole week builds to: does Python see the GPU? The validation script (`validate_gpu.py` in the repo) is deliberately tiny: ``` python import torch print("PyTorch:", torch.__version__) print("CUDA available:", torch.cuda.is_available()) print("CUDA version (torch):", torch.version.cuda) if torch.cuda.is_available(): print("Device:", torch.cuda.get_device_name(0)) print("Capability:", torch.cuda.get_device_capability(0)) x = torch.rand(3, 3, device="cuda") # put a real tensor on the GPU print("Tensor on:", x.device) ``` Run it with the venv's Python: ``` ssh spark '~/venvs/w1/bin/python validate_gpu.py' PyTorch: 2.13.0+cu130 CUDA available: True CUDA version (torch): 13.0 Device: NVIDIA GB10 Capability: (12, 1) Tensor on: cuda:0 ``` Every line matters. `CUDA available: True` means PyTorch found a usable GPU through the driver. `2.13.0+cu130` confirms it's a CUDA 13 build. `Device: NVIDIA` is our chip. GB10`Capability: (12, 1)` is the GPU's *compute capability* — NVIDIA's versioning for GPU features; `12.x` is the Blackwell generation. And `Tensor on: cuda:0` is the real proof: I created a tensor and it physically lives in GPU memory, not on the CPU. This is the line that says "I can run models on this machine now." Earlier I predicted the GPU would beat the CPU by 10x-50x on a large matrix multiply. Time to measure instead of hand-wave. The benchmark (`benchmark.py` ) multiplies two 4096x4096 matrices 20 times on each device and averages. Two details make the GPU timing honest: a **warmup** (the first GPU call pays a one-time kernel-load cost, so I run a few throwaway iterations first), and `torch.cuda.synchronize()` before stopping the clock (CUDA launches kernels asynchronously, so without a sync I'd be timing *queueing*, not *computing*): ``` python def bench(a, b, sync=False): if sync: torch.cuda.synchronize() t0 = time.perf_counter() for _ in range(ITERS): c = a @ b if sync: torch.cuda.synchronize() return (time.perf_counter() - t0) / ITERS cpu_s = bench(a, b) # on CPU ag, bg = a.cuda(), b.cuda() for _ in range(3): _ = ag @ bg # warmup gpu_s = bench(ag, bg, sync=True) # on GPU ssh spark '~/venvs/w1/bin/python benchmark.py' Matrix: 4096x4096 float32, iters: 20 CPU ms/matmul: 173.8 GFLOP/s: 790.8 GPU ms/matmul: 7.53 GFLOP/s: 18263.5 Speedup (CPU/GPU): 23.1 x ``` A single 4096x4096 float32 matmul is about 137 billion floating-point operations. The 20-core ARM CPU chews through it in ~174 ms (~0.79 TFLOP/s); the GB10 does it in ~7.5 ms (~18.3 TFLOP/s) — a **23x speedup**, squarely inside my predicted 10x-50x range. That number *is* the reason this hardware exists: the same math, done thousands-at-a-time instead of a-few-at-a-time. (These are the raw values in `results.json` in the repo; yours will differ by machine.) `nvcc: command not found` is not an error state — the CUDA that matters for running models lives inside PyTorch, not in the toolkit. Installing torch pulled its own CUDA 13 runtime as plain wheels; I never touched the toolkit.Week 1 is done: I know the machine, I understand the NVIDIA stack, and I've proven the GPU is usable from Python with a real speedup. Week 2 leaves inventory behind and runs an actual language model with Ollama — a model runtime with a local HTTP API — where I'll start measuring the things that matter for serving: tokens per second and time to first token. 32-week roadmap — [https://github.com/dramasamy/from-api-to-gpu/tree/main/roadmap](https://github.com/dramasamy/from-api-to-gpu/tree/main/roadmap) ↩ Companion code repository — [https://github.com/dramasamy/from-api-to-gpu](https://github.com/dramasamy/from-api-to-gpu) ↩ Week 1 lab — [https://github.com/dramasamy/from-api-to-gpu/tree/main/week-01-environment](https://github.com/dramasamy/from-api-to-gpu/tree/main/week-01-environment) ↩