I've used AI through APIs for years — POST
a prompt, get tokens back, ship the
feature. I have never once deployed a model myself. No PyTorch, no GPU memory
math, no idea what actually happens between my HTTP request and the text that
comes back. This series is me closing that gap on purpose, one week at a time,
on an NVIDIA DGX Spark.
I'm a software engineer and technical program manager. I'm comfortable with
Linux, Python, Docker, Kubernetes, and APIs. I'm a complete beginner at machine
learning. So Week 1 is deliberately unglamorous: before running any model, I
want to know the machine — what CPU and GPU it has, how its memory works, and
what the NVIDIA software stack underneath is actually made of. Every claim below
is backed by a real command and its real output, so you can run the same thing
on your own box and compare.
From API to GPU is a 32-week journey from AI-API consumer to
local LLM systems architect — running, optimizing, and eventually
fine-tuning models on local hardware, documenting each week as a hands-on lab
plus a blog post. The full week-by-week plan lives in the roadmap1, and
every week's runnable code lands in the companion GitHub repo2. If you have
a similar machine, you can follow along and reproduce every result.
The plan runs in eight phases, with a parallel CUDA track starting around week 5:
| Phase | Weeks | Focus |
|---|---|---|
| 1 | 1–4 | Comfortable running local models |
| 2 | 5–8 | Enough ML to understand inference |
| 3 | 9–13 | Transformers and terminology |
| 4 | 14–16 | Quantization and model formats |
| 5 | 17–20 | Inference engineering |
| 6 | 21–24 | Production model services |
| 7 | 25–28 | RAG and application integration |
| 8 | 29–32 | Fine-tuning |
The goal of this first post is narrow on purpose: stand up and understand
the environment. By the end you'll be able to inventory a DGX Spark, read what
nvidia-smi
tells you, explain how its unified memory differs from a normal
GPU, untangle the NVIDIA driver/CUDA-runtime/toolkit layers, and prove the GPU
is usable from PyTorch with a measured CPU-vs-GPU speedup. No model yet — that's
week 2. This is the foundation everything else builds on.
All the commands in this post are packaged as a runnable script in the companion
repo under week-01-environment/
3 — clone it, set up an SSH alias spark
that reaches your DGX Spark, and run ./inventory.sh
to reproduce everything
here. The repo holds the commands and scripts; this post holds the explanations,
so neither repeats the other.
Here's the whole inventory script, so you can see exactly what it runs without
cloning anything. It just wraps each command in an SSH call to the Spark and
prints a labelled section:
#!/usr/bin/env bash
set -euo pipefail
SPARK_HOST="${SPARK_HOST:-spark}"
run() { echo "=== {% katex inline %}1 ==="; ssh "{% endkatex %}SPARK_HOST" "$2" 2>&1 || true; echo; }
run "uname" 'uname -a'
run "os-release" 'cat /etc/os-release'
run "lscpu" 'lscpu'
run "memory" 'free -h'
run "storage" 'lsblk'
run "gpu" 'nvidia-smi'
run "cuda-nvcc" 'nvcc --version || /usr/local/cuda/bin/nvcc --version'
run "cuda-dirs" 'ls -d /usr/local/cuda*'
run "python" 'python3 --version'
run "docker" 'docker version'
The rest of this post walks through the interesting parts of that output one
section at a time.
The setup is two machines. A MacBook Pro is the control machine — for writing,
editing, and opening SSH sessions. An NVIDIA DGX Spark is the workhorse — where
every model and every GPU command actually runs. I reach it over SSH as spark
.
A theme for this whole series is that I verify facts with a command instead of
assuming them — even the obvious ones. So rather than start by stating what my
machines are, I'll show them. First the control machine:
uname -m && sysctl -n machdep.cpu.brand_string
x86_64
Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
An Intel x86_64 Mac. This matters more than it looks, because it's a
different architecture from the Spark — so I want it on the record, not assumed.
Now the Spark:
ssh spark 'uname -a'
Linux spark-66b9 6.17.0-1026-nvidia ... aarch64 aarch64 aarch64 GNU/Linux
The Spark is aarch64 — ARM64, the same CPU family as phones and Apple
Silicon, but a different architecture from my Intel Mac.
| Machine | Role | Architecture | Verified by |
|---|---|---|---|
| MacBook Pro | control / authoring | x86_64 (Intel i5) | uname -m |
| DGX Spark | model + GPU work | aarch64 (ARM64) | uname -a |
This is not trivia. Because the two machines are different architectures, a
Python wheel or Docker image built for my Intel Mac will not necessarily run on
the ARM64 Spark. That is exactly why all the real work in this series happens
over ssh spark
, on the box itself — and why "it works on my laptop" means
nothing here.
The Spark's CPU is a 20-core ARM chip:
ssh spark 'lscpu'
Architecture: aarch64
CPU(s): 20
Model name: Cortex-X925 (10 performance cores)
Model name: Cortex-A725 (10 efficiency cores)
Twenty cores sounds like a lot, but for running a model the CPU is mostly a
traffic director: it runs the OS, my Python, and the data . The heavy
lifting happens on the GPU. Here's the intuition that finally made "why a GPU"
click for me.
A neural-network layer boils down to one operation repeated endlessly: multiply
a big grid of numbers (the model's weights, W ) by a list of numbers (the
input, x ) and add them up — a matrix multiplication. One output value is:
A model with billions of parameters does billions of these multiply-adds for a
single token. The magic property is that they're independent: computing
y1 doesn't need y2 . In terms I already know, that's embarrassingly parallel — like the map phase of a MapReduce where no shard waits on another.
That's the whole reason a GPU wins:
| Aspect | CPU (the 20 ARM cores) | GPU (the NVIDIA GB10) |
|---|---|---|
| Parallel workers | a few strong cores | thousands of small cores |
| Good at | branching logic, one-at-a-time | the same math on huge data at once |
| Analogy | a few expert chefs | a stadium of line cooks |
A CPU is a handful of very smart workers doing tasks in sequence. A GPU is
thousands of simpler workers all doing the identical multiply-add on different
numbers simultaneously. Since a model is nothing but that identical operation
repeated, the GPU is the right tool.
One caveat I'm carrying forward: those cores are useless if you can't feed
them numbers fast enough, so memory bandwidth — not raw compute — usually
limits how fast a model runs. I'll test the CPU-vs-GPU speed difference directly
with a matmul benchmark once PyTorch is installed; my prediction is a 10x–50x
GPU speedup, with a slow first GPU run due to one-time warmup.
nvidia-smi
: my new top
for the GPU
nvidia-smi
is the command I'll run every day from now on. It's the GPU
equivalent of top
or docker stats
. Here's the real output, lightly trimmed:
ssh spark 'nvidia-smi'
NVIDIA-SMI 580.159.03 Driver Version: 580.159.03 CUDA Version: 13.0
GPU 0: NVIDIA GB10 Persistence-M: On
Temp Perf Pwr:Usage/Cap Memory-Usage GPU-Util Compute M.
35C P8 4W / N/A Not Supported 0% Default
Field by field, and why each one will matter later:
| Field | Value | Meaning |
|---|---|---|
| Driver Version | 580.159.03 | kernel driver talking to the GPU |
| CUDA Version | 13.0 | |
| max CUDA the driver supports | ||
| GPU name | NVIDIA GB10 | the device (Grace-Blackwell) |
| Temp | 35C | die temperature (heat → throttling) |
| Perf | P8 | clock state, P0 = max … P8 = idle |
| Pwr:Usage/Cap | 4W / N/A | current vs max power draw |
| Memory-Usage | Not Supported | would show VRAM used/total — see below |
| GPU-Util | 0% | % of last sample the GPU was busy |
The bottom of the output also has a process table listing every PID holding the
GPU — right now just Xorg, GNOME, and Firefox using it for the desktop. That's
my first stop whenever I hit "out of memory": find the offender, like lsof
on
a stuck port.
For Week 1 I'm using nvidia-smi
purely as an inventory tool — what GPU,
what driver, what max CUDA, who's using it. The deeper use (streaming monitors,
reading utilization and memory bandwidth to decide if a workload is
compute-bound or memory-bound) is a profiling skill I'm deliberately saving for
the CUDA track around Week 5, so I don't tangle the two learning tracks.
The one field that stopped me was Memory-Usage: Not Supported
. On a normal PC
with a discrete GPU, that column is how you answer "did my model fit? how much
VRAM is left?" On the Spark it's blank — and that's not a bug, it's the whole
point of the machine.
A traditional GPU has its own separate memory (VRAM), physically distinct from
system RAM. The DGX Spark's GB10 is a Grace-Blackwell superchip that fuses the
ARM CPU and the GPU onto one package and gives them one shared memory pool:
ssh spark 'free -h'
total used free available
Mem: 121Gi 6.2Gi 67Gi 115Gi
| Traditional discrete GPU | DGX Spark (GB10) |
|---|---|
| System RAM (e.g. 64 GB) + separate VRAM (e.g. 24 GB) | one shared 121 GiB pool |
| Copy data RAM → VRAM over PCIe | CPU and GPU read the same memory |
| "Will it fit?" limited by VRAM (24 GB) | limited by total RAM (121 GB) |
This is unified memory. nvidia-smi
reports "Not Supported" for GPU memory
because there is no separate VRAM to report — the GPU's memory is the system's
121 GiB. In infra terms, a discrete GPU pays a "copy tax" moving weights across
the PCIe bus, like shuffling data between two services with separate caches;
unified memory removes that hop, like two services sharing one in-memory cache.
The practical consequence for me: on the Spark, the ceiling on model size isn't
a stingy 24 GB of VRAM — it's 121 GB. But I have to track model memory
differently, via free -h
or PyTorch's own counters, not the nvidia-smi
memory column. The trade-off (which I'll measure later) is that shared memory
usually has lower peak bandwidth than a top-end discrete card's dedicated VRAM,
so the Spark trades some raw speed for the ability to fit much larger models.
The most confusing part of the NVIDIA stack for a newcomer is that "CUDA" isn't
one thing — it's three separate layers, installed and versioned independently.
Mapping each to infrastructure I already understand finally made it stick:
| Layer | What it is | Infra analogy | Who needs it |
|---|---|---|---|
| NVIDIA driver | kernel module that talks to the GPU | a device driver | everyone using the GPU |
| CUDA runtime (libcudart) | shared libs an app calls to run GPU work | the .so libs you link |
|
| anyone running GPU programs | |||
| CUDA toolkit | the nvcc compiler, headers, profilers |
||
| gcc + headers + build tools | only people compiling CUDA |
The insight that unblocked me: you can run GPU code without the toolkit.
PyTorch ships its own copy of the CUDA runtime inside its wheel. So to run
models I need the driver (system-level) plus a CUDA-enabled PyTorch (which
brings its own runtime). I do not need nvcc
— that's only for compiling
custom CUDA kernels, a much-later CUDA-track activity.
With that lens, two clues from the inventory make sense. First, the header line
CUDA Version: 13.0
is the maximum CUDA the driver supports — a ceiling, not
what's installed. That's the number that matters this week: when I install
PyTorch, I must pick a CUDA build ≤ 13.0 so the driver can run it.
Second, the alarming-looking one:
ssh spark 'nvcc --version'
bash: nvcc: command not found
This looks broken but isn't. It only means the toolkit's compiler isn't on my
PATH
. The driver and runtime clearly work — nvidia-smi
talks to the GPU. And
the toolkit is physically installed:
ssh spark 'ls -d /usr/local/cuda*'
/usr/local/cuda /usr/local/cuda-13 /usr/local/cuda-13.0
So nvcc
exists; it's just at /usr/local/cuda/bin/nvcc
, not on PATH
. Rather
than assert that, I confirmed it by calling the full path:
ssh spark '/usr/local/cuda/bin/nvcc --version'
Cuda compilation tools, release 13.0, V13.0.88
The toolkit is version 13.0.88, matching the driver's CUDA 13.0 ceiling — a
healthy, consistent stack. Nothing is broken; command not found
was a PATH
issue, not a missing install. If I ever want nvcc
on PATH
, it's one line:
export PATH=/usr/local/cuda/bin:$PATH
. But to run models, I never need it.
PyTorch is the Python library I'll use to talk to the GPU. Before installing it,
one habit worth keeping: never install into the system Python. I use a
virtual environment (venv) — an isolated per-project Python with its own
packages, exactly like a per-service dependency sandbox so one project's
libraries can't break another's. On the Spark that's why the earlier
python3 -c "import torch"
failed: the system Python genuinely has no torch, and
I want to keep it that way.
If you're following along, first check whether you already have PyTorch — many
setups ship with it:
ssh spark 'python3 -c "import torch, sys; print(torch.__version__)" \
|| echo "no torch in this Python"'
I didn't, so I made a clean venv and installed torch into it:
ssh spark 'python3 -m venv ~/venvs/w1'
ssh spark '~/venvs/w1/bin/python -m pip install --upgrade pip'
ssh spark '~/venvs/w1/bin/python -m pip install torch numpy'
The install is packaged as setup.sh
in the companion repo, which just runs the
three steps above against requirements.txt
:
#!/usr/bin/env bash
set -euo pipefail
VENV="{% katex inline %}{VENV:-{% endkatex %}HOME/venvs/w1}"
HERE="{% katex inline %}(cd "{% endkatex %}(dirname "$0")" && pwd)"
python3 -m venv "$VENV"
"$VENV/bin/python" -m pip install --upgrade pip
"{% katex inline %}VENV/bin/python" -m pip install -r "{% endkatex %}HERE/requirements.txt"
The interesting part is what pip pulled in. Remember the stack tops out at CUDA
13.0 — and without me specifying any special index, PyPI served an ARM64 + CUDA 13 build automatically:
Successfully installed torch-2.13.0 nvidia-cuda-runtime-13.0.96
nvidia-cudnn-cu13-9.20.0.48 nvidia-cublas-13.1.1.3 ... (aarch64 wheels)
This is the payoff of the "PyTorch ships its own CUDA runtime" point from
earlier: I did not install the CUDA toolkit or touch nvcc
. Torch brought
its own CUDA 13 runtime libraries (libcudart
, cuDNN
, cuBLAS
) as ordinary
Python wheels, matched to the ARM64 architecture and the driver's CUDA 13
ceiling. The driver was the only piece I needed pre-installed.
Now the moment this whole week builds to: does Python see the GPU? The
validation script (validate_gpu.py
in the repo) is deliberately tiny:
import torch
print("PyTorch:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("CUDA version (torch):", torch.version.cuda)
if torch.cuda.is_available():
print("Device:", torch.cuda.get_device_name(0))
print("Capability:", torch.cuda.get_device_capability(0))
x = torch.rand(3, 3, device="cuda") # put a real tensor on the GPU
print("Tensor on:", x.device)
Run it with the venv's Python:
ssh spark '~/venvs/w1/bin/python validate_gpu.py'
PyTorch: 2.13.0+cu130
CUDA available: True
CUDA version (torch): 13.0
Device: NVIDIA GB10
Capability: (12, 1)
Tensor on: cuda:0
Every line matters. CUDA available: True
means PyTorch found a usable GPU
through the driver. 2.13.0+cu130
confirms it's a CUDA 13 build. Device: NVIDIA
is our chip.
GB10Capability: (12, 1)
is the GPU's compute capability —
NVIDIA's versioning for GPU features; 12.x
is the Blackwell generation. And
Tensor on: cuda:0
is the real proof: I created a tensor and it physically lives
in GPU memory, not on the CPU. This is the line that says "I can run models on
this machine now."
Earlier I predicted the GPU would beat the CPU by 10x-50x on a large matrix
multiply. Time to measure instead of hand-wave. The benchmark
(benchmark.py
) multiplies two 4096x4096 matrices 20 times on each device and
averages. Two details make the GPU timing honest: a warmup (the first GPU
call pays a one-time kernel-load cost, so I run a few throwaway iterations
first), and torch.cuda.synchronize()
before stopping the clock (CUDA launches
kernels asynchronously, so without a sync I'd be timing queueing, not
computing):
def bench(a, b, sync=False):
if sync: torch.cuda.synchronize()
t0 = time.perf_counter()
for _ in range(ITERS):
c = a @ b
if sync: torch.cuda.synchronize()
return (time.perf_counter() - t0) / ITERS
cpu_s = bench(a, b) # on CPU
ag, bg = a.cuda(), b.cuda()
for _ in range(3): _ = ag @ bg # warmup
gpu_s = bench(ag, bg, sync=True) # on GPU
ssh spark '~/venvs/w1/bin/python benchmark.py'
Matrix: 4096x4096 float32, iters: 20
CPU ms/matmul: 173.8 GFLOP/s: 790.8
GPU ms/matmul: 7.53 GFLOP/s: 18263.5
Speedup (CPU/GPU): 23.1 x
A single 4096x4096 float32 matmul is about 137 billion floating-point
operations. The 20-core ARM CPU chews through it in ~174 ms (~0.79 TFLOP/s); the
GB10 does it in ~7.5 ms (~18.3 TFLOP/s) — a 23x speedup, squarely inside my
predicted 10x-50x range. That number is the reason this hardware exists: the
same math, done thousands-at-a-time instead of a-few-at-a-time. (These are the
raw values in results.json
in the repo; yours will differ by machine.)
nvcc: command not found
is not an error state — the CUDA that matters for running models lives inside PyTorch, not in the toolkit. Installing torch pulled its own CUDA 13 runtime as plain wheels; I never touched the toolkit.Week 1 is done: I know the machine, I understand the NVIDIA stack, and I've proven
the GPU is usable from Python with a real speedup. Week 2 leaves inventory behind
and runs an actual language model with Ollama — a model runtime with a local
HTTP API — where I'll start measuring the things that matter for serving: tokens
per second and time to first token.
32-week roadmap —
https://github.com/dramasamy/from-api-to-gpu/tree/main/roadmap ↩
Companion code repository —
https://github.com/dramasamy/from-api-to-gpu ↩
Week 1 lab —
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-01-environment ↩