{"slug": "from-api-to-gpu-week-1-understanding-nvidia-dgx-spark-environment", "title": "From API to GPU, Week 1: Understanding NVIDIA DGX Spark Environment", "summary": "A software engineer and technical program manager documents the first week of a 32-week journey from AI-API consumer to local LLM systems architect using an NVIDIA DGX Spark. The engineer inventories the machine's hardware and software stack, including CPU, GPU, memory, and NVIDIA driver layers, without running any models yet. The goal is to build a foundation for running, optimizing, and fine-tuning models locally.", "body_md": "I've used AI through APIs for years — `POST`\n\na prompt, get tokens back, ship the\n\nfeature. I have never once deployed a model myself. No PyTorch, no GPU memory\n\nmath, no idea what actually happens between my HTTP request and the text that\n\ncomes back. This series is me closing that gap on purpose, one week at a time,\n\non an NVIDIA DGX Spark.\n\nI'm a software engineer and technical program manager. I'm comfortable with\n\nLinux, Python, Docker, Kubernetes, and APIs. I'm a complete beginner at machine\n\nlearning. So Week 1 is deliberately unglamorous: before running any model, I\n\nwant to *know the machine* — what CPU and GPU it has, how its memory works, and\n\nwhat the NVIDIA software stack underneath is actually made of. Every claim below\n\nis backed by a real command and its real output, so you can run the same thing\n\non your own box and compare.\n\n*From API to GPU* is a 32-week journey from **AI-API consumer** to\n\n**local LLM systems architect** — running, optimizing, and eventually\n\nfine-tuning models on local hardware, documenting each week as a hands-on lab\n\nplus a blog post. The full week-by-week plan lives in the roadmap1, and\n\nevery week's runnable code lands in the companion GitHub repo2. If you have\n\na similar machine, you can follow along and reproduce every result.\n\nThe plan runs in eight phases, with a parallel CUDA track starting around week 5:\n\n| Phase | Weeks | Focus |\n|---|---|---|\n| 1 | 1–4 | Comfortable running local models |\n| 2 | 5–8 | Enough ML to understand inference |\n| 3 | 9–13 | Transformers and terminology |\n| 4 | 14–16 | Quantization and model formats |\n| 5 | 17–20 | Inference engineering |\n| 6 | 21–24 | Production model services |\n| 7 | 25–28 | RAG and application integration |\n| 8 | 29–32 | Fine-tuning |\n\n**The goal of this first post** is narrow on purpose: stand up and *understand*\n\nthe environment. By the end you'll be able to inventory a DGX Spark, read what\n\n`nvidia-smi`\n\ntells you, explain how its unified memory differs from a normal\n\nGPU, untangle the NVIDIA driver/CUDA-runtime/toolkit layers, and prove the GPU\n\nis usable from PyTorch with a measured CPU-vs-GPU speedup. No model yet — that's\n\nweek 2. This is the foundation everything else builds on.\n\nAll the commands in this post are packaged as a runnable script in the companion\n\nrepo under `week-01-environment/`\n\n3 — clone it, set up an SSH alias `spark`\n\nthat reaches your DGX Spark, and run `./inventory.sh`\n\nto reproduce everything\n\nhere. The repo holds the commands and scripts; this post holds the explanations,\n\nso neither repeats the other.\n\nHere's the whole inventory script, so you can see exactly what it runs without\n\ncloning anything. It just wraps each command in an SSH call to the Spark and\n\nprints a labelled section:\n\n``` bash\n#!/usr/bin/env bash\n# Week 1 — DGX Spark machine inventory. Prereq: an SSH alias `spark`.\nset -euo pipefail\nSPARK_HOST=\"${SPARK_HOST:-spark}\"\n\nrun() { echo \"=== {% katex inline %}1 ===\"; ssh \"{% endkatex %}SPARK_HOST\" \"$2\" 2>&1 || true; echo; }\n\nrun \"uname\"      'uname -a'\nrun \"os-release\" 'cat /etc/os-release'\nrun \"lscpu\"      'lscpu'\nrun \"memory\"     'free -h'\nrun \"storage\"    'lsblk'\nrun \"gpu\"        'nvidia-smi'\nrun \"cuda-nvcc\"  'nvcc --version || /usr/local/cuda/bin/nvcc --version'\nrun \"cuda-dirs\"  'ls -d /usr/local/cuda*'\nrun \"python\"     'python3 --version'\nrun \"docker\"     'docker version'\n```\n\nThe rest of this post walks through the interesting parts of that output one\n\nsection at a time.\n\nThe setup is two machines. A MacBook Pro is the *control* machine — for writing,\n\nediting, and opening SSH sessions. An NVIDIA DGX Spark is the *workhorse* — where\n\nevery model and every GPU command actually runs. I reach it over SSH as `spark`\n\n.\n\nA theme for this whole series is that I verify facts with a command instead of\n\nassuming them — even the obvious ones. So rather than start by *stating* what my\n\nmachines are, I'll show them. First the control machine:\n\n```\n# On the control MacBook\nuname -m && sysctl -n machdep.cpu.brand_string\nx86_64\nIntel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz\n```\n\nAn **Intel x86_64** Mac. This matters more than it looks, because it's a\n\ndifferent architecture from the Spark — so I want it on the record, not assumed.\n\nNow the Spark:\n\n```\nssh spark 'uname -a'\nLinux spark-66b9 6.17.0-1026-nvidia ... aarch64 aarch64 aarch64 GNU/Linux\n```\n\nThe Spark is **aarch64** — ARM64, the same CPU family as phones and Apple\n\nSilicon, but a different architecture from my Intel Mac.\n\n| Machine | Role | Architecture | Verified by |\n|---|---|---|---|\n| MacBook Pro | control / authoring | x86_64 (Intel i5) | `uname -m` |\n| DGX Spark | model + GPU work | aarch64 (ARM64) | `uname -a` |\n\nThis is not trivia. Because the two machines are **different architectures**, a\n\nPython wheel or Docker image built for my Intel Mac will not necessarily run on\n\nthe ARM64 Spark. That is exactly why all the real work in this series happens\n\nover `ssh spark`\n\n, on the box itself — and why \"it works on my laptop\" means\n\nnothing here.\n\nThe Spark's CPU is a 20-core ARM chip:\n\n```\nssh spark 'lscpu'\nArchitecture: aarch64\nCPU(s):       20\nModel name:   Cortex-X925   (10 performance cores)\nModel name:   Cortex-A725   (10 efficiency cores)\n```\n\nTwenty cores sounds like a lot, but for running a model the CPU is mostly a\n\ntraffic director: it runs the OS, my Python, and the data loading. The heavy\n\nlifting happens on the GPU. Here's the intuition that finally made \"why a GPU\"\n\nclick for me.\n\nA neural-network layer boils down to one operation repeated endlessly: multiply\n\na big grid of numbers (the model's **weights**,\nW\n) by a list of numbers (the\n\ninput,\nx\n) and add them up — a **matrix multiplication**. One output value is:\n\nA model with billions of parameters does *billions* of these multiply-adds for a\n\nsingle token. The magic property is that they're **independent**: computing\n\ny1\ndoesn't need\ny2\n. In terms I already know, that's *embarrassingly\nparallel* — like the map phase of a MapReduce where no shard waits on another.\n\nThat's the whole reason a GPU wins:\n\n| Aspect | CPU (the 20 ARM cores) | GPU (the NVIDIA GB10) |\n|---|---|---|\n| Parallel workers | a few strong cores | thousands of small cores |\n| Good at | branching logic, one-at-a-time | the same math on huge data at once |\n| Analogy | a few expert chefs | a stadium of line cooks |\n\nA CPU is a handful of very smart workers doing tasks in sequence. A GPU is\n\nthousands of simpler workers all doing the *identical* multiply-add on different\n\nnumbers simultaneously. Since a model is nothing but that identical operation\n\nrepeated, the GPU is the right tool.\n\nOne caveat I'm carrying forward: those cores are useless if you can't *feed*\n\nthem numbers fast enough, so **memory bandwidth** — not raw compute — usually\n\nlimits how fast a model runs. I'll test the CPU-vs-GPU speed difference directly\n\nwith a matmul benchmark once PyTorch is installed; my prediction is a 10x–50x\n\nGPU speedup, with a slow first GPU run due to one-time warmup.\n\n`nvidia-smi`\n\n: my new `top`\n\nfor the GPU\n`nvidia-smi`\n\nis the command I'll run every day from now on. It's the GPU\n\nequivalent of `top`\n\nor `docker stats`\n\n. Here's the real output, lightly trimmed:\n\n```\nssh spark 'nvidia-smi'\nNVIDIA-SMI 580.159.03   Driver Version: 580.159.03   CUDA Version: 13.0\nGPU 0: NVIDIA GB10   Persistence-M: On\nTemp  Perf  Pwr:Usage/Cap   Memory-Usage    GPU-Util  Compute M.\n35C   P8    4W / N/A        Not Supported   0%        Default\n```\n\nField by field, and why each one will matter later:\n\n| Field | Value | Meaning |\n|---|---|---|\n| Driver Version | 580.159.03 | kernel driver talking to the GPU |\n| CUDA Version | 13.0 |\nmax CUDA the driver supports |\n| GPU name | NVIDIA GB10 | the device (Grace-Blackwell) |\n| Temp | 35C | die temperature (heat → throttling) |\n| Perf | P8 | clock state, P0 = max … P8 = idle |\n| Pwr:Usage/Cap | 4W / N/A | current vs max power draw |\n| Memory-Usage | Not Supported | would show VRAM used/total — see below |\n| GPU-Util | 0% | % of last sample the GPU was busy |\n\nThe bottom of the output also has a process table listing every PID holding the\n\nGPU — right now just Xorg, GNOME, and Firefox using it for the desktop. That's\n\nmy first stop whenever I hit \"out of memory\": find the offender, like `lsof`\n\non\n\na stuck port.\n\nFor Week 1 I'm using `nvidia-smi`\n\npurely as an **inventory** tool — what GPU,\n\nwhat driver, what max CUDA, who's using it. The deeper use (streaming monitors,\n\nreading utilization and memory bandwidth to decide if a workload is\n\ncompute-bound or memory-bound) is a profiling skill I'm deliberately saving for\n\nthe CUDA track around Week 5, so I don't tangle the two learning tracks.\n\nThe one field that stopped me was `Memory-Usage: Not Supported`\n\n. On a normal PC\n\nwith a discrete GPU, that column is how you answer \"did my model fit? how much\n\nVRAM is left?\" On the Spark it's blank — and that's not a bug, it's the whole\n\npoint of the machine.\n\nA traditional GPU has its own separate memory (VRAM), physically distinct from\n\nsystem RAM. The DGX Spark's GB10 is a Grace-Blackwell superchip that fuses the\n\nARM CPU and the GPU onto one package and gives them **one shared memory pool**:\n\n```\nssh spark 'free -h'\ntotal   used   free   available\nMem:           121Gi   6.2Gi  67Gi   115Gi\n```\n\n| Traditional discrete GPU | DGX Spark (GB10) |\n|---|---|\n| System RAM (e.g. 64 GB) + separate VRAM (e.g. 24 GB) | one shared 121 GiB pool |\n| Copy data RAM → VRAM over PCIe | CPU and GPU read the same memory |\n| \"Will it fit?\" limited by VRAM (24 GB) | limited by total RAM (121 GB) |\n\nThis is **unified memory**. `nvidia-smi`\n\nreports \"Not Supported\" for GPU memory\n\nbecause there is no separate VRAM to report — the GPU's memory *is* the system's\n\n121 GiB. In infra terms, a discrete GPU pays a \"copy tax\" moving weights across\n\nthe PCIe bus, like shuffling data between two services with separate caches;\n\nunified memory removes that hop, like two services sharing one in-memory cache.\n\nThe practical consequence for me: on the Spark, the ceiling on model size isn't\n\na stingy 24 GB of VRAM — it's 121 GB. But I have to track model memory\n\ndifferently, via `free -h`\n\nor PyTorch's own counters, not the `nvidia-smi`\n\nmemory column. The trade-off (which I'll measure later) is that shared memory\n\nusually has lower peak bandwidth than a top-end discrete card's dedicated VRAM,\n\nso the Spark trades some raw speed for the ability to fit much larger models.\n\nThe most confusing part of the NVIDIA stack for a newcomer is that \"CUDA\" isn't\n\none thing — it's three separate layers, installed and versioned independently.\n\nMapping each to infrastructure I already understand finally made it stick:\n\n| Layer | What it is | Infra analogy | Who needs it |\n|---|---|---|---|\n| NVIDIA driver | kernel module that talks to the GPU | a device driver | everyone using the GPU |\n| CUDA runtime (libcudart) | shared libs an app calls to run GPU work | the `.so` libs you link |\nanyone running GPU programs |\n| CUDA toolkit | the `nvcc` compiler, headers, profilers |\ngcc + headers + build tools | only people compiling CUDA |\n\nThe insight that unblocked me: **you can run GPU code without the toolkit.**\n\nPyTorch ships its own copy of the CUDA runtime inside its wheel. So to run\n\nmodels I need the *driver* (system-level) plus a *CUDA-enabled PyTorch* (which\n\nbrings its own runtime). I do **not** need `nvcc`\n\n— that's only for compiling\n\ncustom CUDA kernels, a much-later CUDA-track activity.\n\nWith that lens, two clues from the inventory make sense. First, the header line\n\n`CUDA Version: 13.0`\n\nis the **maximum** CUDA the driver supports — a ceiling, not\n\nwhat's installed. That's the number that matters this week: when I install\n\nPyTorch, I must pick a CUDA build ≤ 13.0 so the driver can run it.\n\nSecond, the alarming-looking one:\n\n```\nssh spark 'nvcc --version'\nbash: nvcc: command not found\n```\n\nThis looks broken but isn't. It only means the toolkit's compiler isn't on my\n\n`PATH`\n\n. The driver and runtime clearly work — `nvidia-smi`\n\ntalks to the GPU. And\n\nthe toolkit is physically installed:\n\n```\nssh spark 'ls -d /usr/local/cuda*'\n/usr/local/cuda  /usr/local/cuda-13  /usr/local/cuda-13.0\n```\n\nSo `nvcc`\n\nexists; it's just at `/usr/local/cuda/bin/nvcc`\n\n, not on `PATH`\n\n. Rather\n\nthan assert that, I confirmed it by calling the full path:\n\n```\nssh spark '/usr/local/cuda/bin/nvcc --version'\nCuda compilation tools, release 13.0, V13.0.88\n```\n\nThe toolkit is version **13.0.88**, matching the driver's CUDA 13.0 ceiling — a\n\nhealthy, consistent stack. Nothing is broken; `command not found`\n\nwas a `PATH`\n\nissue, not a missing install. If I ever want `nvcc`\n\non `PATH`\n\n, it's one line:\n\n`export PATH=/usr/local/cuda/bin:$PATH`\n\n. But to *run* models, I never need it.\n\nPyTorch is the Python library I'll use to talk to the GPU. Before installing it,\n\none habit worth keeping: never install into the system Python. I use a\n\n**virtual environment** (venv) — an isolated per-project Python with its own\n\npackages, exactly like a per-service dependency sandbox so one project's\n\nlibraries can't break another's. On the Spark that's why the earlier\n\n`python3 -c \"import torch\"`\n\nfailed: the system Python genuinely has no torch, and\n\nI want to keep it that way.\n\nIf you're following along, first check whether you already have PyTorch — many\n\nsetups ship with it:\n\n``` python\nssh spark 'python3 -c \"import torch, sys; print(torch.__version__)\" \\\n  || echo \"no torch in this Python\"'\n```\n\nI didn't, so I made a clean venv and installed torch into it:\n\n```\nssh spark 'python3 -m venv ~/venvs/w1'\nssh spark '~/venvs/w1/bin/python -m pip install --upgrade pip'\nssh spark '~/venvs/w1/bin/python -m pip install torch numpy'\n```\n\nThe install is packaged as `setup.sh`\n\nin the companion repo, which just runs the\n\nthree steps above against `requirements.txt`\n\n:\n\n``` bash\n#!/usr/bin/env bash\nset -euo pipefail\nVENV=\"{% katex inline %}{VENV:-{% endkatex %}HOME/venvs/w1}\"\nHERE=\"{% katex inline %}(cd \"{% endkatex %}(dirname \"$0\")\" && pwd)\"\npython3 -m venv \"$VENV\"\n\"$VENV/bin/python\" -m pip install --upgrade pip\n\"{% katex inline %}VENV/bin/python\" -m pip install -r \"{% endkatex %}HERE/requirements.txt\"\n```\n\nThe interesting part is what pip pulled in. Remember the stack tops out at CUDA\n\n13.0 — and without me specifying any special index, PyPI served an **ARM64 +\nCUDA 13** build automatically:\n\n```\nSuccessfully installed torch-2.13.0 nvidia-cuda-runtime-13.0.96\n  nvidia-cudnn-cu13-9.20.0.48 nvidia-cublas-13.1.1.3 ... (aarch64 wheels)\n```\n\nThis is the payoff of the \"PyTorch ships its own CUDA runtime\" point from\n\nearlier: I did **not** install the CUDA toolkit or touch `nvcc`\n\n. Torch brought\n\nits own CUDA 13 runtime libraries (`libcudart`\n\n, `cuDNN`\n\n, `cuBLAS`\n\n) as ordinary\n\nPython wheels, matched to the ARM64 architecture and the driver's CUDA 13\n\nceiling. The driver was the only piece I needed pre-installed.\n\nNow the moment this whole week builds to: does Python see the GPU? The\n\nvalidation script (`validate_gpu.py`\n\nin the repo) is deliberately tiny:\n\n``` python\nimport torch\n\nprint(\"PyTorch:\", torch.__version__)\nprint(\"CUDA available:\", torch.cuda.is_available())\nprint(\"CUDA version (torch):\", torch.version.cuda)\n\nif torch.cuda.is_available():\n    print(\"Device:\", torch.cuda.get_device_name(0))\n    print(\"Capability:\", torch.cuda.get_device_capability(0))\n    x = torch.rand(3, 3, device=\"cuda\")   # put a real tensor on the GPU\n    print(\"Tensor on:\", x.device)\n```\n\nRun it with the venv's Python:\n\n```\nssh spark '~/venvs/w1/bin/python validate_gpu.py'\nPyTorch: 2.13.0+cu130\nCUDA available: True\nCUDA version (torch): 13.0\nDevice: NVIDIA GB10\nCapability: (12, 1)\nTensor on: cuda:0\n```\n\nEvery line matters. `CUDA available: True`\n\nmeans PyTorch found a usable GPU\n\nthrough the driver. `2.13.0+cu130`\n\nconfirms it's a CUDA 13 build. `Device: NVIDIA`\n\nis our chip.\n\nGB10`Capability: (12, 1)`\n\nis the GPU's *compute capability* —\n\nNVIDIA's versioning for GPU features; `12.x`\n\nis the Blackwell generation. And\n\n`Tensor on: cuda:0`\n\nis the real proof: I created a tensor and it physically lives\n\nin GPU memory, not on the CPU. This is the line that says \"I can run models on\n\nthis machine now.\"\n\nEarlier I predicted the GPU would beat the CPU by 10x-50x on a large matrix\n\nmultiply. Time to measure instead of hand-wave. The benchmark\n\n(`benchmark.py`\n\n) multiplies two 4096x4096 matrices 20 times on each device and\n\naverages. Two details make the GPU timing honest: a **warmup** (the first GPU\n\ncall pays a one-time kernel-load cost, so I run a few throwaway iterations\n\nfirst), and `torch.cuda.synchronize()`\n\nbefore stopping the clock (CUDA launches\n\nkernels asynchronously, so without a sync I'd be timing *queueing*, not\n\n*computing*):\n\n``` python\ndef bench(a, b, sync=False):\n    if sync: torch.cuda.synchronize()\n    t0 = time.perf_counter()\n    for _ in range(ITERS):\n        c = a @ b\n    if sync: torch.cuda.synchronize()\n    return (time.perf_counter() - t0) / ITERS\n\ncpu_s = bench(a, b)                       # on CPU\nag, bg = a.cuda(), b.cuda()\nfor _ in range(3): _ = ag @ bg            # warmup\ngpu_s = bench(ag, bg, sync=True)          # on GPU\nssh spark '~/venvs/w1/bin/python benchmark.py'\nMatrix: 4096x4096 float32, iters: 20\nCPU  ms/matmul: 173.8   GFLOP/s: 790.8\nGPU  ms/matmul: 7.53    GFLOP/s: 18263.5\nSpeedup (CPU/GPU): 23.1 x\n```\n\nA single 4096x4096 float32 matmul is about 137 billion floating-point\n\noperations. The 20-core ARM CPU chews through it in ~174 ms (~0.79 TFLOP/s); the\n\nGB10 does it in ~7.5 ms (~18.3 TFLOP/s) — a **23x speedup**, squarely inside my\n\npredicted 10x-50x range. That number *is* the reason this hardware exists: the\n\nsame math, done thousands-at-a-time instead of a-few-at-a-time. (These are the\n\nraw values in `results.json`\n\nin the repo; yours will differ by machine.)\n\n`nvcc: command not found`\n\nis not an error state — the CUDA that matters for\nrunning models lives inside PyTorch, not in the toolkit. Installing torch pulled\nits own CUDA 13 runtime as plain wheels; I never touched the toolkit.Week 1 is done: I know the machine, I understand the NVIDIA stack, and I've proven\n\nthe GPU is usable from Python with a real speedup. Week 2 leaves inventory behind\n\nand runs an actual language model with Ollama — a model runtime with a local\n\nHTTP API — where I'll start measuring the things that matter for serving: tokens\n\nper second and time to first token.\n\n32-week roadmap —\n\n[https://github.com/dramasamy/from-api-to-gpu/tree/main/roadmap](https://github.com/dramasamy/from-api-to-gpu/tree/main/roadmap) ↩\n\nCompanion code repository —\n\n[https://github.com/dramasamy/from-api-to-gpu](https://github.com/dramasamy/from-api-to-gpu) ↩\n\nWeek 1 lab —\n\n[https://github.com/dramasamy/from-api-to-gpu/tree/main/week-01-environment](https://github.com/dramasamy/from-api-to-gpu/tree/main/week-01-environment) ↩", "url": "https://wpnews.pro/news/from-api-to-gpu-week-1-understanding-nvidia-dgx-spark-environment", "canonical_source": "https://dev.to/dramasamy/from-api-to-gpu-week-1-understanding-nvidia-dgx-spark-environment-1aol", "published_at": "2026-07-11 20:57:28+00:00", "updated_at": "2026-07-11 21:15:56.880775+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["NVIDIA", "DGX Spark", "PyTorch", "CUDA", "Docker", "Kubernetes", "Linux", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/from-api-to-gpu-week-1-understanding-nvidia-dgx-spark-environment", "markdown": "https://wpnews.pro/news/from-api-to-gpu-week-1-understanding-nvidia-dgx-spark-environment.md", "text": "https://wpnews.pro/news/from-api-to-gpu-week-1-understanding-nvidia-dgx-spark-environment.txt", "jsonld": "https://wpnews.pro/news/from-api-to-gpu-week-1-understanding-nvidia-dgx-spark-environment.jsonld"}}