{"slug": "from-api-to-gpu-week-5-tensors-the-data-structure-behind-every-model", "title": "From API to GPU, Week 5: Tensors, the Data Structure Behind Every Model", "summary": "A developer's ongoing 32-week project to understand AI inference reached Phase 2, focusing on tensors, the core data structure of machine learning models. The developer demonstrated creating tensors in PyTorch, inspecting their properties, and moving them to a GPU, while also starting a parallel CUDA track. The work runs on an NVIDIA DGX Spark system.", "body_md": "*Phase 2 of 8: Enough ML to understand inference. Week 5 of 32.*\n\nPhase 1 was about running models. Phase 2 is about understanding what happens\n\ninside them, starting with the one data structure they are all built from: the\n\n**tensor**. This week I stop talking about model files and start touching the\n\nactual numbers, in PyTorch, on the GPU.\n\nIf you write software, a tensor is a typed, multi-dimensional array that lives on\n\na specific device. By the end of this post I can create tensors, read their shape\n\nand byte size, move them to the GPU, and measure how precision changes both speed\n\nand memory.\n\nThis is also where the parallel **CUDA track** starts. I am not writing GPU\n\nkernels yet, the small programs that run on the GPU. Level 1 is just being a\n\ncompetent CUDA user: picking a device, moving data to it, and timing GPU work\n\ncorrectly.1\n\nEverything runs on the DGX Spark over `ssh spark`\n\n. I reuse the Week 1 PyTorch\n\nenvironment, `~/venvs/w1`\n\n, which already has a CUDA build of PyTorch.\n\nHere are the names used for different array dimensions:\n\n**Rank** is the number of dimensions. **Shape** is the size along each dimension.\n\n**Dtype** is the number format, the same FP32, FP16, and BF16 I measured in\n\nWeek 4. **Device** is where the tensor lives, the CPU or the GPU.\n\nInstead of describing this, I print it. The first script builds a scalar, vector,\n\nmatrix, and 3-D tensor and reports each one's rank, shape, dtype, byte size, and\n\ndevice. It then shows three operations I explain right after: an **element-wise**\n\nadd (position by position), a **broadcast** (a smaller tensor applied across a\n\nlarger one), and a **host-to-device transfer** (moving a tensor to the GPU):\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"Week 5 - tensor basics: rank, shape, dtype, bytes, and device.\n\nShows a scalar, vector, matrix, and 3D tensor, then one element-wise\noperation, one broadcast, and one CPU-to-GPU transfer. Every printed size and\nbyte count comes from PyTorch, not from a hand estimate.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport torch\n\ndef describe(name: str, tensor: torch.Tensor) -> None:\n    print(\n        f\"{name:8} rank={tensor.ndim} \"\n        f\"shape={tuple(tensor.shape)} \"\n        f\"dtype={str(tensor.dtype).replace('torch.', '')} \"\n        f\"elem={tensor.element_size()}B \"\n        f\"total={tensor.nbytes}B \"\n        f\"dev={tensor.device}\"\n    )\n\ndef main() -> None:\n    scalar = torch.tensor(3.0)\n    vector = torch.tensor([1.0, 2.0, 3.0])\n    matrix = torch.zeros(2, 3)\n    tensor3d = torch.zeros(2, 3, 4)\n\n    print(\"=== rank, shape, dtype, bytes, device ===\")\n    describe(\"scalar\", scalar)\n    describe(\"vector\", vector)\n    describe(\"matrix\", matrix)\n    describe(\"tensor3d\", tensor3d)\n\n    print(\"\\n=== same shape, three precisions ===\")\n    for dtype in (torch.float32, torch.float16, torch.bfloat16):\n        describe(str(dtype).replace(\"torch.\", \"\"), torch.zeros(1024, 1024,\n                                                                dtype=dtype))\n\n    print(\"\\n=== element-wise operation ===\")\n    a = torch.tensor([1.0, 2.0, 3.0])\n    b = torch.tensor([10.0, 20.0, 30.0])\n    print(\"a + b =\", (a + b).tolist())\n\n    print(\"\\n=== broadcasting ===\")\n    matrix = torch.ones(2, 3)\n    row = torch.tensor([1.0, 2.0, 3.0])\n    print(\"matrix shape\", tuple(matrix.shape), \"+ row shape\", tuple(row.shape))\n    print(\"result:\\n\", (matrix + row).tolist())\n\n    print(\"\\n=== device transfer ===\")\n    if torch.cuda.is_available():\n        cpu_tensor = torch.ones(3)\n        gpu_tensor = cpu_tensor.to(\"cuda\")\n        print(\"cpu_tensor.device\", cpu_tensor.device)\n        print(\"gpu_tensor.device\", gpu_tensor.device)\n    else:\n        print(\"CUDA not available; skipping GPU transfer\")\n\nif __name__ == \"__main__\":\n    main()\n```\n\nI ran it on the Spark. These commands use `public/...`\n\npaths because I run them\n\nfrom the parent repository. If you cloned the public companion repo, drop the\n\n`public/`\n\nprefix and run from that repo root.\n\n```\nssh spark '~/venvs/w1/bin/python -' \\\n    < public/week-05-pytorch-tensors/tensor_basics.py\n=== rank, shape, dtype, bytes, device ===\nscalar   rank=0 shape=() dtype=float32 elem=4B total=4B dev=cpu\nvector   rank=1 shape=(3,) dtype=float32 elem=4B total=12B dev=cpu\nmatrix   rank=2 shape=(2, 3) dtype=float32 elem=4B total=24B dev=cpu\ntensor3d rank=3 shape=(2, 3, 4) dtype=float32 elem=4B total=96B dev=cpu\n\n=== same shape, three precisions ===\nfloat32  rank=2 shape=(1024, 1024) dtype=float32 elem=4B total=4194304B dev=cpu\nfloat16  rank=2 shape=(1024, 1024) dtype=float16 elem=2B total=2097152B dev=cpu\nbfloat16 rank=2 shape=(1024, 1024) dtype=bfloat16 elem=2B total=2097152B dev=cpu\n\n=== element-wise operation ===\na + b = [11.0, 22.0, 33.0]\n\n=== broadcasting ===\nmatrix shape (2, 3) + row shape (3,)\nresult:\n [[2.0, 3.0, 4.0], [2.0, 3.0, 4.0]]\n\n=== device transfer ===\ncpu_tensor.device cpu\ngpu_tensor.device cuda:0\n```\n\nThere is a lot in that output, so here is what matters.\n\nThe shape notation is worth decoding once. `()`\n\nmeans zero dimensions, a single\n\nnumber. `(3,)`\n\nmeans one dimension holding three values, the trailing comma\n\nmarking it as a shape rather than a plain number. `(2, 3)`\n\nmeans two rows and\n\nthree columns. `(2, 3, 4)`\n\nadds a third dimension.\n\nI never set a dtype in those constructors, and every tensor still came out\n\n`float32`\n\n. That is PyTorch's default floating-point type. The precision block\n\nbelow sets the dtype on purpose.\n\nLook at the vector: shape `(3,)`\n\n, 4 bytes per element, 12 total bytes. That is\n\njust 3 times 4. The matrix is 2 times 3 times 4, which is 24 bytes. The rule is\n\nthe same one from Week 4: total bytes equals element count times bytes per\n\nelement. PyTorch reports it directly with `nbytes`\n\n, so I no longer have to\n\nestimate.\n\nThe three-precision block makes the point sharper. The same `(1024, 1024)`\n\nshape\n\nis 4,194,304 bytes at FP32 but 2,097,152 bytes at FP16 and BF16, exactly half. A\n\ndtype decides two things at once: the byte size of every number, and which values\n\nthe number can represent. This experiment measures the byte size. The next one\n\nlooks at the values.\n\nFP16 and BF16 both report 2 bytes, so they use the same memory. The difference is\n\nhow they split those 16 bits between range (how large a number can get) and step\n\nsize (how finely close numbers can be told apart). I read both directly with\n\n`torch.finfo`\n\n:\n\n``` python\nssh spark '~/venvs/w1/bin/python - <<PY\nimport torch\nfor name, dt in ((\"fp32\", torch.float32),\n                 (\"fp16\", torch.float16),\n                 (\"bf16\", torch.bfloat16)):\n    fi = torch.finfo(dt)\n    print(f\"{name:5} max={fi.max:.3e} smallest_step_near_1={fi.eps:.3e}\")\nPY'\nfp32  max=3.403e+38 smallest_step_near_1=1.192e-07\nfp16  max=6.550e+04 smallest_step_near_1=9.766e-04\nbf16  max=3.390e+38 smallest_step_near_1=7.812e-03\n```\n\nFP16 tops out at about 65,500 but has the finer step. BF16 keeps almost FP32's\n\nhuge range but has a coarser step. That is the tradeoff in one line: FP16 gives\n\nfiner detail over a small range, BF16 gives a wide range with coarser detail.\n\nWhich one is better for training and inference is a later-week topic. This week I\n\nonly need to see that same size does not mean same numbers.\n\nTwo everyday operations show up constantly, so they are worth naming.\n\nAn **element-wise operation** works position by position. Adding `[1, 2, 3]`\n\nand\n\n`[10, 20, 30]`\n\ngives `[11, 22, 33]`\n\n. Nothing mixes across positions.\n\n**Broadcasting** lets PyTorch treat compatible dimensions as repeated, without\n\nmaking a full copy. Here the row's length 3 matches the matrix's last dimension\n\nof 3, so PyTorch adds the row to every row of the matrix. The output shows\n\n`[1, 2, 3]`\n\nadded to both rows of ones, giving `[2, 3, 4]`\n\ntwice. This is how\n\nmodel code adds the same bias vector to every row of a result in one line.\n\nThe last block shows the CUDA-track idea in miniature. A tensor's **device** is\n\nwhere PyTorch will run its operations, either the **host** (the CPU side) or a\n\nCUDA GPU. `cpu_tensor.device`\n\nis `cpu`\n\n. After\n\n`.to(\"cuda\")`\n\n, `gpu_tensor.device`\n\nis `cuda:0`\n\n. That `.to(\"cuda\")`\n\nis a\n\n**host-to-device transfer**, and `cuda:0`\n\nis **device selection**, naming the\n\nfirst GPU. On the DGX Spark the CPU and GPU share one physical memory pool, so\n\nthis is not a copy across a separate VRAM. The output proves the device label\n\nchanged to `cuda:0`\n\n, which is what the GPU needs before it will compute on the\n\ntensor. It does not measure transfer cost.\n\nThis week begins CUDA-track Level 1: pick a device, put your data on it, measure\n\ncorrectly, and know what happens on out-of-memory. Level 1 continues over the\n\nnext few weeks. Week 1 already covered the driver, toolkit, and CUDA-enabled\n\nPyTorch underneath it.\n\nWeek 4 showed precision changes storage. Week 1 showed the GPU running a large\n\nmatrix multiply much faster than the CPU. Week 5 puts those together and adds the\n\nmissing dimension: precision also changes how fast the same operation runs.\n\nThe benchmark multiplies two square matrices many times per precision. Two\n\ndetails keep GPU timing honest, both from the CUDA track. First a **warmup**: the\n\nfirst GPU call pays a one-time setup cost, so I run several throwaway iterations\n\nbefore timing. Second **synchronization**: CUDA runs work asynchronously, so\n\nwithout `torch.cuda.synchronize()`\n\nI would be timing how long it takes to queue\n\nthe work, not to finish it. Here is the script:\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"Week 5 - measure matmul speed and memory for FP32, FP16, and BF16.\n\nTimes a square matrix multiply on the GPU for three precisions with warmup and\nCUDA synchronization, then compares one CPU run with one GPU run at FP32. All\ntimings use time.perf_counter around synchronized GPU work. Results print as a\ntable and optionally save to JSON.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport argparse\nimport json\nimport statistics\nimport time\nfrom pathlib import Path\n\nimport torch\n\nDTYPES = {\n    \"fp32\": torch.float32,\n    \"fp16\": torch.float16,\n    \"bf16\": torch.bfloat16,\n}\n\ndef parse_args() -> argparse.Namespace:\n    parser = argparse.ArgumentParser(description=__doc__)\n    parser.add_argument(\"--size\", type=int, default=4096)\n    parser.add_argument(\"--iters\", type=int, default=20)\n    parser.add_argument(\"--warmup\", type=int, default=5)\n    parser.add_argument(\"--output\", type=Path)\n    return parser.parse_args()\n\ndef time_matmul(a: torch.Tensor, b: torch.Tensor, iters: int,\n                warmup: int) -> float:\n    cuda = a.device.type == \"cuda\"\n    for _ in range(warmup):\n        a @ b\n    if cuda:\n        torch.cuda.synchronize()\n    samples = []\n    for _ in range(iters):\n        start = time.perf_counter()\n        a @ b\n        if cuda:\n            torch.cuda.synchronize()\n        samples.append(time.perf_counter() - start)\n    return statistics.median(samples)\n\ndef gflops(size: int, seconds: float) -> float:\n    return (2 * size ** 3) / seconds / 1e9\n\ndef main() -> None:\n    args = parse_args()\n    size = args.size\n    has_cuda = torch.cuda.is_available()\n    device = \"cuda\" if has_cuda else \"cpu\"\n\n    rows = []\n    print(f\"matmul {size}x{size}, iters={args.iters}, device={device}\\n\")\n    print(f\"{'precision':10} {'median_ms':>10} {'gflops':>10} \"\n          f\"{'peak_mib':>10}\")\n    for name, dtype in DTYPES.items():\n        if not has_cuda:\n            continue\n        torch.cuda.empty_cache()\n        torch.cuda.reset_peak_memory_stats()\n        a = torch.randn(size, size, device=device, dtype=dtype)\n        b = torch.randn(size, size, device=device, dtype=dtype)\n        seconds = time_matmul(a, b, args.iters, args.warmup)\n        peak_mib = torch.cuda.max_memory_allocated() / (1024 ** 2)\n        row = {\n            \"precision\": name,\n            \"median_ms\": round(seconds * 1000, 3),\n            \"gflops\": round(gflops(size, seconds), 1),\n            \"peak_mib\": round(peak_mib, 1),\n        }\n        rows.append(row)\n        print(f\"{name:10} {row['median_ms']:>10.3f} {row['gflops']:>10.1f} \"\n              f\"{row['peak_mib']:>10.1f}\")\n        del a, b\n\n    print(\"\\n=== CPU versus GPU at FP32 ===\")\n    a_cpu = torch.randn(size, size)\n    b_cpu = torch.randn(size, size)\n    cpu_iters = max(3, args.iters // 4)\n    cpu_s = time_matmul(a_cpu, b_cpu, cpu_iters, 1)\n    print(f\"cpu_fp32   {cpu_s * 1000:10.3f} ms   {gflops(size, cpu_s):8.1f} \"\n          f\"gflops\")\n    speedup = None\n    if has_cuda:\n        gpu_fp32 = next(r for r in rows if r[\"precision\"] == \"fp32\")\n        speedup = round(cpu_s * 1000 / gpu_fp32[\"median_ms\"], 1)\n        print(f\"gpu_fp32   {gpu_fp32['median_ms']:10.3f} ms   \"\n              f\"{gpu_fp32['gflops']:8.1f} gflops\")\n        print(f\"gpu_is     {speedup}x faster than cpu at fp32\")\n\n    if args.output:\n        payload = {\n            \"size\": size,\n            \"warmup\": args.warmup,\n            \"gpu_iters\": args.iters,\n            \"cpu_iters\": cpu_iters,\n            \"device\": device,\n            \"precisions\": rows,\n            \"cpu_fp32_ms\": round(cpu_s * 1000, 3),\n            \"gpu_speedup_fp32\": speedup,\n        }\n        args.output.write_text(json.dumps(payload, indent=2) + \"\\n\")\n\nif __name__ == \"__main__\":\n    main()\n```\n\n`gflops`\n\nreports billions of floating-point operations per second. A square\n\nmatmul of size `n`\n\nproduces `n^2`\n\noutput cells, and each cell does about `n`\n\nmultiplications and `n`\n\nadditions, so the total is about `2 * n^3`\n\noperations.\n\nDividing that by the seconds taken gives a speed I can compare across precisions.\n\nThe GPU results below are the median of 20 timed iterations after warmup; the CPU\n\nresult is the median of 5, since each CPU run is much slower. I ran it at 4096:\n\n```\nssh spark '~/venvs/w1/bin/python - --size 4096 --iters 20' \\\n    < public/week-05-pytorch-tensors/benchmark_matmul.py\nmatmul 4096x4096, iters=20, device=cuda\n\nprecision   median_ms     gflops   peak_mib\nfp32            7.440    18473.0      224.0\nfp16            1.548    88807.9      128.0\nbf16            1.576    87202.3      128.0\n\n=== CPU versus GPU at FP32 ===\ncpu_fp32      168.281 ms      816.7 gflops\ngpu_fp32        7.440 ms    18473.0 gflops\ngpu_is     22.6x faster than cpu at fp32\n```\n\nTwo clear results.\n\nFirst, the CPU-versus-GPU gap. The CPU does the FP32 matmul in about 168 ms. The\n\nGB10 GPU does it in about 7.4 ms, roughly 22.6 times faster. That matches the\n\n22.5x I measured in Week 1, a good sign the setup is consistent.\n\nSecond, the new result: precision changes speed. FP16 and BF16 finished the same\n\nmatmul in about 1.55 ms versus 7.44 ms for FP32, roughly 4.8 times faster, on the\n\nexact same shape and hardware, just by using 16-bit numbers.\n\nMemory dropped too. The two input matrices plus the temporary output matrix hold\n\n192 MiB of payload at FP32 and 96 MiB at FP16 or BF16, exactly half. PyTorch's\n\nmeasured peak, which is the most memory it had allocated at once (1 MiB is\n\n1024 x 1024 bytes), fell from 224 MiB to 128 MiB. That is about 43 percent, not\n\nexactly half, because the peak includes roughly 32 MiB of extra working\n\nallocation on top of the three matrices.\n\nFP16 and BF16 landed nearly equal in this run, about 1.55 versus 1.58 ms. They\n\nare the same size and run at nearly the same speed here, and they can swap order\n\nbetween runs. The choice between them is about the range-and-step tradeoff shown\n\nearlier, not about performance on this test.\n\nHere is Week 5 in one place:\n\n| Item | Verified value |\n|---|---|\n| FP32 bytes per element | 4 |\n| FP16 / BF16 bytes per element | 2 |\n| 1024x1024 FP32 tensor | 4,194,304 bytes |\n| 1024x1024 FP16 / BF16 tensor | 2,097,152 bytes |\n| GPU FP32 matmul (4096) | 7.44 ms, ~18,473 GFLOP/s |\n| GPU FP16 matmul (4096) | 1.55 ms, ~88,808 GFLOP/s |\n| GPU BF16 matmul (4096) | 1.58 ms, ~87,202 GFLOP/s |\n| CPU FP32 matmul (4096) | 168.28 ms, ~817 GFLOP/s |\n| GPU vs CPU at FP32 | about 22.6x faster |\n| FP16 vs FP32 speed | about 4.8x faster |\n| FP32 vs FP16 measured peak | 224 MiB vs 128 MiB |\n\nThe byte sizes matching Week 4 exactly was reassuring rather than surprising: a\n\n1024x1024 FP32 tensor really is 4,194,304 bytes, no rounding.\n\nThe real surprise was that precision buys speed, not only memory. I already knew\n\n16-bit numbers use half the bytes. Seeing the same matmul run almost five times\n\nfaster in FP16 made it clear why production inference leans on lower precision. It\n\nsaves memory and time at once, at least on this one operation.\n\nMy first timing attempt forgot `torch.cuda.synchronize()`\n\n. Because CUDA runs work\n\nasynchronously, I was timing how long it took to launch the matmul, not to finish\n\nit, so the GPU looked far faster than it really is. Adding a warmup and a\n\nsynchronize per timed iteration fixed it. Week 1 shows the related effect: its\n\ncold first GPU run took much longer than its warmed-up runs. GPU work is\n\nasynchronous, so time it with a warmup and a sync or you time the wrong thing.\n\nSingle timings are also noisy, so the benchmark reports the median of 20 GPU\n\niterations and 5 CPU iterations. The absolute milliseconds shift between runs,\n\nand FP16 and BF16 can trade places, but the groupings hold: 16-bit beats FP32,\n\nand the GPU beats the CPU.\n\nThis week explains a choice I will keep seeing: serving models in FP16 or BF16\n\ninstead of FP32. Lower precision halves the tensor payload and, on this hardware,\n\nran this dense matmul almost five times faster. That is one operation, not a\n\nwhole model. A real model also reads weights from memory and runs other steps, so\n\nit will not simply be 4.8 times faster end to end. Even so, the direction is why\n\nproduction inference leans on 16-bit precision: it saves both memory and time.\n\nThe CUDA-track habits matter for the rest of the series too. Correct GPU timing,\n\nexplicit device placement, and watching peak memory are the starting tools.\n\nThey measure elapsed time and capacity, not utilization or bandwidth, so they\n\ncannot yet classify a workload as compute-bound or memory-bound. Later CUDA-track\n\nprofiling adds those measurements.\n\nWeek 6 steps up from single tensors to a tiny neural network: inputs, a linear\n\nlayer, an activation, a loss, and a training loop small enough to watch the\n\nweights change. That connects this week's tensor operations to how a model\n\nactually learns.2\n\nThe public Week 5 lab has the tensor-basics script, the matmul benchmark, the\n\ncaptured results, observations, and troubleshooting notes.3\n\nCUDA track (parallel), Level 1:\n\n[https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/cuda-track.md](https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/cuda-track.md) ↩\n\nWeek 6 roadmap:\n\n[https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-06.md](https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-06.md) ↩\n\nWeek 5 companion lab:\n\n[https://github.com/dramasamy/from-api-to-gpu/tree/main/week-05-pytorch-tensors](https://github.com/dramasamy/from-api-to-gpu/tree/main/week-05-pytorch-tensors) ↩", "url": "https://wpnews.pro/news/from-api-to-gpu-week-5-tensors-the-data-structure-behind-every-model", "canonical_source": "https://dev.to/dramasamy/from-api-to-gpu-week-5-tensors-the-data-structure-behind-every-model-18gc", "published_at": "2026-08-17 01:12:44+00:00", "updated_at": "2026-08-17 01:41:22.845868+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools", "ai-infrastructure"], "entities": ["PyTorch", "CUDA", "NVIDIA DGX Spark"], "alternates": {"html": "https://wpnews.pro/news/from-api-to-gpu-week-5-tensors-the-data-structure-behind-every-model", "markdown": "https://wpnews.pro/news/from-api-to-gpu-week-5-tensors-the-data-structure-behind-every-model.md", "text": "https://wpnews.pro/news/from-api-to-gpu-week-5-tensors-the-data-structure-behind-every-model.txt", "jsonld": "https://wpnews.pro/news/from-api-to-gpu-week-5-tensors-the-data-structure-behind-every-model.jsonld"}}