{"slug": "the-whole-of-pytorch-on-one-page", "title": "The whole of PyTorch on one page", "summary": "PyTorch 2.11.0's Python-facing layer is a 49 KB stub that loads 235 MB of compiled C++ code, with libtorch_cpu.dylib at 206.5 MB and libtorch_python.dylib at 28.5 MB, according to a technical blog series by Khalil Li that maps the framework's eight layers from Python to hardware. The series, measured on an Apple M3 Max laptop, promises reproducible scripts for every number and aims to explain the full stack from tensor operations to kernels and distributed training.", "body_md": "## Loss, Death, Robots, Part 0\n\n- The Map (this part)\n- Tensor\n- Autograd\n- Daily PyTorch\n- Seeing PyTorch\n- The Machinery\n- Extending PyTorch\n- The Compiler\n- Kernels & Hardware\n- Distributed\n- Ship It\n- Working on PyTorch\n\n# The Map\n\nThe whole of PyTorch on one page.\n\n## Table of Contents\n\nYou have typed something like this a thousand times. This series exists so that, by its end, you know everything these lines do. All of it: the Python they touch, the C++ they land in, the graph they record, the kernels they choose, the memory they use, and the two clocks they run on. Each of those words gets a plain meaning on its floor below.\n\nThis is Part 0, the map. First we go down through all the layers once, fast. Then we draw the territory. Then twelve ideas that make the rest of the codebase predictable. Then how this series works, and how to read it. Nothing here gets its full story. Everything here gets a place, and every full story has a numbered part waiting for it.\n\nOne promise before we start. Every measured number in this series comes from a small script you can run yourself, linked right where the number appears. I measured these on an Apple M3 Max laptop with torch 2.11.0 [[1](#ref-1)]. Your numbers will differ. The pattern they make will not.\n\n[The fall](#the-fall)\n\nPyTorch is deep. Between your keyboard and the chip there are eight levels. I will call them floors, and this meter shows all of them. It returns through the whole series, so you always know how deep you are.\n\nThe fastest way to learn a building is to go down through it once without stopping. That is this section.\n\n[Floor one: python](#floor-one-python)\n\n`torch.randn`\n\nlooks like a Python function. Ask Python what it actually is:\n\n```\n>>> type(torch.randn)\n<class 'builtin_function_or_method'>\n```\n\nPython gives that type only to functions written in compiled code. Compiled code means: code that was translated to machine instructions before you ever installed it, so there is no Python body inside it to read, and no line for your debugger to stop on.\n\nSo where do those machine instructions live? In shared libraries. A shared library is a file of compiled code that a program loads while it runs. They sit inside the torch package on your disk, and you can look at them ([proof](https://tensor.khalilli.ai/blog/part-0-the-map/#proof-p0-the-library)):\n\n``` php\ntorch._C -> _C.cpython-312-darwin.so  (49 KB, the loader)\nlibtorch_cpu.dylib        206.5 MB   (tensors and kernels)\nlibtorch_python.dylib      28.5 MB   (the python side of the border)\n```\n\n## `p0_the_library.py`\n\nthe proof, ready to read or run\n\n```\n\"\"\"Proof: where the compiled part of pytorch actually lives.\n\ntorch._C is a thin compiled stub; the weight of the framework is in\nthe shared libraries next to it. Prints the files and their sizes.\n\"\"\"\nimport glob\nimport os\nimport torch\n\nstub = torch._C.__file__\nprint(f\"torch {torch.__version__}\")\nprint(f\"torch._C -> {os.path.basename(stub)}  \"\n      f\"({os.path.getsize(stub)/1024:.0f} KB stub)\")\nlibdir = os.path.join(os.path.dirname(stub), \"lib\")\nfor lib in [\"libtorch_cpu.dylib\", \"libtorch_python.dylib\"]:\n    p = os.path.join(libdir, lib)\n    if os.path.exists(p):\n        print(f\"{lib:24s} {os.path.getsize(p)/1024/1024:6.1f} MB\")\n```\n\n[download and run it](/ldr/ch00/proofs/p0_the_library.py)\n\nRead the sizes, and then look at them:\n\nThe part of PyTorch you can see from Python is a 49 KB file whose only job is to load the other two. The real body is 235 MB of compiled code. `import torch`\n\nbrings it into your process, and after that, calling `torch.randn`\n\nmeans jumping into that body. Today we only need to know these files exist.\n\nThis is the first honest surprise of the codebase: the Python you write all day is the smallest layer of it.\n\n[The boundary](#the-boundary)\n\nThe call leaves Python at once. Where does it land?\n\nIn a C++ function named `THPVariable_randn`\n\n, inside that 28.5 MB library from the last floor. And here is a strange fact you can keep: this function does not exist in the PyTorch repository. Clone the repo, search for the name, and you find nothing. A program writes this function during the build, together with thousands of its siblings. Idea 4 below explains why, and Part 5 shows the program that does the writing.\n\nCrossing this border costs time. To see the cost alone, time the smallest possible operation, where almost no arithmetic hides it ([proof](https://tensor.khalilli.ai/blog/part-0-the-map/#proof-p2-dispatch-cost)):\n\n```\nadd, 1 element   :    0.538 microseconds per call\nadd, 4M elements :  337.264 microseconds per call\n```\n\n## `p2_dispatch_cost.py`\n\nthe proof, ready to read or run\n\n```\n\"\"\"Proof: the fixed cost of one eager op, and why size hides it.\n\nTimes the same `a + b` at two sizes. The one-element add is nearly\npure machinery (dispatch, wrapping, allocation); the 4M-element add is\nnearly pure arithmetic. CPU, single process.\n\"\"\"\nimport time\nimport torch\n\ndef per_op_us(a, b, iters):\n    # warmup\n    for _ in range(2000):\n        a + b\n    t0 = time.perf_counter()\n    for _ in range(iters):\n        a + b\n    return (time.perf_counter() - t0) / iters * 1e6\n\ntiny = per_op_us(torch.ones(1), torch.ones(1), 200_000)\nbig_n = 4_000_000\nbig = per_op_us(torch.ones(big_n), torch.ones(big_n), 2_000)\n\nprint(f\"torch {torch.__version__}, cpu\")\nprint(f\"add, 1 element   : {tiny:8.3f} us/op\")\nprint(f\"add, 4M elements : {big:8.3f} us/op\")\nprint(f\"machinery share of the tiny op: ~all of it\")\nprint(f\"ops/sec you can issue from python: {1e6/tiny:,.0f}\")\n\n# The sweep behind the toll meter: the same add at twelve sizes,\n# 1 to 4M elements in powers of four. Every dot on the widget's\n# axis is one line of this output.\nimport json\nsweep = []\nfor k in range(12):\n    n = 4 ** k\n    iters = max(1_000, min(200_000, 40_000_000 // max(n, 1)))\n    us = per_op_us(torch.ones(n), torch.ones(n), iters)\n    sweep.append({\"n\": n, \"us\": round(us, 3)})\n    print(f\"add, {n:>9,} elements : {us:9.3f} us/op\")\nprint(\"JSON_SWEEP=\" + json.dumps(sweep))\n```\n\n[download and run it](/ldr/ch00/proofs/p2_dispatch_cost.py)\n\nThe one-element add does almost no math. So its 0.54 microseconds is almost pure crossing cost: leave Python, check the arguments, build the result object, return. Half a microsecond sounds like nothing. It means Python can issue at most about 1.9 million operations per second, and a single training step contains thousands of operations. Keep this number. It returns in Idea 6.\n\n[The dispatcher](#the-dispatcher)\n\nUnder the border, the call reaches the strangest machine in PyTorch: the dispatcher. The dispatcher is the router that decides, for every operation, which pieces of code run and in what order.\n\nLook at what it must decide. Your three lines never said \"record gradients\". No `if`\n\nstatement in your code turns that on. Yet somewhere, something decided that this matrix multiplication should be remembered for `backward()`\n\n. That something is the dispatcher. Every operation passes down through a fixed stack of layers. Each layer can act on the call, change it, or let it pass unchanged. Autograd, the part of PyTorch that computes gradients, is one such layer. Mixed precision is another. On this run, only autograd is awake.\n\n[The kernel](#the-kernel)\n\nAt the bottom of the stack, one concrete function is chosen. Chosen is the right word. This torch build has 3,677 registered operation names ( \n\n```\n\"\"\"Micro-proofs quoted in Part 0: storage sharing, view errors,\nmutation rewriting history, the no_grad layer, float32 absorption.\"\"\"\nimport torch\n\nprint(f\"torch {torch.__version__}\\n\")\n\n# 1. a tensor is a window over storage\nx = torch.arange(6, dtype=torch.float32)\nv = x.view(2, 3)\nprint(\"same bytes under both:\", x.data_ptr() == v.data_ptr())\nprint(\"v.stride():\", v.stride(), \" v.t().stride():\", v.t().stride())\ntry:\n    v.t().view(-1)\nexcept RuntimeError as e:\n    print(\"v.t().view(-1) ->\", str(e).split(\".\")[0])\n\n# 2. mutation rewrites the recorded program\na = torch.ones(3, requires_grad=True)\ny = a * 2\nprint(\"\\nbefore add_:\", type(y.grad_fn).__name__)\ny.add_(1)\nprint(\"after  add_:\", type(y.grad_fn).__name__)\n\n# 3. no_grad removes one dispatcher layer\nwith torch.no_grad():\n    z = a * 2\nprint(\"\\ninside no_grad, grad_fn:\", z.grad_fn)\n\n# 4. float32 absorbs small numbers\nt = torch.tensor(1e8)\nprint(\"\\n(1e8 + 1) - 1e8 in float32 =\", ((t + 1) - t).item())\n\n# 5. the size of the operation list (idea 3)\nprint(\"\\nregistered operation names:\",\n      len(torch._C._dispatch_get_all_op_names()))\n```\n\n[proof](https://tensor.khalilli.ai/blog/part-0-the-map/#proof-p4-micro-proofs) prints the count), and a name is not a function body. The operation `addmm`\n\n, the matrix multiplication behind `model(x)`\n\n, has separate bodies for CPU and for each kind of GPU, for each data type, for dense and for sparse tensors. A body like this, written for one device and one data type, is called a kernel. The dispatcher's last job is to pick one:##\n\n`p4_micro_proofs.py`\n\nthe proof, ready to read or run[download and run it](/ldr/ch00/proofs/p4_micro_proofs.py)\n\nThe full list of operations lives in one file in the repository: [ native_functions.yaml](https://github.com/pytorch/pytorch/blob/v2.11.0/aten/src/ATen/native/native_functions.yaml) [\n\n[2](#ref-2)]. Its sibling\n\n[[](https://github.com/pytorch/pytorch/blob/v2.11.0/tools/autograd/derivatives.yaml)\n\n`derivatives.yaml`\n\n[3](#ref-3)] lists the derivative of each operation. Everything else grows from these two files. No other file in the repository tells you as much per line.\n\nOne floor down sits memory. `torch.randn(64, 128)`\n\nneeds 32,768 bytes: 64 rows times 128 numbers times 4 bytes per number. On the CPU this is an ordinary allocation. On a GPU it is not. There, PyTorch runs its own allocator, a keeper of memory that asks the GPU driver for large blocks once and then reuses them, because asking the driver every time is slow. This allocator decides when you run out of memory and what the error means. Part 4 examines it.\n\n[The two clocks](#the-two-clocks)\n\nHere the story splits in two. What follows is the single most useful performance fact in PyTorch.\n\nOn a GPU, your Python line does not do the work. It requests the work, and the request returns at once. The GPU does the work on its own clock, while Python continues. I measured it on this machine's GPU ([proof](https://tensor.khalilli.ai/blog/part-0-the-map/#proof-p3-two-timelines)):\n\n```\ntime to request 50 matrix multiplications :   1.58 ms\ntime until the work was actually done     :  73.83 ms\npython was free during                    :  72.25 ms  (98%)\n```\n\n## `p3_two_timelines.py`\n\nthe proof, ready to read or run\n\n```\n\"\"\"Proof: the CPU runs ahead of the GPU.\n\nQueues 50 large matmuls on the MPS device and measures two times:\nhow long Python took to *ask* for the work, and how long the work\nactually took. The difference is the gap the chapter draws.\n\"\"\"\nimport time\nimport torch\n\nassert torch.backends.mps.is_available(), \"needs an Apple-silicon GPU\"\ndev = torch.device(\"mps\")\n\na = torch.randn(2048, 2048, device=dev)\nb = torch.randn(2048, 2048, device=dev)\nfor _ in range(5):          # warmup\n    (a @ b)\ntorch.mps.synchronize()\n\nt0 = time.perf_counter()\nfor _ in range(50):\n    c = a @ b\nt_queue = time.perf_counter() - t0\ntorch.mps.synchronize()\nt_done = time.perf_counter() - t0\n\nprint(f\"torch {torch.__version__}, mps\")\nprint(f\"time to queue 50 matmuls : {t_queue*1e3:8.2f} ms\")\nprint(f\"time until work finished : {t_done*1e3:8.2f} ms\")\nprint(f\"python was free for      : {(t_done-t_queue)*1e3:8.2f} ms ({(t_done-t_queue)/t_done:.0%} of the wall time)\")\n\n# The three ways to read the loss, measured, for the two-clocks\n# widget: never, once at the end, after every step.\nimport json\ndef run_mode(mode, iters=50):\n    for _ in range(5):\n        (a @ b)\n    torch.mps.synchronize()\n    t0 = time.perf_counter()\n    t_free = 0.0\n    for i in range(iters):\n        c = a @ b\n        if mode == \"every\":\n            c[0, 0].item()\n    t_q = time.perf_counter() - t0\n    if mode == \"once\":\n        c[0, 0].item()\n    torch.mps.synchronize()\n    total = time.perf_counter() - t0\n    return {\"mode\": mode, \"queue_ms\": round(t_q * 1e3, 2),\n            \"total_ms\": round(total * 1e3, 2),\n            \"free_ms\": round((total - t_q) * 1e3, 2)}\n\nmodes = [run_mode(m) for m in (\"never\", \"once\", \"every\")]\nfor m in modes:\n    print(f\"read {m['mode']:>5}: total {m['total_ms']:8.2f} ms, \"\n          f\"python busy {m['queue_ms']:8.2f} ms\")\nprint(\"JSON_MODES=\" + json.dumps(modes))\n```\n\n[download and run it](/ldr/ch00/proofs/p3_two_timelines.py)\n\nPython asked for all fifty multiplications in under two milliseconds, then waited, free, while the GPU computed for another seventy-two. On the CPU there is no such split; the math happens before your line returns. On any accelerator, the split is the normal state of the program.\n\nThis is why eager PyTorch is fast enough to use: Python runs ahead and the GPU never waits for it. It is also why simple timing code gives wrong answers, and why one `loss.item()`\n\ninside a training loop can slow the whole step. `.item()`\n\nneeds the actual number. The number sits at the end of a queue of work the GPU has not finished yet, so Python must stop and wait for the whole queue:\n\nPart 4 teaches honest measurement on top of exactly this picture.\n\n[The turn](#the-turn)\n\nLine three: `loss.backward()`\n\n. Nothing so far explains how this line can work. The forward computation is over. How does PyTorch know what to differentiate?\n\nIt knows because the forward pass had a second job. Every time an operation passed the autograd layer of the dispatcher, a small record was written: which operation ran, and which recorded steps produced its inputs. Records that point at records form a graph, and that word here always means exactly this recorded structure. By the time `loss`\n\nexists, its graph exists too ([proof](https://tensor.khalilli.ai/blog/part-0-the-map/#proof-p1-graph-chain)):\n\n```\nloss.grad_fn     = SumBackward0\nSumBackward0\n  AddmmBackward0\n    AccumulateGrad\n```\n\n## `p1_graph_chain.py`\n\nthe proof, ready to read or run\n\n```\n\"\"\"Proof: loss.backward() walks a graph that forward quietly recorded.\n\nBuilds the chapter's three-line program and prints the autograd graph\nthat exists before backward is ever called.\n\"\"\"\nimport torch\nimport torch.nn as nn\n\ntorch.manual_seed(0)\nmodel = nn.Sequential(nn.Linear(128, 256), nn.ReLU(), nn.Linear(256, 10))\n\nx = torch.randn(64, 128)\nloss = model(x).sum()\n\nprint(f\"torch {torch.__version__}\")\nprint(f\"x.grad_fn        = {x.grad_fn}\")\nprint(f\"loss.grad_fn     = {type(loss.grad_fn).__name__}\")\n\nnode, depth = loss.grad_fn, 0\nwhile node is not None and depth < 10:\n    print(\"  \" * depth + type(node).__name__)\n    nexts = [n for n, _ in node.next_functions if n is not None]\n    node = nexts[0] if nexts else None\n    depth += 1\n```\n\n[download and run it](/ldr/ch00/proofs/p1_graph_chain.py)\n\n`backward()`\n\ninvents nothing. It walks the graph from the loss back to your inputs, runs each recorded derivative, and stores the results in `.grad`\n\n. The walk ends at `AccumulateGrad`\n\n, the record that does the storing. And it starts nowhere else: `x.grad_fn`\n\nis `None`\n\n, because `x`\n\nwas created directly, not computed.\n\nOne question should bother you here. A derivative needs values. The derivative of a matrix multiplication needs the matrices that were multiplied, and the forward pass is long over. Write the derivative out and the need is visible:\n\nSo where are they? They were saved, next to the records, during the forward pass:\n\nSo the forward pass silently decides how much memory training costs. Part 2 shows the exact saving rules. Part 4 shows how to watch it happen. And a method called activation checkpointing trades that memory for extra compute; it has its own chapter in Part 2.\n\nCarry one sentence out of this section: backward can only walk what forward wrote. It sounds small. In Part 9 it becomes the rule that decides which GPUs in a cluster must talk to each other.\n\nThat was the whole fall: a name, a border, a stack of layers, a chosen kernel, a keeper of memory, two clocks, and a graph that is walked backward. Now the territory, properly.\n\n[The territory](#the-territory)\n\nPyTorch is built in layers, and each layer speaks only to its neighbors. Every box below is at least one part of this series.\n\nThe same territory, seen as folders in the repository. If you ever open the codebase, this is the map that stops you from being lost:\n\nThree facts about this map save you weeks. First: `torch/`\n\nis plain Python, and you can read every file in it today. Second: `aten/`\n\nand `c10/`\n\nare C++; the tensors, the kernels and the dispatcher live there, and `torch/csrc/`\n\nis the single bridge that connects the two languages. Third: `torchgen/`\n\nis the program from the boundary floor, the one that writes code during the build. The repository you read is the input. The library you run is the output. That is why searching the repository for `THPVariable_randn`\n\nfinds nothing:\n\nAbove the core sits the ecosystem. It looks endless, but it has a simple shape: every library attaches to PyTorch at a specific, nameable place. Know the attachment places and you know the ecosystem.\n\nRead the picture from the center out. `transformers`\n\nbuilds its models as `nn.Module`\n\nclasses, so if you understand Part 3, you can read its source. `deepspeed`\n\nreplaces the distributed engine, so its home is Part 9. `vllm`\n\nand `sglang`\n\nkeep the model weights and replace the runtime around them. And `trl`\n\nand `peft`\n\ndo not touch PyTorch directly at all; they build on `transformers`\n\n.\n\nThe whole ecosystem fits in one table. The second column names the place in the PyTorch repository where each family attaches:\n\n| attaches at | the pytorch side | who | what they keep, what they bring |\n|---|---|---|---|\n| nn.Module | `torch/nn/` | transformers, diffusers, timm | models are Modules; torch runs them |\n| the training loop | `torch/autograd/` `torch/optim/` | lightning, accelerate | torch stays the engine; they drive it |\n| the distributed engine | `torch/distributed/` | deepspeed | swaps the engine, brings ZeRO |\n| the eager runtime | `torch/nn/` `torch/library.py` | vllm, sglang, TensorRT-LLM | keep the weights, replace the runtime, each with a csrc/ of its own kernels |\n| the operation list | `aten/` `torch/library.py` | flash-attention, torchvision ops | new names on the list |\n| two floors at once | `torch/autograd/` + transformers | unsloth | trains through transformers, brings its own Triton kernels |\n| only the weights | none; the weights file | TEI, llama.cpp, MLX | left pytorch, kept the weights; llama.cpp re-encodes them to GGUF |\n\nTwo rows of the table deserve pictures. The first is the eager runtime, the thing the serving engines replace:\n\nThe second is Triton, the kernel language that appears through the whole table: PyTorch's compiler writes it, and libraries bring their own:\n\nRead the table downward and less of PyTorch survives each row. The last row keeps nothing but the weights file. That is the quiet law of the ecosystem: the weights outlive the runtime. Part 10 walks these attachment points one by one.\n\n[The twelve ideas](#the-twelve-ideas)\n\nMost of PyTorch is not thousands of separate decisions. It is a small set of ideas, applied everywhere. These twelve make the rest of the codebase predictable before you read it. Each one returns later as a full chapter or part. Each one comes with runnable evidence now; the small proofs share one script ([proof](https://tensor.khalilli.ai/blog/part-0-the-map/#proof-p4-micro-proofs)).\n\n[1. A tensor is a window over storage](#1-a-tensor-is-a-window-over-storage)\n\nA tensor does not hold numbers. It holds a description of where to look: a pointer into one flat block of memory, the sizes of each dimension, and the strides. A stride is the number of steps to move in that flat block to reach the next element of a dimension. Two tensors can look completely different and read the same bytes:\n\n```\n>>> x = torch.arange(6.); v = x.view(2, 3)\n>>> x.data_ptr() == v.data_ptr()   # same address in memory\nTrue\n>>> v.stride(), v.t().stride()     # transpose swapped the strides\n((3, 1), (1, 3))\n```\n\nThe transpose moved no data. It swapped two numbers in the description. Some descriptions are impossible to write down, and that is exactly why `v.t().view(-1)`\n\nraises an error while `reshape`\n\nsilently copies the data instead. Part 1 opens with this puzzle and solves it completely.\n\n[2. Autograd records a program you never wrote](#2-autograd-records-a-program-you-never-wrote)\n\nIn your code, `y`\n\nis one name, and you overwrite it freely. Line two destroys the value line one made. The graph cannot afford that: backward will need every step. So autograd writes one record per change, and no record is ever overwritten. Watch the record change as `y`\n\ndoes:\n\n```\n>>> a = torch.ones(3, requires_grad=True)\n>>> y = a * 2\n>>> type(y.grad_fn).__name__\n'MulBackward0'\n>>> y.add_(1)                      # change y in place\n>>> type(y.grad_fn).__name__\n'AddBackward0'\n>>> y[0] = 9                       # overwrite one slot\n>>> type(y.grad_fn).__name__\n'CopySlices'\n```\n\nThree statements, three records, one chain. `y.grad_fn`\n\nalways holds the newest record, and each record points at the one before it, so the whole history stays reachable. That history is the program you never wrote. The machinery that keeps it correct under every kind of in-place change has real depth, and it is one of the best chapters of Part 2.\n\n[3. One list of operations is the whole interface](#3-one-list-of-operations-is-the-whole-interface)\n\nThe 3,677 registered names are PyTorch's real interface. Each backend implements its share of them. The compiler rewrites programs made of them: `torch.compile`\n\nreads the list your program became and returns a shorter one, where a matrix multiplication and the add after it can fuse into one `addmm`\n\n, and a chain of small elementwise operations becomes one generated kernel. Export formats store them: `torch.export`\n\nwrites the list to disk as a graph of exactly these names, and an ONNX file is the same idea with each name translated into ONNX's vocabulary. Quantization replaces them: the float32 matmul is swapped for an int8 body, the same place on the list, different arithmetic. When you meet a new PyTorch technology, ask one question first: what does it do to the operations? The answer usually explains the whole design.\n\n[4. PyTorch writes most of its own code](#4-pytorch-writes-most-of-its-own-code)\n\n`native_functions.yaml`\n\ndeclares every operation. `derivatives.yaml`\n\ndeclares every derivative. At build time, `torchgen/`\n\nreads both and writes the Python bindings, the autograd record classes and the dispatcher tables. This is why searching the repository for a function you just called can find nothing: you searched the input of the build, and the function is in the output. People who work on PyTorch read the yaml first. After Part 5, so will you.\n\n[5. Features are layers with a switch](#5-features-are-layers-with-a-switch)\n\nPyTorch's features combine cleanly because each one is a layer in the dispatcher, watching the same stream of operations:\n\n```\n>>> with torch.no_grad():\n...     z = a * 2\n>>> z.grad_fn is None              # nothing was recorded\nTrue\n```\n\n`no_grad`\n\nedited no function. It set a flag that sends operations past the autograd layer, so nothing gets recorded. Mixed precision, tracing and vmap work the same way, and that is why they can be combined without knowing about each other. Part 5 opens the machinery under the flag.\n\n[6. Every operation pays a fixed cost first](#6-every-operation-pays-a-fixed-cost-first)\n\nHalf a microsecond of crossing and routing before any math, on every single operation. That was the measurement on the boundary floor. Applied honestly, this one number explains why `torch.compile`\n\nexists, why fused optimizers exist, and why the first question about any slow model is: is it limited by compute, or by the cost of issuing many small operations?\n\n[7. Memory, not speed, is what kills training runs](#7-memory-not-speed-is-what-kills-training-runs)\n\nA slow program still finishes. A program that runs out of GPU memory dies with `CUDA out of memory`\n\n, and that is the most common death in all of PyTorch. The forward pass saves values for backward (the turn, above). The allocator keeps and reuses blocks. Between them they decide how large a model you can train. This series treats memory as a first-class subject in Part 4.\n\n[8. Python is why it won, and what it costs](#8-python-is-why-it-won-and-what-it-costs)\n\nPyTorch won because you write it in ordinary Python, with ordinary debuggers and print statements. The price is the border cost from Idea 6, paid on every operation. The history of the framework is a sequence of attempts to keep the first while reducing the second. TorchScript tried to replace Python with its own language; it is now in maintenance mode [[4](#ref-4)]. The current compiler watches your Python run and translates what it can, and it is winning. The pattern to remember: inside PyTorch, betting against Python has always lost.\n\n[9. Forward decides what backward must do](#9-forward-decides-what-backward-must-do)\n\nBackward can only walk what forward wrote. On one machine this sounds like a detail. At scale it becomes the law of the land: in distributed training, the way a tensor is split across GPUs in the forward pass decides which GPUs must exchange data in the backward pass. One idea, from a laptop to a cluster. It is the spine of Part 9.\n\n[10. Shared bytes plus in-place writes cause the hardest problems](#10-shared-bytes-plus-in-place-writes-cause-the-hardest-problems)\n\nIdea 1 lets many tensors read the same bytes. Idea 2 lets you change those bytes in place. Combine them: one write can change the meaning of several tensors at once, and any system that records programs (autograd, the compiler, export) must notice and stay correct. When a corner of PyTorch looks strangely complicated, ask what shared bytes plus an in-place write would do to it. That is usually the answer.\n\n[11. The code keeps its history](#11-the-code-keeps-its-history)\n\nThe repository holds the remains of every era: the original C code from 2016, the Caffe2 merge of 2018, TorchScript from 2019, the compiler district growing since 2023. When a file looks strange, the explanation is usually historical: something older lived there first. Part 5 tells this history where it explains the present.\n\n[12. Floating point is a contract; read it](#12-floating-point-is-a-contract-read-it)\n\n```\n>>> t = torch.tensor(1e8)\n>>> ((t + 1) - t).item()\n0.0\n```\n\nA float32 number has about 7 decimal digits of precision, so adding 1 to one hundred million changes nothing [[5](#ref-5)]. This is not a bug; it is the number format doing what it promises. Add the faster, less precise formats used in training, plus the fact that some GPU kernels sum in different orders on different runs, and \"why did my loss change between runs\" becomes a question with exact answers. Part 4 reads this contract clause by clause.\n\n[How this series draws](#how-this-series-draws)\n\nEvery still figure you just saw is a real Excalidraw scene, and the scene files ship with the series; you can open any drawing and edit it. The four instruments you can operate follow the same language, and every number inside them comes from the proof scripts. All of them speak one visual language, so that by Part 2 you read them without thinking:\n\nOrange always marks the subject: the one thing moving. Ink is structure. Grey is context. Dashed means recorded, implied, or asleep. The depth meter marks the floor. And the robot appears at most once per part, because a mascot that is everywhere stops being funny.\n\n[How to read this](#how-to-read-this)\n\nTwelve parts. Each is one long page like this one. And the series has one quiet goal behind every part: by the end, you should know the machine well enough to build a small PyTorch yourself. Every drawing that shows a mechanism, every formula next to a figure, and every proof script is a piece of that.\n\n| Part | What is behind the door |\n|---|---|\n| 0. The Map | you are here |\n| 1. Tensor | storage, strides, views, data types, broadcasting |\n| 2. Autograd | the graph, in-place writes, checkpointing, double backward |\n| 3. Daily PyTorch | nn, optim, data loading, mixed precision, seen from inside |\n| 4. Seeing PyTorch | the profiler, memory, floating point, honest measurement |\n| 5. The Machinery | the dispatcher, aten, torchgen, the history |\n| 6. Extending PyTorch | subclasses, custom operations, new backends |\n| 7. The Compiler | dynamo, aot autograd, inductor, dynamic shapes |\n| 8. Kernels & Hardware | the gpu model, triton, cutlass, what fast means |\n| 9. Distributed | collectives, ddp, fsdp, dtensor, parallel training |\n| 10. Ship It | export, quantization, executorch, the ecosystem |\n| 11. Working on PyTorch | the contributor's field guide |\n\nYou do not have to read front to back. Three reading lines run through the parts, like lines through stations:\n\nEvery chapter inside every part follows the same seven steps, so the rhythm becomes familiar fast:\n\nAnd the method, stated plainly, because you should know what you are trusting. Every mechanism claim is checked against the source code or shown by a script before it is published. The scripts are linked in place and pinned to one torch version. When PyTorch moves and a claim goes stale, the chapter is corrected and the correction is noted on the page. A series about internals that cannot admit drift would be wrong within a year.\n\n[What you can now say](#what-you-can-now-say)\n\nTest yourself against this list. After one reading you should be able to say, in your own words:\n\n- what\n`type(torch.randn)`\n\nreturns, and where the compiled body actually lives on your disk - what the dispatcher is, and how\n`no_grad`\n\nstops autograd without editing any function - what a kernel is, and what decides which one runs\n- why the CPU and the GPU run on two clocks, and why that makes simple timing code lie\n- what the graph is, who writes it, and why backward can never do anything forward did not write down\n- and the twelve ideas, each in one sentence\n\nIf one of these is fuzzy, return to its floor; each one is only a minute long. That is what this page is for.\n\n[Try it yourself](#try-it-yourself)\n\nThe five proof scripts are the exercises. For each one: predict the output first, then run it, then explain the difference.\n\n[p0_the_library.py](https://tensor.khalilli.ai/blog/part-0-the-map/#proof-p0-the-library): how big are the compiled libraries in your own torch install?[p1_graph_chain.py](https://tensor.khalilli.ai/blog/part-0-the-map/#proof-p1-graph-chain): what graph remains after a two-layer model runs?[p2_dispatch_cost.py](https://tensor.khalilli.ai/blog/part-0-the-map/#proof-p2-dispatch-cost): what is the fixed cost per operation on your machine?[p3_two_timelines.py](https://tensor.khalilli.ai/blog/part-0-the-map/#proof-p3-two-timelines): how long is your GPU still working after Python is done asking?[p4_micro_proofs.py](https://tensor.khalilli.ai/blog/part-0-the-map/#proof-p4-micro-proofs): the twelve ideas, compressed into five small experiments.\n\n[Pick a door](#pick-a-door)\n\nPart 1 is the tensor. It opens with the puzzle from Idea 1, and now you have seen the error with your own eyes:\n\n```\n>>> v.t().view(-1)\nRuntimeError: view size is not compatible with input tensor's\nsize and stride ...\n>>> v.t().reshape(-1)   # this one works. why?\ntensor([0., 3., 1., 4., 2., 5.])\n```\n\nSame tensor. Same request. One line refuses, the other quietly copies the data. The difference between those two lines is the whole first part of this series.\n\nSee you on the next floor down.\n\n[References](#references)\n\n[1] Khalilli, *five proof scripts, measured on an Apple M3 Max, torch 2.11.0, CPU and Apple GPU*, 2026. Linked in place above; rerun them to check me.\n\n[2] PyTorch source, *native_functions.yaml*, pinned to the v2.11.0 tag. [https://github.com/pytorch/pytorch/blob/v2.11.0/aten/src/ATen/native/native_functions.yaml](https://github.com/pytorch/pytorch/blob/v2.11.0/aten/src/ATen/native/native_functions.yaml)\n\n[3] PyTorch source, *derivatives.yaml*, pinned to the v2.11.0 tag. [https://github.com/pytorch/pytorch/blob/v2.11.0/tools/autograd/derivatives.yaml](https://github.com/pytorch/pytorch/blob/v2.11.0/tools/autograd/derivatives.yaml)\n\n[4] PyTorch documentation, *TorchScript*, which states it is in maintenance mode. [https://docs.pytorch.org/docs/stable/jit.html](https://docs.pytorch.org/docs/stable/jit.html)\n\n[5] IEEE, *754 single precision*: 24 binary digits of precision, about 7 decimal digits.\n\nThree good things to read after this page: Edward Yang's [PyTorch internals talk](http://blog.ezyang.com/2019/05/pytorch-internals/), which maps the C++ side in depth; the [PyTorch Developer Podcast](https://pytorch-dev-podcast.simplecast.com/), short episodes by the same author; and the repository's own [CONTRIBUTING.md](https://github.com/pytorch/pytorch/blob/v2.11.0/CONTRIBUTING.md), which describes the folder layout in the maintainers' words.", "url": "https://wpnews.pro/news/the-whole-of-pytorch-on-one-page", "canonical_source": "https://tensor.khalilli.ai/blog/part-0-the-map/", "published_at": "2026-08-11 19:05:10+00:00", "updated_at": "2026-08-11 19:14:25.097234+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools"], "entities": ["PyTorch", "Khalil Li", "Apple M3 Max", "libtorch_cpu.dylib", "libtorch_python.dylib"], "alternates": {"html": "https://wpnews.pro/news/the-whole-of-pytorch-on-one-page", "markdown": "https://wpnews.pro/news/the-whole-of-pytorch-on-one-page.md", "text": "https://wpnews.pro/news/the-whole-of-pytorch-on-one-page.txt", "jsonld": "https://wpnews.pro/news/the-whole-of-pytorch-on-one-page.jsonld"}}