Transformers are black boxes … mostly. We normally train them to predict the next token and call the result an LLM. But for years, I’ve wondered what else you could put inside one by constructing its weights directly.
This led me down the road of creating torchwright, a compiler whose output format is a transformer. Feed it a computation graph — a definition of how to get from input tokens to output tokens — and it emits transformer weights. Run inference on the resulting model, and the graph executes. I started by making calculators — simple transformers that evaluate expressions like “12*34”. Once those worked as vanilla Hugging Face checkpoints, I wondered how far I could go. The natural question any nerd would ask: Can it run Doom?
The answer? Yes.
I ported Doom’s renderer into an ordinary LLM architecture. Hugging Face loads the checkpoint with no custom code. Instead of training it to imitate Doom frames, I translated Doom’s original rendering algorithm into that architecture as faithfully as I could. Given a prompt containing the level data, the player’s position, and the viewing direction, the transformer generates the frame Doom would have drawn. My compiler constructed every weight directly from the resulting computation graph. There was no training anywhere.
What exactly did I build? #
Before I started, I had to decide what would count as running Doom. I wanted an honest win: the checkpoint itself had to execute substantially the same rendering algorithm as Doom. The map and player state had to enter through the prompt, and the program outside the weights could only convert the model’s output mechanically into pixels.
The result is a stock, decoder-only checkpoint that Hugging Face loads with no
custom model code (it uses Phi3ForCausalLM
). Generation produces a stream of drawing operations and intermediate values. The host ignores most of them and acts on five drawing commands:
- setCursorX(x) and setCursorY(y) move the cursor.
- setCursorDirectionX and setCursorDirectionY choose whether it advances horizontally or vertically after a draw.
- pixel(color, w) paints a run of w pixels at the cursor using Doom’s palette.
The checkpoint is not tied to a particular frame, player position, or even E1M1. The prompt contains the map geometry, BSP tree, sector heights, texture references, light levels, player position, and viewing direction. Those can all change without recompiling. What is fixed at compile time is the 320×200 output and the texture library (nine wall textures and six floor and ceiling textures chosen for E1M1’s opening scene). The same checkpoint can render any world scene built from those textures; adding another texture requires recompiling.
This is Doom’s renderer, not the complete game. It uses Doom’s low-detail mode, rendering the 3D view in 160 two-pixel-wide columns. Sprites are not implemented, and the weapon and status bar are fixed to Doom’s pistol-start state. For the frame above, 97% of the 64,000 output pixels exactly match the reference renderer.
Below is the model rendering the opening frame of E1M1. On the left is the sequence produced by the transformer. On the right, the host mechanically applies the drawing commands as they are generated.
Outside the weights
The program outside the weights is small. It remembers a cursor, looks up one of Doom’s 256 RGB colors, and paints the horizontal runs requested by the model. The complete 43-line program is below if you want to check that boundary for yourself.
How it works #
The replay above contains 53,747 generated tokens, but only some of them draw pixels. The other tokens are used to manage the current state of the rendering algorithm. Each pass through the transformer executes one bounded step and then emits one token. That token can be the name of the next operation, a result that some later operation will need, or one of the drawing commands handled by the tiny loop above.
I run the model with Hugging Face Transformers, a widely used library for generating text with LLMs. Each time the model predicts a token, that token is appended to the sequence and the model runs again. On the next pass, the model can read the entire sequence. The prompt is the read-only scene input. The generated portion is effectively append-only working memory. It contains the execution path the renderer has taken as well as the values it has calculated along the way.
Before I explain how the generated sequence represents the renderer’s state, it helps to know a little about Doom’s renderer. In Doom, a level is fundamentally a 2D map divided into polygonal sectors, each with a floor height, ceiling height, and light level. A precomputed binary space partitioning tree — a BSP tree — divides that map into regions. The renderer walks this tree from front to back, projects walls onto the screen, and draws their visible pieces as vertical columns. Near opaque walls cover parts of the screen, so they can clip walls and entire branches that come later. Floors and ceilings are recorded during this walk, then drawn in a separate pass.
One renderer step per token
A readable expression such as bspFront(node=40, depth=0)
is one item in the
model’s vocabulary, not a string the model has to parse character by character.
When a transformer reads a token, it first turns it into a vector called an
embedding. Because my compiler chooses that embedding, designated positions in
the vector can mean “run bspFront
,” “the node is 40,” and “the tree depth is 0.”
Before walking the BSP tree, torchdoom makes a setup pass over its nodes. For each node, a setup operation retrieves its partition line — an origin and direction — from the prompt, compares the camera position against it, and emits a record saying which side contains the camera. Those records are keyed by node ID and remain in the generated history.
Now consider bspFront(node=40, depth=0)
. Its embedding supplies the current
operation, node ID, and tree depth at layer 0. One attention lookup retrieves
node 40’s two child references from the prompt. Another retrieves the saved
camera-side result keyed by node 40. Together they identify which child to visit
first — node 39 in this case. The remaining layers increment the depth and make
bspFront(node=39, depth=1)
the next token.
That is one complete renderer step: the latest token says what to do, attention retrieves its inputs by fields such as node ID, and the layers calculate what happens next.
The tree walk is recursive, so it also needs to remember where to return. Each descent appends a breadcrumb recording the parent node and whether the child was visited first or second. When a leaf finishes, attention retrieves that breadcrumb and resumes at the parent. The generated history is therefore both the instruction stream and the call stack.
State in an append-only history
Doom’s renderer, unsurprisingly, updates several data structures as it draws. This poses a challenge for torchdoom, because the sequence generated by a transformer is append-only. Once a token has been emitted, it cannot be changed. Torchdoom represents each logical update as a new record rather than rewriting an old one. Later operations retrieve the records they need with attention.
Exactly how those records are retrieved depends on the state they represent. The call stack above uses recency: attention retrieves the most recent breadcrumb matching the current child and depth. Wall coverage, on the other hand, accumulates.
Once a nearby opaque wall covers a horizontal range of the screen, nothing
farther away can appear there. Doom adds that range to a mutable list called
solidsegs
, merging ranges when they overlap. Torchdoom does not merge them. It appends each new range to the generated history, where it remains alongside all the earlier ranges. A new record augments the previous records rather than replacing them.
To clip a farther wall, the model uses attention to determine whether any recorded range covers its current screen column. If so, it retrieves the end of that range and jumps past it. If not, it finds the next covered range and draws until it reaches it. Repeating those two queries makes the separate records behave like Doom’s merged list. The same accumulated coverage lets the renderer skip a farther branch of the BSP tree when nearer walls already hide its entire projected bounding box.
Floors and ceilings also accumulate records. During the wall pass, Torchdoom records the plane identifier and the visible top and bottom rows for each screen column the plane occupies. Together those records describe the structures Doom calls visplanes. After the BSP walk finishes, a later pass retrieves the column boundaries and turns them into horizontal pixel runs. This is the progression visible in the replay: first the walls appear as vertical columns, then the floors and ceilings sweep across the frame.
The generated history therefore supports several kinds of state. Sometimes the most recent matching record wins; sometimes every matching record remains relevant. In either case, the model recovers the current state without changing earlier tokens.
Trading tokens for layers
Renderer state is only one use for the generated history. Torchdoom also uses it to divide long calculations across several decoding steps.
Within one decoding step, calculations that depend on one another have to happen in order through the transformer’s layers. The longer that chain, the deeper the compiled transformer needs to be. To keep the model reasonably shallow, torchdoom sometimes emits an intermediate result rather than carrying the calculation all the way to the next renderer operation. Once that result is fed back as input, the next part of the calculation can begin again near the first layer.
The BSP camera-side records mentioned earlier are the simplest example. Doom
normally calculates the side test when it visits each BSP node. Torchdoom
instead uses a setup pass that emits one operation and one result for each node.
Without those records, retrieving the partition line, comparing it with the
camera, choosing the nearer child, and producing the next traversal operation
would all form one long dependency chain. With them, bspFront
can retrieve the finished side test and begin from there.
Wall projection is a more representative example. To draw a wall segment, the renderer has to project its two endpoints onto the screen. The generated sequence contains intermediate results like these:
angle1(119.75) theta1(29.75)
angle2(60.25) theta2(-29.75)
angle1
is the direction from the camera to the wall’s first endpoint. theta1
is that direction relative to where the camera is facing. The second pair does the same for the wall’s other endpoint.
For each endpoint, the renderer has to retrieve its position, subtract the
camera position, calculate the resulting angle, subtract the camera direction,
wrap the result, and project it onto the screen. If all of that happened inside
the operation that consumes the projected endpoint, the transformer would need
to be substantially deeper. Torchdoom cuts the calculation into pieces:
angle1
records the world angle, theta1
records the camera-relative angle, and the following operation begins with the projected screen column already available. This frame repeats that process for 81 wall segments.
The same pattern repeats throughout. Torchdoom records a wall column’s scale before projecting its height, its clipping boundary before working out which spans are visible, and its starting texture coordinate before drawing pixels. These are short-lived handoffs between parts of a calculation.
These handoffs cost tokens, but they keep the transformer substantially shallower. A deeper transformer would make every generated token more expensive; in this case, spending additional tokens reduces the total cost of inference. The 320×200 transformer ultimately ended up 38 layers deep.
What it costs #
In 1993, John Carmack famously squeezed every ounce of computing power out of a 486 to render Doom in real time. I honor that efficiency by porting it to this incredibly inefficient substrate. While Doom achieved 35 frames per second on a 486, I achieve 0.0004 frames per second on a B200 (quite literally more than a billion times as powerful).
In more familiar units, generating a single frame takes just under 40 minutes. The frame begins with a 3,614-token prompt and takes another 53,747 generated tokens to finish. Including the time to load the 21-billion-parameter, 85.87 GB checkpoint, the complete run takes 42.1 minutes.
I find it amusing that the computational power required to run Doom as a transformer happens to be about the same order of magnitude as the compute readily available to me in 2026. If it had worked out an order of magnitude worse, there’s no way I would have been able to build this.
Run it yourself #
The full-resolution checkpoint is the model described above. If you are interested in running it yourself, I’d highly recommend using the smaller checkpoint, which renders at a resolution of 80×50. It is a separate compilation with the same textures, and supports the same prompt. I still recommend running it with 80GB of GPU memory, but theoretically you can run it with 64GB of vram.
Torchwright currently requires that everything runs in fp32.
The published example.py is the program above with the practical model ID and an 80×50 screen. After installing a CUDA-enabled PyTorch build for your machine, run:
python -m pip install "transformers>=5" accelerate pillow
hf download physicsrob/torchwright-doom-e1m1-80x50 example.py --local-dir doom-example
python doom-example/example.py
The script downloads the checkpoint, prompt, and Doom palette through the
normal Hugging Face cache and writes frame.png
. The code is small; the 34.09 GB download is not.
Both model repositories also include infer.py
, the exact prompt, and the small tools that turn the generated text into a PNG. The source repository contains the compiler-facing graph and the full reproduction path.
References #
- id Software (1997).
DOOM source code release. The renderer is primarily implemented in,
r_bsp.c
, andr_segs.c
.r_plane.c
- Hall, Tom (1992). The DOOM Bible. - Howard, Simon (2003). The Doom rendering engine. - Sanglard, Fabien (2010). Doom engine code review. - Lindner, David et al. (2023). Tracr: Compiled Transformers as a Laboratory for Interpretability.
Citation #
Robert Porter. "Doom, compiled into a transformer." Out of Distribution, August 2026. https://ood.dev/posts/doom/
@misc{porter2026doom,
author = {Porter, Robert},
title = {Doom, compiled into a transformer},
year = {2026},
month = {aug},
howpublished = {\url{https://ood.dev/posts/doom/}},
note = {Out of Distribution (blog)}
}