Show HN: Doom Compiled into an LLM

A developer known as physicsrob has compiled the classic video game DOOM into a transformer-based large language model, creating a stock Hugging Face Phi3ForCausalLM that renders the game autoregressively one token at a time. The project, built on the torchwright compiler, encodes all rendering logic inside the transformer, with the host only feeding tokens and blitting pixels, and the production artifact loads through ordinary auto classes without custom code. The work demonstrates a novel approach to compiling game logic into neural network weights.

DOOM rendering and game graph compilation, built on torchwright https://github.com/physicsrob/torchwright . Read the full article: Doom, compiled into a transformer This package builds a computation graph that the torchwright compiler turns into a transformer , which then renders DOOM autoregressively — one discrete input token in, one output token out, per step, the same loop that drives any chat model. Output tokens carry pixel information; the host copies each output token to the next input and blits pixels to the screen in production the blit happens post-hoc, by the shipped decode tools . The production artifact is a stock Hugging Face Phi3ForCausalLM , compiled directly from the Doom graph into sharded fp32 safetensors. Both model and the data-only WordLevel tokenizer a plain tokenizer.json whose token words are human-readable — no tokenizer code load through ordinary auto classes with no custom model/tokenizer code and no trust remote code=True . ONNX is a diagnostic backend only diagnostics/ , never a render or publication input. Canonical numbers resolution, token counts, timings, accuracy : FACTS.md . The dumb-host principle governs everything: all rendering logic — wall selection, visibility, distance, texture lookup, compositing — lives inside the transformer. During generation the host only feeds tokens and writes pixels; it does no geometry, no sorting, and no arithmetic on the values the model computes. Pure bitblitting. Two boundary rules make that claim precise: Input side before generation : the host builds the prompt — it crops the level file to a fixed world-space rectangle declared once in the scene config view-independent; see subset in GLOSSARY.md and encodes static map facts as tokens. The rule: the prompt may bake any view-independent static-scene fact; all view-dependent work — visibility, ordering, projection, occlusion — happens inside the transformer. This is input preparation, like loading a level file. Output side after generation : the decode tools apply the cursor protocol the model itself emits — every cursor set, direction mark, and run width is a model output token; the host just keeps track of the cursor and blits. See CLAUDE.md for the principle as enforced during development. The graph lives in — everything there compiles into the transformer; everything outside it runs on the host. Shared kernel modules vocabulary, token↔residual codec, attention plumbing, shared math sit flat at torchwright doom/model/ model/ root; the pipeline stages are model/scene/ , model/protocol/ , model/traversal/ , model/raster/ , model/assets/ . model/ init .py has the per-module map. Entry point: render main.forward torchwright doom/model/render main.py constructs the per-token forward pass — compile-time graph code, run once; the compiled transformer then executes it at every AR step. It builds the read side decode the input token + consult static map facts , has each write-side protocol owner publish its channels, builds each dispatch branch's next-token, and selects one by the current token's type. Reading path one forward pass, read side → write side, all under model/ : vocab / tokens → embedding / extract → scene/ static read side → protocol/ the dispatch table → render main.forward assembly → the write side: traversal/bsp traversal R RenderBSPNode → raster/seg projection → the wall / visplane / flat rasterizers → the pixel pass. Prefill pipeline WAD → tokens the model reads before autoregression : doom1.wad the freely redistributable shareware 1.9 WAD, committed at repo root → prompt/wad.py raw MapData → prompt/subset.py sliced to the config's fixed region: rectangle, renumbered, mean-centred → prompt/build.py list Token → tokenizer/rows.py row indices → the model. Production entry: prompt/scene.prefill rows for . — the canonical numbers resolution, token counts, timings, checkpoint size, accuracy . Other surfaces quote from it. FACTS.md — full module layout, the production HF runtime, and the graph-debugging tool sequence. CLAUDE.md — plain-English definitions of the coined vocabulary carrier, head, marker, owner, subcontext, visplane, flat, … . GLOSSARY.md — the row vocabulary, raw and pretty text formats, stock tokenizer/detokenizer, carrier folding, and worked examples. TOKENIZATION.md — the pixel protocol: the exact per-frame token sequence prefill + every AR phase , in the readable-surface token names. PROTOCOL.md — how near-first BSP traversal, occlusion, and the attention-backed return stack determine wall order. BSP TRAVERSAL.md — the generated table of the token protocol every token type, its phase, role, and dispatch wiring , for a top-down view of the AR protocol. protocol registry.render protocol table make compile creates the complete Phi-3 bundle on Modal publication, bundle/ ; "validated" there means manifest completeness plus shipped-tool smoke checks — pixel accuracy is the separate gate below . make run resolves that same bundle and executes its exact bundle-root infer.py on the configured GPU portable inference — the only generation path in the project. Everything after the subprocess is interpretation interpret/ . configs/e1m1.yaml is the sole full-resolution publication configuration, while configs/e1m1 lowres.yaml builds a separate 80×50 checkpoint sized for 64 GiB of total accelerator memory—one 64-GiB-class device, or two 32-GiB consumer GPUs through device map="auto" . The full 320×200 checkpoint still needs a B200-class machine; the practical checkpoint trades resolution for a 7,007-token frame and an 8,000-token generation cap. Its full A100-80GB render peaked at 43.48 GiB reserved, so it also fits two 32-GiB consumer GPUs through automatic device mapping. Published checkpoints: 320×200 flagship https://huggingface.co/physicsrob/torchwright-doom-e1m1 — 38 layers, 85.87 GB of fp32 weight shards. 80×50 practical https://huggingface.co/physicsrob/torchwright-doom-e1m1-80x50 — 70 layers, 34.09 GB of fp32 weight shards. Correctness gate: make run COMPARE=1 scores every rendered frame pixel-by-pixel against the vendored plain-Python reference renderer pydoom/ , reporting coverage and within-option color see GLOSSARY.md and writing a diff PNG. The production render's scores are in FACTS.md . The load path that backs the "stock transformer" claim is the ordinary Transformers text-generation pipeline: python from pathlib import Path from transformers import pipeline generate = pipeline "text-generation", model=bundle, device map="auto" prompt = Path bundle, "examples/e1m1 prompt.txt" .read text generated text = generate prompt, return full text=False 0 "generated text" The bundle contains its executable E1M1 text prompt examples/e1m1 prompt.txt and infer.py at the bundle root. That isolated script drives the same pipeline with progress and identity checks, and is the inference program used by production renders. It writes canonical integer row ids and their raw standard-tokenizer text. tools/pretty text.py formats that text for reading, and tools/txt to png.py independently turns the same text into a frame by executing the cursor protocol the model emitted — every cursor set, direction mark, and run width in the stream is a model output; the tool applies them plus palette lookup and last-write-wins blitting: python infer.py --model . --prompt examples/e1m1 prompt.txt --output out python tools/pretty text.py --input out/output.txt --output out/output.pretty.txt python tools/txt to png.py --input out/output.txt --output out/frame.png