{"slug": "show-hn-doom-compiled-into-an-llm", "title": "Show HN: Doom Compiled into an LLM", "summary": "A developer known as physicsrob has compiled the classic video game DOOM into a transformer-based large language model, creating a stock Hugging Face Phi3ForCausalLM that renders the game autoregressively one token at a time. The project, built on the torchwright compiler, encodes all rendering logic inside the transformer, with the host only feeding tokens and blitting pixels, and the production artifact loads through ordinary auto classes without custom code. The work demonstrates a novel approach to compiling game logic into neural network weights.", "body_md": "DOOM rendering and game graph compilation, built on\n[torchwright](https://github.com/physicsrob/torchwright).\n\n**Read the full article: Doom, compiled into a transformer**\n\nThis package builds a **computation graph** that the `torchwright`\n\ncompiler\nturns into a **transformer**, which then renders DOOM **autoregressively** —\none discrete input token in, one output token out, per step, the same loop\nthat drives any chat model. Output tokens carry pixel information; the host\ncopies each output token to the next input and blits pixels to the screen\n(in production the blit happens post-hoc, by the shipped decode tools).\n\nThe production artifact is a stock Hugging Face `Phi3ForCausalLM`\n\n, compiled\ndirectly from the Doom graph into sharded fp32 safetensors. Both model and the\ndata-only WordLevel tokenizer (a plain `tokenizer.json`\n\nwhose token words are\nhuman-readable — no tokenizer code) load through ordinary auto classes with\nno custom model/tokenizer code and no `trust_remote_code=True`\n\n. ONNX is a\ndiagnostic backend only (`diagnostics/`\n\n), never a render or publication input.\nCanonical numbers (resolution, token counts, timings, accuracy): `FACTS.md`\n\n.\n\nThe **dumb-host principle** governs everything: all rendering logic — wall\nselection, visibility, distance, texture lookup, compositing — lives inside\nthe transformer. During generation the host only feeds tokens and writes\npixels; it does no geometry, no sorting, and no arithmetic on the values the\nmodel computes. Pure bitblitting.\n\nTwo boundary rules make that claim precise:\n\n**Input side (before generation):** the host builds the prompt — it crops the level file to a fixed world-space rectangle declared once in the scene config (view-independent; see*subset*in`GLOSSARY.md`\n\n) and encodes static map facts as tokens. The rule: the prompt may bake any**view-independent** static-scene fact; all**view-dependent** work — visibility, ordering, projection, occlusion — happens inside the transformer. This is input preparation, like loading a level file.**Output side (after generation):** the decode tools apply the cursor protocol the model itself emits — every cursor set, direction mark, and run width is a model output token; the host just keeps track of the cursor and blits.\n\n(See `CLAUDE.md`\n\nfor the principle as enforced during development.)\n\n**The graph lives in**— everything there compiles into the transformer; everything outside it runs on the host. Shared kernel modules (vocabulary, token↔residual codec, attention plumbing, shared math) sit flat at`torchwright_doom/model/`\n\n`model/`\n\nroot; the pipeline stages are`model/scene/`\n\n,`model/protocol/`\n\n,`model/traversal/`\n\n,`model/raster/`\n\n,`model/assets/`\n\n.`model/__init__.py`\n\nhas the per-module map.**Entry point:**`render_main.forward`\n\n(`torchwright_doom/model/render_main.py`\n\n)*constructs*the per-token forward pass — compile-time graph code, run once; the compiled transformer then executes it at every AR step. It builds the read side (decode the input token + consult static map facts), has each write-side protocol owner publish its channels, builds each dispatch branch's next-token, and selects one by the current token's type.**Reading path**(one`forward()`\n\npass, read side → write side, all under`model/`\n\n):`vocab`\n\n/`tokens`\n\n→`embedding`\n\n/`extract`\n\n→`scene/`\n\n(static read side) →`protocol/`\n\n(the dispatch table) →`render_main.forward`\n\n(assembly) → the write side:`traversal/bsp_traversal`\n\n(R_RenderBSPNode) →`raster/seg_projection`\n\n→ the`wall_*`\n\n/`visplane_*`\n\n/`flat_*`\n\nrasterizers → the pixel pass.**Prefill pipeline**(WAD → tokens the model reads before autoregression):`doom1.wad`\n\n(the freely redistributable shareware 1.9 WAD, committed at repo root) →`prompt/wad.py`\n\n(raw`MapData`\n\n) →`prompt/subset.py`\n\n(sliced to the config's fixed`region:`\n\nrectangle, renumbered, mean-centred) →`prompt/build.py`\n\n(`list[Token]`\n\n) →`tokenizer/rows.py`\n\n(row indices) → the model. Production entry:`prompt/scene.prefill_rows_for`\n\n.\n\n— the canonical numbers (resolution, token counts, timings, checkpoint size, accuracy). Other surfaces quote from it.`FACTS.md`\n\n— full module layout, the production HF runtime, and the graph-debugging tool sequence.`CLAUDE.md`\n\n— plain-English definitions of the coined vocabulary (carrier, head, marker, owner, subcontext, visplane, flat, …).`GLOSSARY.md`\n\n— the row vocabulary, raw and pretty text formats, stock tokenizer/detokenizer, carrier folding, and worked examples.`TOKENIZATION.md`\n\n— the pixel protocol: the exact per-frame token sequence (prefill + every AR phase), in the readable-surface token names.`PROTOCOL.md`\n\n— how near-first BSP traversal, occlusion, and the attention-backed return stack determine wall order.`BSP_TRAVERSAL.md`\n\n— the generated table of the token protocol (every token type, its phase, role, and dispatch wiring), for a top-down view of the AR protocol.`protocol_registry.render_protocol_table()`\n\n`make compile`\n\ncreates the complete Phi-3 bundle on Modal (publication,\n`bundle/`\n\n; \"validated\" there means manifest completeness plus shipped-tool\nsmoke checks — pixel accuracy is the separate gate below). `make run`\n\nresolves that same bundle and executes its exact bundle-root `infer.py`\n\non\nthe configured GPU (portable inference) — the only generation path in the\nproject. Everything after the subprocess is interpretation (`interpret/`\n\n).\n`configs/e1m1.yaml`\n\nis the sole full-resolution publication configuration,\nwhile `configs/e1m1_lowres.yaml`\n\nbuilds a separate 80×50 checkpoint sized for\n64 GiB of total accelerator memory—one 64-GiB-class device, or two 32-GiB\nconsumer GPUs through `device_map=\"auto\"`\n\n. The full 320×200 checkpoint still\nneeds a B200-class machine; the practical checkpoint trades resolution for a\n7,007-token frame and an 8,000-token generation cap. Its full A100-80GB render\npeaked at 43.48 GiB reserved, so it also fits two 32-GiB consumer GPUs through\nautomatic device mapping.\n\nPublished checkpoints:\n\n[320×200 flagship](https://huggingface.co/physicsrob/torchwright-doom-e1m1)— 38 layers, 85.87 GB of fp32 weight shards.[80×50 practical](https://huggingface.co/physicsrob/torchwright-doom-e1m1-80x50)— 70 layers, 34.09 GB of fp32 weight shards.\n\n**Correctness gate:** `make run COMPARE=1`\n\nscores every rendered frame\npixel-by-pixel against the vendored plain-Python reference renderer\n(`pydoom/`\n\n), reporting coverage and within-option color (see `GLOSSARY.md`\n\n)\nand writing a diff PNG. The production render's scores are in `FACTS.md`\n\n.\n\nThe load path that backs the \"stock transformer\" claim is the ordinary Transformers text-generation pipeline:\n\n``` python\nfrom pathlib import Path\nfrom transformers import pipeline\n\ngenerate = pipeline(\"text-generation\", model=bundle, device_map=\"auto\")\nprompt = Path(bundle, \"examples/e1m1_prompt.txt\").read_text()\ngenerated_text = generate(prompt, return_full_text=False)[0][\"generated_text\"]\n```\n\nThe bundle contains its executable E1M1 text prompt (`examples/e1m1_prompt.txt`\n\n)\nand `infer.py`\n\nat the bundle root. That isolated script drives the same\npipeline with progress and identity checks, and is the inference program used\nby production renders. It writes canonical integer row ids and their raw\nstandard-tokenizer text. `tools/pretty_text.py`\n\nformats that text for reading, and `tools/txt_to_png.py`\n\nindependently turns\nthe same text into a frame by executing the cursor protocol the model\nemitted — every cursor set, direction mark, and run width in the stream is a\nmodel output; the tool applies them plus palette lookup and last-write-wins\nblitting:\n\n```\npython infer.py --model . --prompt examples/e1m1_prompt.txt --output out\npython tools/pretty_text.py --input out/output.txt --output out/output.pretty.txt\npython tools/txt_to_png.py  --input out/output.txt --output out/frame.png\n```\n\n", "url": "https://wpnews.pro/news/show-hn-doom-compiled-into-an-llm", "canonical_source": "https://github.com/physicsrob/torchwright_doom/", "published_at": "2026-08-31 20:22:17+00:00", "updated_at": "2026-08-31 20:52:25.518139+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-tools"], "entities": ["physicsrob", "torchwright", "Phi3ForCausalLM", "Hugging Face", "DOOM"], "alternates": {"html": "https://wpnews.pro/news/show-hn-doom-compiled-into-an-llm", "markdown": "https://wpnews.pro/news/show-hn-doom-compiled-into-an-llm.md", "text": "https://wpnews.pro/news/show-hn-doom-compiled-into-an-llm.txt", "jsonld": "https://wpnews.pro/news/show-hn-doom-compiled-into-an-llm.jsonld"}}