{"slug": "explored-a-new-programming-language-in-pure-python-lexer-pratt-parser-bytecode", "title": "Explored a new Programming Language in Pure Python — Lexer, Pratt Parser, Bytecode Compiler, Stack VM", "summary": "A developer built lumen-lang, a small programming language implemented entirely in pure Python with no dependencies or parser generators. The project includes a hand-written lexer with line and column tracking, a Pratt parser using recursive descent with precedence-climbing methods, a bytecode compiler, and a stack-based virtual machine supporting closures via upvalue capture. The author says the disassembler is one of the most useful parts of the project for tracing VM behavior back to the compiler stage.", "body_md": "I Built a Programming Language in Pure Python — Lexer, Pratt Parser, Bytecode Compiler, Stack VM\n\nA few months ago I decided to tickle at making a programming language, have AI's help... implementation and English translation.\n\nThe result is **lumen-lang** — a small sorta complete programming language in Pure Python, no dependencies, no parser generators, no shortcuts. It has its own syntax, compiles to bytecode, and runs on a stack-based virtual machine I wrote from scratch.\n\nHere's what's inside and why I made the decisions I did.\n\n```\nSource text → Lexer → Token stream → Pratt Parser → AST → Compiler → Bytecode → VM → Output\n```\n\nEvery stage is hand-written. That was deliberate — the goal was to understand what's actually happening at each step, not to produce something in the least amount of code.\n\nThe scanner is hand-written with line and column tracking. Nothing exotic — you walk the source character by character, emit tokens, handle string escapes and number literals, and report clean errors with position info.\n\nThe part people skip: good error messages at the lexer level save enormous debugging pain later. A bare `unexpected character` with no position is useless. `[line 12, col 7] unexpected character: '@'` is not.\n\nThe parser uses **recursive descent with explicit precedence-climbing methods**. Each method handles one precedence level and calls the next:\n\n```\n_equality → _comparison → _term → _factor → _unary → _primary\n```\n\nThis is sometimes called a Pratt parser (or top-down operator precedence parsing). The key advantage: adding a new operator or precedence level means adding one method and wiring it into the chain. It doesn't require touching a grammar table or regenerating anything.\n\nIt's also readable. You can look at `_term` and immediately see that it handles `+` and `-`. No indirection.\n\nTyped node hierarchy covering all language constructs. Every node is a dataclass — no dict-based AST, no stringly-typed anything. The compiler walks the AST with a visitor pattern and emits instructions for each node type.\n\nThe compiler tree-walks the AST and emits a compact instruction set into a `Chunk` — a flat array of opcodes and operands. Constants live in a constant pool indexed by operand.\n\nYou can inspect the compiled output for any program:\n\n```\nlumen --disassemble examples/demo.lm\n```\n\nOutput for a counter closure:\n\n```\n== make_counter ==\n0000 GET_LOCAL          1\n0001 CLOSURE            0  <fn next>\n0002 GET_LOCAL          3\n0003 RETURN\n\n== next ==\n0000 GET_UPVALUE        0\n0001 CONSTANT           0\n0002 ADD\n0003 SET_UPVALUE        0\n0004 POP\n0005 GET_UPVALUE        0\n0006 RETURN\n```\n\nThe disassembler is one of the most useful parts of this project. When something in the VM behaves wrong, you can look at exactly what the compiler emitted and trace the problem to the right stage.\n\nThe VM is stack-based. Each function call pushes a call frame with its own stack window and instruction pointer.\n\nThe interesting part is **upvalue capture** — how closures capture variables from enclosing scopes.\n\nWhen the compiler sees a function reference a variable from an outer scope, it emits `GET_UPVALUE` / `SET_UPVALUE` instructions instead of `GET_LOCAL`. The VM maintains an upvalue list per closure object. While the enclosing function is still on the stack, upvalues point directly into the stack. When the enclosing function returns, open upvalues are \"closed\" — their value is copied out of the stack into the upvalue object itself.\n\nThis is the mechanism that makes this work correctly:\n\n``` js\nfn make_counter(start) {\n    let value = start;\n    fn next() {\n        value = value + 1;\n        return value;\n    }\n    return next;\n}\n\nlet counter = make_counter(10);\nprint(counter());   // 11\nprint(counter());   // 12\n```\n\n`next` captures `value` from `make_counter`'s scope. After `make_counter` returns, `value` no longer exists on the stack — but `counter` still has a valid reference to it through the closed upvalue.\n\nGetting this right took the most time of any part of the project.\n\n```\n// Fibonacci\nfn fib(n) {\n    if (n < 2) { return n; }\n    return fib(n - 1) + fib(n - 2);\n}\nprint(fib(10));   // 55\n\n// Lists and dictionaries\nlet primes = [2, 3, 5, 7, 11];\nprint(primes[2]);   // 5\n\nlet person = {\"name\": \"Ada\", \"field\": \"computing\"};\nprint(person[\"name\"]);   // Ada\n```\n\nDynamically typed, expression-oriented, C-ish syntax. Closures, recursion, lists, dicts, builtins (`print`, `len`, `type`, `clock`).\n\n```\ngit clone https://github.com/BleedingCodes/lumen-lang.git\ncd lumen-lang\npython -m lumen                          # REPL\npython -m lumen examples/demo.lm        # Run a file\npython -m lumen --disassemble examples/demo.lm   # Inspect bytecode\n```\n\nNo dependencies. Python 3.11+. MIT license.\n\nRepo: [github.com/BleedingCodes/lumen-lang](https://github.com/BleedingCodes/lumen-lang)\n\n*Built by MainbyteLabs — Python tooling for electronics labs, hardware shops, and Linux-based tech teams.*\n\n[github.com/MR-MainbyteLabs](https://github.com/MR-MainbyteLabs)", "url": "https://wpnews.pro/news/explored-a-new-programming-language-in-pure-python-lexer-pratt-parser-bytecode", "canonical_source": "https://dev.to/bleedingcodes/i-built-a-programming-language-in-pure-python-lexer-pratt-parser-bytecode-compiler-stack-vm-26fi", "published_at": "2026-09-17 13:29:17+00:00", "updated_at": "2026-09-17 13:53:24.831600+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["lumen-lang", "Python"], "alternates": {"html": "https://wpnews.pro/news/explored-a-new-programming-language-in-pure-python-lexer-pratt-parser-bytecode", "markdown": "https://wpnews.pro/news/explored-a-new-programming-language-in-pure-python-lexer-pratt-parser-bytecode.md", "text": "https://wpnews.pro/news/explored-a-new-programming-language-in-pure-python-lexer-pratt-parser-bytecode.txt", "jsonld": "https://wpnews.pro/news/explored-a-new-programming-language-in-pure-python-lexer-pratt-parser-bytecode.jsonld"}}