Philip Gage

Shrijith Venkatramana, an engineer building the AI code review tool LiveReview, traces modern LLM tokenization back to Byte Pair Encoding, a 1994 data-compression algorithm by Philip Gage. The writeup…

A tokenizer converts text into integer IDs that a language model can process, and its design—typically learned via byte-pair encoding—determines context-window usage, input cost, and how well the mode…

Byte-Pair Encoding (BPE), originally a data compression algorithm published by Philip Gage in 1994, was adapted in 2016 by Sennrich, Haddow, and Birch for subword tokenization in neural machine transl…