02:38
2026-08-10
dev.to
large-language-models
Grep Without Word Boundaries: 70 Tokens Across 7.68M Words, and os Is Real 0.1% of the Time
A developer's scan of a 7.68-million-word knowledge base found that the token 'os' appears as a real word only 0.1% of the time, highlighting the challenges of tokenization without word boundaries. Th…