cd /news/large-language-models/grep-without-word-boundaries-70-toke… · home topics large-language-models article
[ARTICLE · art-89772] src=dev.to ↗ pub= topic=large-language-models verified=true sentiment=· neutral

Grep Without Word Boundaries: 70 Tokens Across 7.68M Words, and os Is Real 0.1% of the Time

A developer's scan of a 7.68-million-word knowledge base found that the token 'os' appears as a real word only 0.1% of the time, highlighting the challenges of tokenization without word boundaries. The analysis also cites OpenAI's GPT-4 contamination report and a proposed method to detect citation laundering in retrieval systems.

read1 min views1 publishedAug 10, 2026
Grep Without Word Boundaries: 70 Tokens Across 7.68M Words, and os Is Real 0.1% of the Time
Image: Dev.to

Sources #

· OpenAI, "GPT-4 Technical Report," arXiv:2303.08774, Appendix C ("Contamination on professional and academic exams") — contamination methodology and its stated limitations, quoted verbatim from the report.

· Liu, Z., et al., "HERALD: Counterfactual Audits and Minimal Repairs for Proof-of-Retrieval Rewards," arXiv:2608.06012 (submitted August 6, 2026) — abstract quoted verbatim; the citation-laundering finding and repair.

· Corpus statistics and token measurements are from our own knowledge-base scan of August 9, 2026 (2,011 files; 7,683,956 words), reproducible with the measurement script described above; carrier examples are quoted from that corpus.

── more in #large-language-models 4 stories · sorted by recency
── more on @openai 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/grep-without-word-bo…] indexed:0 read:1min 2026-08-10 ·