19:30
2026-10-01
dev.to
natural-language-processing
Invisible soft hyphens wrecked RAG search on converted books — 4,000 U+00AD characters hiding in clean text
A developer building a RAG pipeline over converted EPUB books found that invisible U+00AD soft hyphens inherited from print typography were breaking full-text search and embeddings, with 4,213 such ch…