{"slug": "rag-pipeline-crashes-on-scientific-characters-bge-small-en-v1-5", "title": "RAG pipeline crashes on scientific characters: BGE-small-en-v1.5", "summary": "A RAG pipeline using LlamaIndex and the BGE-small-en-v1.5 embedding model crashes with a 'TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]' error when processing scientific PDFs containing Greek letters, mathematical operators, or non-standard Unicode characters. The error occurs during semantic chunking with SemanticSplitterNodeParser, as the sentence-transformers model receives invalid input from a single problematic chunk, killing the entire batch. Developers are exploring Unicode normalization and custom cleaning wrappers as programmatic fixes.", "body_md": "# RAG pipeline crashes on scientific characters: BGE-small-en-v1.5\n\n`TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]`\n\nerror is a nightmare when you're dealing with large-scale PDF ingestion. I've been hitting this while trying to build a [RAG](/en/tags/rag/)pipeline with LlamaIndex, and it seems that regardless of the parser—whether it's LlamaParse (markdown) or PyMuPDF—the embedding model just chokes when it hits specific scientific symbols or non-standard Unicode characters in academic papers.\n\nThe weird part is that it doesn't fail immediately. If I test `docs[0]`\n\nindividually, it works fine. But the second I run the full batch through the `SemanticSplitterNodeParser`\n\n, the whole process collapses. Since the data is preserved correctly in the `.pkl`\n\nfile, the issue isn't the reading phase; it's definitely happening during the embedding call.\n\nHere is the setup that's causing the crash:\n\n``` python\nfrom llama_index.core import SimpleDirectoryReader\nfrom llama_parse import LlamaParse\nimport nest_asyncio\nimport os\nnest_asyncio.apply()\nfrom dotenv import load_dotenv\n\nload_dotenv()\nfirst_api = os.getenv(\"LLAMA_CLOUD_API_KEY\")\n\nparser = LlamaParse(\n    api_key=first_api,\n    result_type=\"markdown\", \n    user_prompt=\"Make sure the structure of all info is preserved\",\n    extract_charts=True,\n    auto_mode_trigger_on_table_in_page=True, \n    auto_mode_trigger_on_image_in_page=True,\n)\n\ndocuments = SimpleDirectoryReader(\n    input_dir=\"./docs\", \n    file_extractor={\"pdf\": parser}\n).load_data()\n\nimport pickle\nwith open(\"mds.pkl\", 'wb') as f: \n    pickle.dump(documents, f)\n```\n\nThen the crash happens here during the semantic chunking phase:\n\n``` python\nfrom llama_index.core.node_parser import SemanticSplitterNodeParser\nimport pickle\nfrom llama_index.embeddings.huggingface import HuggingFaceEmbedding\nimport nest_asyncio\nnest_asyncio.apply()\nfrom dotenv import load_dotenv\nimport os\nload_dotenv()\n\nembed_model = HuggingFaceEmbedding(\n    model_name=\"BAAI/bge-small-en-v1.5\"\n)\n\nwith open(\"mds.pkl\", \"rb\") as f:\n    docs = pickle.load(f)\n\nsplitter = SemanticSplitterNodeParser(buffer_size=1, embed_model=embed_model, include_metadata=True)\nnodes = splitter.get_nodes_from_documents(docs)\n```\n\nThe logs are pretty vague, but they point to a type mismatch in the sentence-transformers preprocessing:\n\n```\nEmbedding attempt failed: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]\n---------------------------------------------------------------------------\nTypeError Traceback (most recent call last)\nFile /opt/anaconda3/lib/python3.12/site-packages/sentence_transformers/base/model.py:557, in BaseModel.preprocess(self, inputs, prompt, **kwargs)\n--> 557 preprocessed = self[0].preprocess(inputs, prompt=prompt, **kwargs)\n```\n\n## Diagnosis and the \"Scientific Character\" Problem\n\nAfter digging into the traceback, it looks like the `bge-small-en-v1.5`\n\nmodel (via `sentence-transformers`\n\n) is receiving an input that it doesn't recognize as a valid string sequence. In scientific papers, you get a lot of Greek letters, mathematical operators, or weird ligatures that might be parsed as `None`\n\nor a non-string object if the parser glitches on a specific character, or simply a character that the tokenizer cannot handle.\n\nBecause `SemanticSplitterNodeParser`\n\niterates through the documents to find breakpoints based on embedding distances, one single \"bad\" chunk of text triggers this `TypeError`\n\nand kills the entire loop.\n\n## Potential Fixes and Workarounds\n\nI can't manually audit thousands of pages, so I need a programmatic way to handle this. I've been looking into a few options for this AI workflow:\n\n1. **Unicode Normalization**: Trying to force all text into NFKC normalization to see if that resolves the character mismatch before it hits the embed model.\n\n2. **Custom Cleaning Wrapper**: Implementing a regex-based cleaner to strip out non-printable characters or replacing complex LaTeX-style symbols with placeholders.\n\n3. **Switching the Embed Model**: Testing if a more robust model (like a larger BGE or a Cohere model) handles these tokens better, though that increases latency.\n\nIf anyone has a specific regex or a `text_cleaning`\n\nfunction that handles scientific PDF noise without losing the semantic meaning of the equations, please share. It feels like a gap in the beginner-friendly implementation of LlamaIndex's semantic splitter when dealing with real-world academic data.\n\n[Next RAG Evaluation: Why \"Vibe Checks\" Fail →](/en/threads/2969/)", "url": "https://wpnews.pro/news/rag-pipeline-crashes-on-scientific-characters-bge-small-en-v1-5", "canonical_source": "https://promptcube3.com/en/threads/2981/", "published_at": "2026-07-25 00:48:02+00:00", "updated_at": "2026-07-25 01:05:14.400430+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["LlamaIndex", "BGE-small-en-v1.5", "BAAI", "LlamaParse", "PyMuPDF", "SemanticSplitterNodeParser", "HuggingFaceEmbedding", "sentence-transformers"], "alternates": {"html": "https://wpnews.pro/news/rag-pipeline-crashes-on-scientific-characters-bge-small-en-v1-5", "markdown": "https://wpnews.pro/news/rag-pipeline-crashes-on-scientific-characters-bge-small-en-v1-5.md", "text": "https://wpnews.pro/news/rag-pipeline-crashes-on-scientific-characters-bge-small-en-v1-5.txt", "jsonld": "https://wpnews.pro/news/rag-pipeline-crashes-on-scientific-characters-bge-small-en-v1-5.jsonld"}}