RAG pipeline crashes on scientific characters: BGE-small-en-v1.5

A RAG pipeline using LlamaIndex and the BGE-small-en-v1.5 embedding model crashes with a 'TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]' error when processing scientific PDFs containing Greek letters, mathematical operators, or non-standard Unicode characters. The error occurs during semantic chunking with SemanticSplitterNodeParser, as the sentence-transformers model receives invalid input from a single problematic chunk, killing the entire batch. Developers are exploring Unicode normalization and custom cleaning wrappers as programmatic fixes.

RAG pipeline crashes on scientific characters: BGE-small-en-v1.5 TextEncodeInput must be Union TextInputSequence, Tuple InputSequence, InputSequence error is a nightmare when you're dealing with large-scale PDF ingestion. I've been hitting this while trying to build a RAG /en/tags/rag/ pipeline with LlamaIndex, and it seems that regardless of the parser—whether it's LlamaParse markdown or PyMuPDF—the embedding model just chokes when it hits specific scientific symbols or non-standard Unicode characters in academic papers. The weird part is that it doesn't fail immediately. If I test docs 0 individually, it works fine. But the second I run the full batch through the SemanticSplitterNodeParser , the whole process collapses. Since the data is preserved correctly in the .pkl file, the issue isn't the reading phase; it's definitely happening during the embedding call. Here is the setup that's causing the crash: python from llama index.core import SimpleDirectoryReader from llama parse import LlamaParse import nest asyncio import os nest asyncio.apply from dotenv import load dotenv load dotenv first api = os.getenv "LLAMA CLOUD API KEY" parser = LlamaParse api key=first api, result type="markdown", user prompt="Make sure the structure of all info is preserved", extract charts=True, auto mode trigger on table in page=True, auto mode trigger on image in page=True, documents = SimpleDirectoryReader input dir="./docs", file extractor={"pdf": parser} .load data import pickle with open "mds.pkl", 'wb' as f: pickle.dump documents, f Then the crash happens here during the semantic chunking phase: python from llama index.core.node parser import SemanticSplitterNodeParser import pickle from llama index.embeddings.huggingface import HuggingFaceEmbedding import nest asyncio nest asyncio.apply from dotenv import load dotenv import os load dotenv embed model = HuggingFaceEmbedding model name="BAAI/bge-small-en-v1.5" with open "mds.pkl", "rb" as f: docs = pickle.load f splitter = SemanticSplitterNodeParser buffer size=1, embed model=embed model, include metadata=True nodes = splitter.get nodes from documents docs The logs are pretty vague, but they point to a type mismatch in the sentence-transformers preprocessing: Embedding attempt failed: TextEncodeInput must be Union TextInputSequence, Tuple InputSequence, InputSequence --------------------------------------------------------------------------- TypeError Traceback most recent call last File /opt/anaconda3/lib/python3.12/site-packages/sentence transformers/base/model.py:557, in BaseModel.preprocess self, inputs, prompt, kwargs -- 557 preprocessed = self 0 .preprocess inputs, prompt=prompt, kwargs Diagnosis and the "Scientific Character" Problem After digging into the traceback, it looks like the bge-small-en-v1.5 model via sentence-transformers is receiving an input that it doesn't recognize as a valid string sequence. In scientific papers, you get a lot of Greek letters, mathematical operators, or weird ligatures that might be parsed as None or a non-string object if the parser glitches on a specific character, or simply a character that the tokenizer cannot handle. Because SemanticSplitterNodeParser iterates through the documents to find breakpoints based on embedding distances, one single "bad" chunk of text triggers this TypeError and kills the entire loop. Potential Fixes and Workarounds I can't manually audit thousands of pages, so I need a programmatic way to handle this. I've been looking into a few options for this AI workflow: 1. Unicode Normalization : Trying to force all text into NFKC normalization to see if that resolves the character mismatch before it hits the embed model. 2. Custom Cleaning Wrapper : Implementing a regex-based cleaner to strip out non-printable characters or replacing complex LaTeX-style symbols with placeholders. 3. Switching the Embed Model : Testing if a more robust model like a larger BGE or a Cohere model handles these tokens better, though that increases latency. If anyone has a specific regex or a text cleaning function that handles scientific PDF noise without losing the semantic meaning of the equations, please share. It feels like a gap in the beginner-friendly implementation of LlamaIndex's semantic splitter when dealing with real-world academic data. Next RAG Evaluation: Why "Vibe Checks" Fail → /en/threads/2969/