RAG pipeline crashes on scientific characters: BGE-small-en-v1.5

TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]

error is a nightmare when you're dealing with large-scale PDF ingestion. I've been hitting this while trying to build a RAGpipeline with LlamaIndex, and it seems that regardless of the parser—whether it's LlamaParse (markdown) or PyMuPDF—the embedding model just chokes when it hits specific scientific symbols or non-standard Unicode characters in academic papers.

The weird part is that it doesn't fail immediately. If I test docs[0]

individually, it works fine. But the second I run the full batch through the SemanticSplitterNodeParser

, the whole process collapses. Since the data is preserved correctly in the .pkl

file, the issue isn't the reading phase; it's definitely happening during the embedding call.

Here is the setup that's causing the crash:

from llama_index.core import SimpleDirectoryReader
from llama_parse import LlamaParse
import nest_asyncio
import os
nest_asyncio.apply()
from dotenv import load_dotenv

load_dotenv()
first_api = os.getenv("LLAMA_CLOUD_API_KEY")

parser = LlamaParse(
    api_key=first_api,
    result_type="markdown", 
    user_prompt="Make sure the structure of all info is preserved",
    extract_charts=True,
    auto_mode_trigger_on_table_in_page=True, 
    auto_mode_trigger_on_image_in_page=True,
)

documents = SimpleDirectoryReader(
    input_dir="./docs", 
    file_extractor={"pdf": parser}
).load_data()

import pickle
with open("mds.pkl", 'wb') as f: 
    pickle.dump(documents, f)

Then the crash happens here during the semantic chunking phase:

from llama_index.core.node_parser import SemanticSplitterNodeParser
import pickle
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
import nest_asyncio
nest_asyncio.apply()
from dotenv import load_dotenv
import os
load_dotenv()

embed_model = HuggingFaceEmbedding(
    model_name="BAAI/bge-small-en-v1.5"
)

with open("mds.pkl", "rb") as f:
    docs = pickle.load(f)

splitter = SemanticSplitterNodeParser(buffer_size=1, embed_model=embed_model, include_metadata=True)
nodes = splitter.get_nodes_from_documents(docs)

The logs are pretty vague, but they point to a type mismatch in the sentence-transformers preprocessing:

Embedding attempt failed: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
File /opt/anaconda3/lib/python3.12/site-packages/sentence_transformers/base/model.py:557, in BaseModel.preprocess(self, inputs, prompt, **kwargs)
--> 557 preprocessed = self[0].preprocess(inputs, prompt=prompt, **kwargs)

Diagnosis and the "Scientific Character" Problem #

After digging into the traceback, it looks like the bge-small-en-v1.5

model (via sentence-transformers

) is receiving an input that it doesn't recognize as a valid string sequence. In scientific papers, you get a lot of Greek letters, mathematical operators, or weird ligatures that might be parsed as None

or a non-string object if the parser glitches on a specific character, or simply a character that the tokenizer cannot handle.

Because SemanticSplitterNodeParser

iterates through the documents to find breakpoints based on embedding distances, one single "bad" chunk of text triggers this TypeError

and kills the entire loop.

Potential Fixes and Workarounds #

I can't manually audit thousands of pages, so I need a programmatic way to handle this. I've been looking into a few options for this AI workflow:

1. Unicode Normalization: Trying to force all text into NFKC normalization to see if that resolves the character mismatch before it hits the embed model.

2. Custom Cleaning Wrapper: Implementing a regex-based cleaner to strip out non-printable characters or replacing complex LaTeX-style symbols with placeholders.

3. Switching the Embed Model: Testing if a more robust model (like a larger BGE or a Cohere model) handles these tokens better, though that increases latency.

If anyone has a specific regex or a text_cleaning

function that handles scientific PDF noise without losing the semantic meaning of the equations, please share. It feels like a gap in the beginner-friendly implementation of LlamaIndex's semantic splitter when dealing with real-world academic data.

Next RAG Evaluation: Why "Vibe Checks" Fail →