# RAG pipeline crashes on scientific characters: BGE-small-en-v1.5

> Source: <https://promptcube3.com/en/threads/2981/>
> Published: 2026-07-25 00:48:02+00:00

# RAG pipeline crashes on scientific characters: BGE-small-en-v1.5

`TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]`

error is a nightmare when you're dealing with large-scale PDF ingestion. I've been hitting this while trying to build a [RAG](/en/tags/rag/)pipeline with LlamaIndex, and it seems that regardless of the parser—whether it's LlamaParse (markdown) or PyMuPDF—the embedding model just chokes when it hits specific scientific symbols or non-standard Unicode characters in academic papers.

The weird part is that it doesn't fail immediately. If I test `docs[0]`

individually, it works fine. But the second I run the full batch through the `SemanticSplitterNodeParser`

, the whole process collapses. Since the data is preserved correctly in the `.pkl`

file, the issue isn't the reading phase; it's definitely happening during the embedding call.

Here is the setup that's causing the crash:

``` python
from llama_index.core import SimpleDirectoryReader
from llama_parse import LlamaParse
import nest_asyncio
import os
nest_asyncio.apply()
from dotenv import load_dotenv

load_dotenv()
first_api = os.getenv("LLAMA_CLOUD_API_KEY")

parser = LlamaParse(
    api_key=first_api,
    result_type="markdown", 
    user_prompt="Make sure the structure of all info is preserved",
    extract_charts=True,
    auto_mode_trigger_on_table_in_page=True, 
    auto_mode_trigger_on_image_in_page=True,
)

documents = SimpleDirectoryReader(
    input_dir="./docs", 
    file_extractor={"pdf": parser}
).load_data()

import pickle
with open("mds.pkl", 'wb') as f: 
    pickle.dump(documents, f)
```

Then the crash happens here during the semantic chunking phase:

``` python
from llama_index.core.node_parser import SemanticSplitterNodeParser
import pickle
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
import nest_asyncio
nest_asyncio.apply()
from dotenv import load_dotenv
import os
load_dotenv()

embed_model = HuggingFaceEmbedding(
    model_name="BAAI/bge-small-en-v1.5"
)

with open("mds.pkl", "rb") as f:
    docs = pickle.load(f)

splitter = SemanticSplitterNodeParser(buffer_size=1, embed_model=embed_model, include_metadata=True)
nodes = splitter.get_nodes_from_documents(docs)
```

The logs are pretty vague, but they point to a type mismatch in the sentence-transformers preprocessing:

```
Embedding attempt failed: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
File /opt/anaconda3/lib/python3.12/site-packages/sentence_transformers/base/model.py:557, in BaseModel.preprocess(self, inputs, prompt, **kwargs)
--> 557 preprocessed = self[0].preprocess(inputs, prompt=prompt, **kwargs)
```

## Diagnosis and the "Scientific Character" Problem

After digging into the traceback, it looks like the `bge-small-en-v1.5`

model (via `sentence-transformers`

) is receiving an input that it doesn't recognize as a valid string sequence. In scientific papers, you get a lot of Greek letters, mathematical operators, or weird ligatures that might be parsed as `None`

or a non-string object if the parser glitches on a specific character, or simply a character that the tokenizer cannot handle.

Because `SemanticSplitterNodeParser`

iterates through the documents to find breakpoints based on embedding distances, one single "bad" chunk of text triggers this `TypeError`

and kills the entire loop.

## Potential Fixes and Workarounds

I can't manually audit thousands of pages, so I need a programmatic way to handle this. I've been looking into a few options for this AI workflow:

1. **Unicode Normalization**: Trying to force all text into NFKC normalization to see if that resolves the character mismatch before it hits the embed model.

2. **Custom Cleaning Wrapper**: Implementing a regex-based cleaner to strip out non-printable characters or replacing complex LaTeX-style symbols with placeholders.

3. **Switching the Embed Model**: Testing if a more robust model (like a larger BGE or a Cohere model) handles these tokens better, though that increases latency.

If anyone has a specific regex or a `text_cleaning`

function that handles scientific PDF noise without losing the semantic meaning of the equations, please share. It feels like a gap in the beginner-friendly implementation of LlamaIndex's semantic splitter when dealing with real-world academic data.

[Next RAG Evaluation: Why "Vibe Checks" Fail →](/en/threads/2969/)