Embedding models fail possibly on scientific characters no matter how the pdf is parsed

A developer using LlamaIndex reports that embedding models such as BAAI/bge-small-en-v1.5 fail on scientific papers containing special characters, throwing a TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]], regardless of whether the PDFs are parsed with LlamaParse or PyMuPDF. The issue occurs during semantic chunking with SemanticSplitterNodeParser, while simple texts and individual documents work, suggesting the embedding process struggles with certain scientific notation.

I am trying to create a RAG pipline using Llamaindex. Anytime I try to use an embedding model on these papers fpr semantic chunking or anything, it does not follow through, which is I think is possibly due to the presence of scientific characters. I’m working with large amount of papers so it’s not possible to audit each separately. Are there any options other than removing the chunks/papers that are problematic? The documents have been parsed in Llamaparse and saved in markdown all with SimpleDirectoryReader - same issue arose in the past when I parsed with PyMuPDF and SimpleDirectoryReader, so my thoughts go towards the embedding process as the main issue. The parsed data itself has been properly preserved in the .pkl file so I don’t think any issues with reading that. It works on simple texts, or docs 0 when tested individually but the entire process fails together. python from llama index.core import SimpleDirectoryReader from llama parse import LlamaParse import nest asyncio import os nest asyncio.apply from dotenv import load dotenv load dotenv first api = os.getenv "LLAMA CLOUD API KEY" parser = LlamaParse api key = first api ,result type="markdown", user prompt="Make sure the structure of all info is preserved", extract charts=True,auto mode trigger on table in page=True, auto mode trigger on image in page=True, documents = SimpleDirectoryReader input dir="./docs", file extractor={"pdf": parser} .load data print "docs ingested" print documents 0 .text :100 import pickle with open "mds ", 'wb' as f: pickle.dump documents, f changed the file extension to mds.pkl afterwards python from llama index.core.node parser import SemanticSplitterNodeParser, SentenceSplitter import pickle from llama index.embeddings.huggingface import HuggingFaceEmbedding import nest asyncio nest asyncio.apply from dotenv import load dotenv import os load dotenv embed model = HuggingFaceEmbedding model name="BAAI/bge-small-en-v1.5" with open "/Users/rifah/Desktop/p/species/mds.pkl", "rb" as f: docs = pickle.load f splitter = SemanticSplitterNodeParser buffer size=1, embed model=embed model,include metadata=True nodes = splitter.get nodes from documents docs 2026-07-24 16:27:07.093685: I tensorflow/core/platform/cpu feature guard.cc:210 This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. /opt/anaconda3/lib/python3.12/site-packages/keras/src/export/tf2onnx lib.py:8: FutureWarning: In the future np.object will be defined as the corresponding NumPy scalar. if not hasattr np, "object" : Embedding attempt failed: TextEncodeInput must be Union TextInputSequence, Tuple InputSequence, InputSequence Embedding attempt failed: TextEncodeInput must be Union TextInputSequence, Tuple InputSequence, InputSequence Embedding attempt failed: TextEncodeInput must be Union TextInputSequence, Tuple InputSequence, InputSequence --------------------------------------------------------------------------- TypeError Traceback most recent call last File /opt/anaconda3/lib/python3.12/site-packages/sentence transformers/base/model.py:557, in BaseModel.preprocess self, inputs, prompt, kwargs 556 try: -- 557 preprocessed = self 0 .preprocess inputs, prompt=prompt, kwargs 558 except TypeError: File /opt/anaconda3/lib/python3.12/site-packages/sentence transformers/base/modules/transformer.py:988, in Transformer.preprocess self, inputs, prompt, processing kwargs, kwargs 987 with suggest extra on exception : -- 988 processor output = self. call processor 989 modality, 990 processor inputs, 991 modality kwargs, 992 common kwargs, 993 chat template kwargs=chat template kwargs, 994 996 if num images per sample is not None and "image grid thw" in processor output: File /opt/anaconda3/lib/python3.12/site-packages/sentence transformers/base/modules/transformer.py:1241, in Transformer. call processor self, modality, processor inputs, modality kwargs, common kwargs, chat template kwargs 1239 return self. call multimodal processor modality, processor inputs, modality kwargs, common kwargs - 1241 return self. call single modality processor modality, processor inputs, modality kwargs, common kwargs File /opt/anaconda3/lib/python3.12/site-packages/sentence transformers/base/modules/transformer.py:1302, in Transformer. call single modality processor self, modality, processor inputs, modality kwargs, common kwargs 1301 primary input = processor inputs.pop modality type - 1302 return self.processor primary input, processor inputs, call kwargs 1303 return self.processor processor inputs, call kwargs File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization utils base.py:3078, in PreTrainedTokenizerBase. call self, text, text pair, text target, text pair target, add special tokens, padding, truncation, max length, stride, is split into words, pad to multiple of, padding side, return tensors, return token type ids, return attention mask, return overflowing tokens, return special tokens mask, return offsets mapping, return length, verbose, kwargs 3077 self. switch to input mode - 3078 encodings = self. call one text=text, text pair=text pair, all kwargs 3079 if text target is not None: File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization utils base.py:3166, in PreTrainedTokenizerBase. call one self, text, text pair, add special tokens, padding, truncation, max length, stride, is split into words, pad to multiple of, padding side, return tensors, return token type ids, return attention mask, return overflowing tokens, return special tokens mask, return offsets mapping, return length, verbose, split special tokens, kwargs 3165 batch text or text pairs = list zip text, text pair if text pair is not None else text - 3166 return self.batch encode plus 3167 batch text or text pairs=batch text or text pairs, 3168 add special tokens=add special tokens, 3169 padding=padding, 3170 truncation=truncation, 3171 max length=max length, 3172 stride=stride, 3173 is split into words=is split into words, 3174 pad to multiple of=pad to multiple of, 3175 padding side=padding side, 3176 return tensors=return tensors, 3177 return token type ids=return token type ids, 3178 return attention mask=return attention mask, 3179 return overflowing tokens=return overflowing tokens, 3180 return special tokens mask=return special tokens mask, 3181 return offsets mapping=return offsets mapping, 3182 return length=return length, 3183 verbose=verbose, 3184 split special tokens=split special tokens, 3185 kwargs, 3186 3187 else: File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization utils base.py:3367, in PreTrainedTokenizerBase.batch encode plus self, batch text or text pairs, add special tokens, padding, truncation, max length, stride, is split into words, pad to multiple of, padding side, return tensors, return token type ids, return attention mask, return overflowing tokens, return special tokens mask, return offsets mapping, return length, verbose, split special tokens, kwargs 3358 padding strategy, truncation strategy, max length, kwargs = self. get padding truncation strategies 3359 padding=padding, 3360 truncation=truncation, ... 3364 kwargs, 3365 - 3367 return self. batch encode plus 3368 batch text or text pairs=batch text or text pairs, 3369 add special tokens=add special tokens, 3370 padding strategy=padding strategy, 3371 truncation strategy=truncation strategy, 3372 max length=max length, 3373 stride=stride, 3374 is split into words=is split into words, 3375 pad to multiple of=pad to multiple of, 3376 padding side=padding side, 3377 return tensors=return tensors, 3378 return token type ids=return token type ids, 3379 return attention mask=return attention mask, 3380 return overflowing tokens=return overflowing tokens, 3381 return special tokens mask=return special tokens mask, 3382 return offsets mapping=return offsets mapping, 3383 return length=return length, 3384 verbose=verbose, 3385 split special tokens=split special tokens, 3386 kwargs, 3387 File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization utils fast.py:553, in PreTrainedTokenizerFast. batch encode plus self, batch text or text pairs, add special tokens, padding strategy, truncation strategy, max length, stride, is split into words, pad to multiple of, padding side, return tensors, return token type ids, return attention mask, return overflowing tokens, return special tokens mask, return offsets mapping, return length, verbose, split special tokens 551 self. tokenizer.encode special tokens = split special tokens -- 553 encodings = self. tokenizer.encode batch 554 batch text or text pairs, 555 add special tokens=add special tokens, 556 is pretokenized=is split into words, 557 559 Convert encoding to dict 560 Tokens has type: tuple 561 list dict str, list list int or list dict str, 2D-Tensor , 562 list EncodingFast 563 564 with nested dimensions corresponding to batch, overflows, sequence length TypeError: TextEncodeInput must be Union TextInputSequence, Tuple InputSequence, InputSequence During handling of the above exception, another exception occurred: TypeError Traceback most recent call last Cell In 4 , line 21 15 docs = pickle.load f 18 splitter = SemanticSplitterNodeParser buffer size=1, embed model=embed model,include metadata=True --- 21 nodes = splitter.get nodes from documents docs File /opt/anaconda3/lib/python3.12/site-packages/llama index/core/node parser/interface.py:176, in NodeParser.get nodes from documents self, documents, show progress, kwargs 171 doc id to document = {doc.id : doc for doc in documents} 173 with self.callback manager.event 174 CBEventType.NODE PARSING, payload={EventPayload.DOCUMENTS: documents} 175 as event: -- 176 nodes = self. parse nodes documents, show progress=show progress, kwargs 177 nodes = self. postprocess parsed nodes nodes, doc id to document 179 event.on end {EventPayload.NODES: nodes} File /opt/anaconda3/lib/python3.12/site-packages/llama index instrumentation/dispatcher.py:413, in Dispatcher.span.<locals .wrapper func, instance, args, kwargs 410 logger.debug f"Failed to reset active span id: {e}" 412 try: -- 413 result = func args, kwargs 414 if isinstance result, asyncio.Future : 415 If the result is a Future, wrap it 416 new future = asyncio.ensure future result File /opt/anaconda3/lib/python3.12/site-packages/llama index/core/node parser/text/semantic splitter.py:137, in SemanticSplitterNodeParser. parse nodes self, nodes, show progress, kwargs 134 nodes with progress = get tqdm iterable nodes, show progress, "Parsing nodes" 136 for node in nodes with progress: -- 137 nodes = self.build semantic nodes from documents node , show progress 138 all nodes.extend nodes 140 return all nodes File /opt/anaconda3/lib/python3.12/site-packages/llama index/core/node parser/text/semantic splitter.py:173, in SemanticSplitterNodeParser.build semantic nodes from documents self, documents, show progress 169 text splits = self.sentence splitter text 171 sentences = self. build sentence groups text splits -- 173 combined sentence embeddings = self.embed model.get text embedding batch 174 s "combined sentence" for s in sentences , 175 show progress=show progress, 176 178 for i, embedding in enumerate combined sentence embeddings : 179 sentences i "combined sentence embedding" = embedding File /opt/anaconda3/lib/python3.12/site-packages/llama index instrumentation/dispatcher.py:413, in Dispatcher.span.<locals .wrapper func, instance, args, kwargs 410 logger.debug f"Failed to reset active span id: {e}" 412 try: -- 413 result = func args, kwargs 414 if isinstance result, asyncio.Future : 415 If the result is a Future, wrap it 416 new future = asyncio.ensure future result File /opt/anaconda3/lib/python3.12/site-packages/llama index/core/base/embeddings/base.py:517, in BaseEmbedding.get text embedding batch self, texts, show progress, kwargs 515 self.rate limiter.acquire 516 if not self.embeddings cache: -- 517 embeddings = self. get text embeddings cur batch 518 elif self.embeddings cache is not None: 519 embeddings = self. get text embeddings cached cur batch File /opt/anaconda3/lib/python3.12/site-packages/llama index/embeddings/huggingface/base.py:333, in HuggingFaceEmbedding. get text embeddings self, texts 322 def get text embeddings self, texts: List str - List List float : 323 """ 324 Generates Embeddings for text. 325 ... 331 332 """ -- 333 return self. embed texts, prompt name="text" File /opt/anaconda3/lib/python3.12/site-packages/llama index/embeddings/huggingface/base.py:268, in HuggingFaceEmbedding. embed self, inputs, prompt name 248 def embed 249 self, 250 inputs: List Union str, BytesIO , 251 prompt name: Optional str = None, 252 - List List float : 253 """ 254 Generates Embeddings with input validation and retry mechanism. 255 ... 266 267 """ -- 268 return self. embed with retry inputs, prompt name File /opt/anaconda3/lib/python3.12/site-packages/tenacity/ init .py:331, in BaseRetrying.wraps.<locals .wrapped f args, kw 329 copy = self.copy 330 wrapped f.statistics = copy.statistics type: ignore attr-defined -- 331 return copy f, args, kw File /opt/anaconda3/lib/python3.12/site-packages/tenacity/ init .py:470, in Retrying. call self, fn, args, kwargs 468 retry state = RetryCallState retry object=self, fn=fn, args=args, kwargs=kwargs 469 while True: -- 470 do = self.iter retry state=retry state 471 if isinstance do, DoAttempt : 472 try: File /opt/anaconda3/lib/python3.12/site-packages/tenacity/ init .py:371, in BaseRetrying.iter self, retry state 369 result = None 370 for action in self.iter state.actions: -- 371 result = action retry state 372 return result File /opt/anaconda3/lib/python3.12/site-packages/tenacity/ init .py:413, in BaseRetrying. post stop check actions.<locals .exc check rs 411 retry exc = self.retry error cls fut 412 if self.reraise: -- 413 raise retry exc.reraise 414 raise retry exc from fut.exception File /opt/anaconda3/lib/python3.12/site-packages/tenacity/ init .py:184, in RetryError.reraise self 182 def reraise self - t.NoReturn: 183 if self.last attempt.failed: -- 184 raise self.last attempt.result 185 raise self File /opt/anaconda3/lib/python3.12/concurrent/futures/ base.py:449, in Future.result self, timeout 447 raise CancelledError 448 elif self. state == FINISHED: -- 449 return self. get result 451 self. condition.wait timeout 453 if self. state in CANCELLED, CANCELLED AND NOTIFIED : File /opt/anaconda3/lib/python3.12/concurrent/futures/ base.py:401, in Future. get result self 399 if self. exception: 400 try: -- 401 raise self. exception 402 finally: 403 Break a reference cycle with the exception in self. exception 404 self = None File /opt/anaconda3/lib/python3.12/site-packages/tenacity/ init .py:473, in Retrying. call self, fn, args, kwargs 471 if isinstance do, DoAttempt : 472 try: -- 473 result = fn args, kwargs 474 except BaseException: noqa: B902 475 retry state.set exception sys.exc info type: ignore arg-type File /opt/anaconda3/lib/python3.12/site-packages/llama index/embeddings/huggingface/base.py:236, in HuggingFaceEmbedding. embed with retry self, inputs, prompt name 234 self. model.stop multi process pool pool=pool 235 else: -- 236 emb = self. model.encode 237 inputs, 238 batch size=self.embed batch size, 239 prompt name=prompt name, 240 normalize embeddings=self.normalize, 241 show progress bar=self.show progress bar, 242 243 return emb.tolist 244 except Exception as e: File /opt/anaconda3/lib/python3.12/site-packages/torch/utils/ contextlib.py:115, in context decorator.<locals .decorate context args, kwargs 112 @functools.wraps func 113 def decorate context args, kwargs : 114 with ctx factory : -- 115 return func args, kwargs File /opt/anaconda3/lib/python3.12/site-packages/sentence transformers/util/decorators.py:41, in deprecated kwargs.<locals .decorator.<locals .wrapper args, kwargs 39 else: 40 kwargs.pop old name --- 41 return func args, kwargs File /opt/anaconda3/lib/python3.12/site-packages/sentence transformers/sentence transformer/model.py:654, in SentenceTransformer.encode self, inputs, prompt name, prompt, batch size, show progress bar, output value, precision, convert to numpy, convert to tensor, device, normalize embeddings, truncate dim, pool, chunk size, kwargs 652 for start index in trange 0, len inputs sorted , batch size, desc="Batches", disable=not show progress bar : 653 inputs batch = inputs sorted start index : start index + batch size -- 654 features = self.preprocess inputs batch, prompt=prompt, kwargs 656 if is hpu: 657 features = self. pad features for hpu features File /opt/anaconda3/lib/python3.12/site-packages/sentence transformers/base/model.py:561, in BaseModel.preprocess self, inputs, prompt, kwargs 559 if prompt and modality == "text": 560 inputs = prompt + inp 0 , + inp 1: if isinstance inp, tuple else prompt + inp for inp in inputs -- 561 preprocessed = self 0 .preprocess inputs, kwargs 562 except AttributeError: 563 if prompt and modality == "text": File /opt/anaconda3/lib/python3.12/site-packages/sentence transformers/base/modules/transformer.py:988, in Transformer.preprocess self, inputs, prompt, processing kwargs, kwargs 985 num images per sample, num videos per sample = count media per sample processor inputs "message" 987 with suggest extra on exception : -- 988 processor output = self. call processor 989 modality, 990 processor inputs, 991 modality kwargs, 992 common kwargs, 993 chat template kwargs=chat template kwargs, 994 996 if num images per sample is not None and "image grid thw" in processor output: 997 processor output "num images per sample" = torch.tensor num images per sample, dtype=torch.long File /opt/anaconda3/lib/python3.12/site-packages/sentence transformers/base/modules/transformer.py:1241, in Transformer. call processor self, modality, processor inputs, modality kwargs, common kwargs, chat template kwargs 1238 if isinstance self.processor, ProcessorMixin : 1239 return self. call multimodal processor modality, processor inputs, modality kwargs, common kwargs - 1241 return self. call single modality processor modality, processor inputs, modality kwargs, common kwargs File /opt/anaconda3/lib/python3.12/site-packages/sentence transformers/base/modules/transformer.py:1302, in Transformer. call single modality processor self, modality, processor inputs, modality kwargs, common kwargs 1300 if modality type in processor inputs: 1301 primary input = processor inputs.pop modality type - 1302 return self.processor primary input, processor inputs, call kwargs 1303 return self.processor processor inputs, call kwargs 1305 raise RuntimeError 1306 f"Could not determine how to call processor of type {type self.processor . name } " 1307 f"for modality '{format modality modality }'" 1308 File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization utils base.py:3078, in PreTrainedTokenizerBase. call self, text, text pair, text target, text pair target, add special tokens, padding, truncation, max length, stride, is split into words, pad to multiple of, padding side, return tensors, return token type ids, return attention mask, return overflowing tokens, return special tokens mask, return offsets mapping, return length, verbose, kwargs 3076 if not self. in target context manager: 3077 self. switch to input mode - 3078 encodings = self. call one text=text, text pair=text pair, all kwargs 3079 if text target is not None: 3080 self. switch to target mode File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization utils base.py:3166, in PreTrainedTokenizerBase. call one self, text, text pair, add special tokens, padding, truncation, max length, stride, is split into words, pad to multiple of, padding side, return tensors, return token type ids, return attention mask, return overflowing tokens, return special tokens mask, return offsets mapping, return length, verbose, split special tokens, kwargs 3161 raise ValueError 3162 f"batch length of text : {len text } does not match batch length of text pair :" 3163 f" {len text pair }." 3164 3165 batch text or text pairs = list zip text, text pair if text pair is not None else text - 3166 return self.batch encode plus 3167 batch text or text pairs=batch text or text pairs, 3168 add special tokens=add special tokens, 3169 padding=padding, 3170 truncation=truncation, 3171 max length=max length, 3172 stride=stride, 3173 is split into words=is split into words, 3174 pad to multiple of=pad to multiple of, 3175 padding side=padding side, 3176 return tensors=return tensors, 3177 return token type ids=return token type ids, 3178 return attention mask=return attention mask, 3179 return overflowing tokens=return overflowing tokens, 3180 return special tokens mask=return special tokens mask, 3181 return offsets mapping=return offsets mapping, 3182 return length=return length, 3183 verbose=verbose, 3184 split special tokens=split special tokens, 3185 kwargs, 3186 3187 else: 3188 return self.encode plus 3189 text=text, 3190 text pair=text pair, ... 3208 kwargs, 3209 File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization utils base.py:3367, in PreTrainedTokenizerBase.batch encode plus self, batch text or text pairs, add special tokens, padding, truncation, max length, stride, is split into words, pad to multiple of, padding side, return tensors, return token type ids, return attention mask, return overflowing tokens, return special tokens mask, return offsets mapping, return length, verbose, split special tokens, kwargs 3357 Backward compatibility for 'truncation strategy', 'pad to max length' 3358 padding strategy, truncation strategy, max length, kwargs = self. get padding truncation strategies 3359 padding=padding, 3360 truncation=truncation, ... 3364 kwargs, 3365 - 3367 return self. batch encode plus 3368 batch text or text pairs=batch text or text pairs, 3369 add special tokens=add special tokens, 3370 padding strategy=padding strategy, 3371 truncation strategy=truncation strategy, 3372 max length=max length, 3373 stride=stride, 3374 is split into words=is split into words, 3375 pad to multiple of=pad to multiple of, 3376 padding side=padding side, 3377 return tensors=return tensors, 3378 return token type ids=return token type ids, 3379 return attention mask=return attention mask, 3380 return overflowing tokens=return overflowing tokens, 3381 return special tokens mask=return special tokens mask, 3382 return offsets mapping=return offsets mapping, 3383 return length=return length, 3384 verbose=verbose, 3385 split special tokens=split special tokens, 3386 kwargs, 3387 File /opt/anaconda3/lib/python3.12/site-packages/transformers/tokenization utils fast.py:553, in PreTrainedTokenizerFast. batch encode plus self, batch text or text pairs, add special tokens, padding strategy, truncation strategy, max length, stride, is split into words, pad to multiple of, padding side, return tensors, return token type ids, return attention mask, return overflowing tokens, return special tokens mask, return offsets mapping, return length, verbose, split special tokens 550 if self. tokenizer.encode special tokens = split special tokens: 551 self. tokenizer.encode special tokens = split special tokens -- 553 encodings = self. tokenizer.encode batch 554 batch text or text pairs, 555 add special tokens=add special tokens, 556 is pretokenized=is split into words, 557 559 Convert encoding to dict 560 Tokens has type: tuple 561 list dict str, list list int or list dict str, 2D-Tensor , 562 list EncodingFast 563 564 with nested dimensions corresponding to batch, overflows, sequence length 565 tokens and encodings = 566 self. convert encoding 567 encoding=encoding, ... 576 for encoding in encodings 577 TypeError: TextEncodeInput must be Union TextInputSequence, Tuple InputSequence, InputSequence Are there any fixes or workarounds?