A chunk can fit your token limit and still be a bad retrieval result: half a code sample, a table row without its header, or an answer separated from its heading.
For crawled documentation, try structure-aware splitting first, then apply a tokenizer limit to oversized sections.
This standard-library example keeps fenced code together while producing sections:
FENCE = chr(96) * 3
def heading_chunks(markdown: str):
path, lines, fenced = [], [], False
for line in markdown.splitlines():
if line.strip().startswith(FENCE):
fenced = not fenced
if line.startswith("#") and not fenced and lines:
body = "\n".join(lines).strip()
if body:
yield " / ".join(path), body
level = len(line) - len(line.lstrip("#"))
path = path[:level - 1] + [line[level:].strip()]
lines = [line]
else:
lines.append(line)
if lines:
body = "\n".join(lines).strip()
if body:
yield " / ".join(path), body
chunks = [
{"heading_path": path, "text": text}
for path, text in heading_chunks(markdown)
]
It is a first pass. A single large section can still exceed the embedding model’s input limit, so split that section again using paragraphs and a tokenizer.
If a child section is called “Limits,” the parent path may explain whether it means API limits, model limits, or account limits. Store that path as metadata instead of duplicating the full page into every vector.
Treat special elements deliberately:
Do not pick a strategy by looking at pretty boundaries alone. Create 10–30 queries from real documentation tasks. Include parameter lookup, a cross-section question, a table lookup, a code question, and an unanswerable query.
For each strategy, inspect the top results and record whether the answer-bearing text appears in the top k, whether heading and source metadata are present, whether code or table context was lost, whether duplicate chunks appear, and how much prompt token cost changes.
Use the same pages, embedding model, index settings, and questions for each run. The LangChain text-splitter guide explains common recursive options; the OpenAI embeddings guide describes embedding inputs and model behavior.
An evaluation table can start with five columns: question, expected source section, retrieved top-k sections, answerable from context, and notes. Review failures by category. If parameter questions fail but table lookups pass, changing global chunk size may make the working cases worse; try preserving parameter blocks or heading context first.
A limit such as 1,200 characters is not 1,200 tokens. Use your actual tokenizer to enforce the model’s input ceiling. Keep overlap small and add it only when evaluation shows that a boundary removed context. Large overlap can raise duplicate matches and storage without improving answers.
No strategy wins for every corpus. Choose the simplest one that retrieves complete, citable evidence on your own test questions.