Better Vector Search for Long Documents: Chunking Inside Manticore Search

Manticore Search added a `chunk_strategy` option to model-backed vector columns in `CREATE TABLE`, letting the database split long documents into chunks, embed each chunk, and search all of them without an external ingest pipeline or a separate chunks table. The feature offers five strategies — `truncate`, `mean`, `fixed`, `recursive`, and `sentence` — with tuning knobs `max_tokens`, `overlap_tokens`, and `max_chunks`. Measured on the 189-page, roughly 298,000-word Manticore manual, recall@5 for content buried past the embedding model's input window rose from 55.1% to 83.3% and MRR from 0.44 to 0.70, at about 2.5× the RAM and roughly 4× the ingest time.

Say you are building search over your team's internal documentation — guides, runbooks, postmortems. You have a table with auto embeddings https://manticoresearch.com/blog/auto-embeddings/ : you insert text, Manticore runs the model and fills the vector column for you. If that is new to you, start with vector search in Manticore https://manticoresearch.com/blog/vector-search/ . You load a 4,000-word document. The insert succeeds. The search works. Everything looks fine. Except the model you picked has a 512-token input window, and that document is about 5,000 tokens long. The model read the first 380 words and threw away the other 3,600. Nothing in the document past that point can ever be retrieved, and nothing anywhere told you. The embedding may not represent the document as a whole either. Until now, you would usually split the document into several pieces yourself, create embeddings for each one, and then work out how to combine the results if you wanted document search rather than chunk search. Manticore now handles this in the table definition: add chunk strategy to the vector column in CREATE TABLE , and Manticore splits each document into chunks, embeds every chunk, and searches all of them: DROP TABLE IF EXISTS docs; CREATE TABLE docs title text, content text, chunks float vector array knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title,content' chunk strategy='sentence' max tokens='256' overlap tokens='32' ; That is the whole feature. No ingest pipeline, no splitter library, no second table for chunks, no GROUP BY to fold chunk hits back into documents. TL;DR - Five strategies : truncate the old default , mean , fixed , recursive , sentence . Set with chunk strategy on a model-backed vector column. - truncate and mean produce one vector per document and work on a float vector column. fixed , recursive and sentence produce many, so they need a float vector array https://manual.manticoresearch.com/Creating a table/Data types Float-vector-array column. - A document is still one search result. Chunks compete individually, and Manticore returns the document once, with knn dist reporting the distance to its closest chunk. k counts documents, not chunks. - Tuning knobs : max tokens chunk size , overlap tokens shared tokens between neighbors , max chunks ceiling per document . - Measured on the Manticore manual 189 pages, ~298k words : for content buried past the model's window, recall@5 went from 55.1% → 83.3% and MRR from 0.44 → 0.70 , at ~2.5× the RAM and ~4× the ingest time. - Queries are never chunked. A query is short enough to embed as a whole; only stored documents are split. The problem, shown with a small example Suppose you have four documents: 1. Backup and restore runbook — about 700 words, roughly 900 tokens. Backup schedules, retention, restore drills, credentials, capacity planning. The last section explains how to rotate the TLS certificate used by the replication port. 2. Monitoring and alerting guide — unrelated. 3. Getting started with the CLI — unrelated. 4. TLS and certificates for the HTTP API — a short page that is entirely about certificates, and never mentions rotation or replication. You can create the table and add the documents using the commands below. So, what we have is: one table, three vector columns with the same source text — one column per strategy. A single INSERT fills all three, so the comparison conditions are identical: DROP TABLE IF EXISTS docs; CREATE TABLE docs title text, body text, v truncate float vector knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title,body', v mean float vector knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title,body' chunk strategy='mean', v sentence float vector array knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title,body' chunk strategy='sentence' max tokens='128' overlap tokens='32' ; Insert the four documents INSERT INTO docs id, title, body VALUES 1, 'Backup and restore runbook', 'Nightly backups run at 02:00 UTC from the standby node. The job snapshots every table directory, writes a manifest, and uploads the result to object storage. Retention is thirty daily copies, twelve monthly copies, and one yearly copy. A restore drill runs on the first Monday of each month against a scratch cluster. The drill counts as passed only when a full-text search over the restored data returns the same document count as production. Anything less is treated as a failed drill and investigated the same week. Before a restore, freeze the target cluster so that no writes land while files are being replaced. Copy the manifest first and verify its checksum. If the checksum does not match, stop: a partial restore is worse than no restore, because the cluster will start and silently serve half the corpus. After the files are in place, unfreeze and let replication catch up. Watch the queue depth. If it does not drain within ten minutes, the node is probably still reading from cold storage and needs a warm-up pass before it can serve traffic. Backup failures page the on-call engineer. The three most common causes are an expired object storage credential, a disk that filled up while the snapshot was being written, and a table left frozen by a previous failed run. All three are recoverable without data loss. Check the job log first, then the disk, then the freeze state of every table. Capacity planning for backups is boring but it matters. A daily copy of the search cluster is roughly the size of the data directory plus fifteen percent for the manifest and metadata. Multiply by the retention count, add the transfer cost, and you have the monthly bill. Most teams discover too late that the yearly copies dominate the storage line. Object storage lifecycle rules do most of the retention work. Daily copies move to infrequent access after seven days and expire after thirty. Monthly copies move to archive after sixty days. Yearly copies never expire automatically; deleting one is a manual action that requires a second approver. Credentials for the backup job live in the secret manager and are issued to a role, not to a person. The role can write new objects and list the bucket. It cannot delete, and it cannot read objects older than the current day. That last restriction is the cheapest defence against a compromised backup runner turning into a data exfiltration path. Documentation for each table lives next to its schema: what the table is for, who owns it, how large it is expected to get, and whether it can be rebuilt from an upstream source. A table that can be rebuilt does not need thirty daily copies. Roughly half of most clusters turns out to be derived data that nobody had marked as derived. Verification is not the same as the job exiting zero. The job can succeed while producing an unusable copy: an empty table, a truncated upload, a manifest that references a file that was never written. The verification step reads the manifest back, checks every referenced object exists and matches its recorded size, and compares row counts on three sampled tables against production. Rotating the replication TLS certificate is a separate procedure and the step people most often get wrong. The certificate that secures the replication port is not the same as the one the HTTP API uses, and replacing one does not replace the other. Generate the new key and signing request on the node that will be rotated first, sign them with the cluster certificate authority, and place the files next to the existing ones rather than on top of them. Then update the node configuration to point at the new paths and reload. Do one node at a time and confirm that the cluster reports every peer as synced before moving on. A half-rotated cluster where two nodes trust different authorities will keep accepting writes on both sides and diverge quietly. When every node has been rotated, remove the old key material and revoke the retired certificate at the authority.' , 2, 'Monitoring and alerting guide', 'Every node exports metrics over an HTTP endpoint that a scraper collects once per fifteen seconds. The dashboards are grouped into four rows: traffic, latency, saturation, and errors. Traffic is queries per second broken down by table. Latency is the ninety-fifth and ninety-ninth percentile of query time, measured server side. Alerting is deliberately thin. Paging alerts fire on sustained error rate above one percent for five minutes, on ninety-ninth percentile latency above two seconds for ten minutes, and on a node dropping out of the cluster. Everything else is a ticket, not a page. Teams that page on every anomaly stop reading pages within a month. Log retention is fourteen days hot and ninety days cold. The query log records the query text, the table, the match count, and the elapsed time. Turning it on costs a few percent of throughput and is almost always worth it, because most performance investigations start with a slow query nobody knew was being issued.' , 3, 'Getting started with the CLI', 'The command line client connects over the MySQL wire protocol, so any MySQL client works and you do not need to install anything special. Point it at port 9306 and you get an interactive shell. The shell understands the usual conveniences: history, tab completion of table names, and vertical output when a row is too wide for the terminal. Start by listing tables, then look at one with SHOW CREATE TABLE. The output is the exact statement that would recreate the table, including every option that was applied implicitly, which makes it the fastest way to find out what a table actually does rather than what someone documented two years ago. Bulk loading from the shell is possible but rarely what you want. For anything above a few thousand rows, use the HTTP bulk endpoint or one of the log shipper integrations, both of which batch and retry for you.' , 4, 'TLS and certificates for the HTTP API', 'The HTTP API can be served over TLS. You supply a certificate, a private key, and optionally a chain file, and the listener starts speaking HTTPS instead of HTTP. Clients that present a certificate of their own can be authenticated by it, which is the usual way to lock an internal API down without putting a password in every config file. Certificates for the HTTP API come from wherever your organisation gets certificates: a public authority, an internal authority, or an automated issuer. The file format is PEM. Both the certificate and the key must be readable by the user the server runs as, and the key must not be world readable or the listener refuses to start. Debugging TLS problems is mostly about reading the handshake. A client that reports an unknown authority is missing the chain. A client that reports a hostname mismatch is connecting by an address that is not in the certificate. A client that hangs is usually talking TLS to a plaintext port.' ; Now ask a question whose answer lives in the runbook's last section, once per strategy: SELECT title, knn dist FROM docs WHERE knn v truncate, 4, 'how do I rotate the TLS certificate used for replication' ; SELECT title, knn dist FROM docs WHERE knn v mean, 4, 'how do I rotate the TLS certificate used for replication' ; SELECT title, knn dist FROM docs WHERE knn v sentence, 4, 'how do I rotate the TLS certificate used for replication' ; | Strategy | 1st result | 2nd result | |---|---|---| | truncate default | TLS and certificates for the HTTP API — 0.762 | Backup and restore runbook — 0.936 | | mean | Backup and restore runbook — 0.656 | TLS and certificates for the HTTP API — 0.762 | | sentence , 128 tokens, 32 overlap | Backup and restore runbook — 0.254 | TLS and certificates for the HTTP API — 0.700 | With truncate , the document that actually answers the question loses to a decoy that merely looks like it is about certificates. The runbook's single vector was built from its opening pages on backup schedules and restore drills, because that is all the model was allowed to read. With sentence chunking, the runbook is stored as nine vectors instead of one: SELECT id, title, LENGTH v sentence AS chunks FROM docs ORDER BY id ASC; +------+---------------------------------------+--------+ | id | title | chunks | +------+---------------------------------------+--------+ | 1 | Backup and restore runbook | 9 | | 2 | Monitoring and alerting guide | 2 | | 3 | Getting started with the CLI | 2 | | 4 | TLS and certificates for the HTTP API | 2 | +------+---------------------------------------+--------+ One of those nine is the certificate-rotation paragraph. It matches the query almost exactly, so the document wins by a wide margin: 0.254 against 0.700. More about the chunking strategies | Strategy | Vectors per document | Column type | What it does | |---|---|---|---| | truncate | 1 | float vector | Embeds as much as fits the model's window, drops the rest. The only mode available in older versions, and still the default. | | mean | 1 | float vector | Splits the whole document, embeds every piece, averages them into one vector. | | fixed | N | float vector array | Fixed windows of max tokens tokens. | | recursive | N | float vector array | Splits on a separator hierarchy — paragraph, then line, then sentence, then space — keeping each piece within max tokens . | | sentence | N | float vector array | Sentence boundaries Unicode UAX 29 https://www.unicode.org/reports/tr29/ , packed up to max tokens . | The important distinction is not how the text is cut. It is what a match means . With one vector per document, search asks: "is this document, as a whole, similar to the query?" A single relevant paragraph is diluted by everything around it, and a document that covers five topics ends up not really matching any of them. With one vector per chunk, search asks: "does this document contain something similar?" Each chunk competes on its own merits, and Manticore returns the document once, scored by its best chunk. truncate — keep it when your documents are short title text, v float vector knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title' This is what you already have; chunk strategy='truncate' is the default and you never have to write it. It's the right choice — and the fastest, and the smallest — whenever your text genuinely fits the model's window: product titles, short descriptions, tags, chat messages, log lines, search queries, commit subjects. How much fits? More than most people assume, and less than they hope. all-MiniLM-L6-v2 takes 512 tokens, roughly 380 English words. text-embedding-3-small takes 8,192. If your 95th-percentile document is comfortably under the limit, stop reading and keep truncate . When it hurts: anything long-form. Documentation pages, knowledge-base articles, contracts, transcripts, email threads, wiki pages, README files, incident postmortems. mean — one vector, but the whole document v float vector knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title,content' chunk strategy='mean' Manticore splits the document, embeds every chunk, and averages the chunk vectors into a single normalized vector. Storage and search cost are identical to truncate — one vector per document, one HNSW node — but nothing is thrown away. Use it when: - You want the tail to count but can't afford more vectors — a very large corpus where index RAM is the binding constraint. - The column is a plain float vector and you can't change the type for example you're adding the column to an existing table with ALTER , which multi-vector strategies don't support . - Your documents are about one thing , just long. A single product's full description, one recipe, one job posting. Do not use it when a document covers several unrelated topics. Averaging a legal contract's indemnity clause with its payment terms produces a vector that sits between them and is close to neither. In our benchmark below, mean recovered about a third of the gap that chunking closes — a real improvement, and clearly not the same thing. fixed — predictable, cheapest to reason about v float vector array knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='content' chunk strategy='fixed' max tokens='256' overlap tokens='32' Cut every max tokens tokens, no matter what the text is doing at that point. Chunk count is a straight function of document length, so index size is predictable before you load anything. Use it when the text has no reliable structure to exploit: OCR output, scraped HTML that lost its paragraphs, machine transcripts without punctuation, log dumps, minified content. Also a fine default when you simply want the cheapest thing that stops truncation. The cost: a boundary can land mid-sentence, and a chunk that begins in the middle of a thought embeds badly. That is exactly what overlap tokens is for — see below. recursive — the best general default for prose v float vector array knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title,content' chunk strategy='recursive' max tokens='256' overlap tokens='32' Same token budget as fixed , but each cut is pulled back to the nearest natural boundary: a blank line first, then a line break, then a sentence end, then a space. A chunk stops where the text stops, not where the counter runs out. The boundary is never dragged back past the midpoint of the chunk, so you don't get a stream of tiny fragments. If you have used LangChain's RecursiveCharacterTextSplitter , this is the same idea, except it runs inside the database on the model's real tokens instead of characters, and there is nothing to install. Use it for: Markdown and HTML documentation, wiki pages, knowledge bases, blog posts, README files, structured reports — anything written by a human in paragraphs. This scored highest on deep content in our benchmark. sentence — when a chunk must be a complete thought v float vector array knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='content' chunk strategy='sentence' max tokens='256' overlap tokens='32' Detects sentence boundaries with Unicode UAX 29 https://www.unicode.org/reports/tr29/ , then greedily packs whole sentences until the token budget is reached. A chunk never starts or ends mid-sentence. A single sentence longer than the budget is split by the token window, as a last resort. Use it for: support tickets and email threads, chat and meeting transcripts, legal and policy text, news, customer reviews, medical and scientific abstracts — anything where a fragment of a sentence changes or destroys the meaning. It is also the strategy to pick when chunks will be fed to an LLM afterwards, because a chunk that ends mid-clause reads badly in a prompt. sentence is a little more conservative than recursive : it produced fewer, cleaner chunks in our tests and scored about the same on recall@5. The three knobs chunk strategy = truncate | mean | fixed | recursive | sentence max tokens = chunk size in tokens; 0 default = the model's own limit overlap tokens = tokens shared between consecutive chunks; needs a non-zero max tokens max chunks = ceiling on vectors per document; 0 default = unlimited max tokens is capped at what the model can actually accept — ask for 4,096 on a 512-token model and you still get 512, not an error. Smaller chunks mean sharper matches and more vectors; larger chunks mean more context per vector and fewer of them. For English prose, 128–512 covers almost every use case; we used 256 throughout the benchmark. overlap tokens repeats the tail of each chunk at the head of the next, so a sentence that straddles a boundary still appears intact somewhere. 10–20% of max tokens is the usual setting. Manticore guarantees forward progress: fixed and recursive cap the overlap at half the chunk size, and sentence re-seeds the next chunk with at most overlap tokens worth of trailing whole sentences while always advancing by at least one sentence. It requires an explicit non-zero max tokens — overlap against "whatever the model's limit happens to be" isn't a meaningful setting, so Manticore rejects it. max chunks limits the impact of unusually large documents. Without it, a 400-page PDF pasted into one row becomes thousands of HNSW nodes. With it, Manticore merges the overflow into the last kept chunk, then truncates it to the model's window when embedding: -- a ~600-token document, chunked at 64 tokens chunk strategy='fixed' max tokens='64' -- 22 vectors chunk strategy='fixed' max tokens='64' max chunks='3' -- 3 vectors Use it as a guard rail against outliers, not as a way to save memory across the board. What search looks like Nothing about your query changes from before. There is no chunk table, no nested field, no join, no GROUP BY . Here is the complete example: DROP TABLE IF EXISTS notes; CREATE TABLE notes title text, body text, chunks float vector array knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title,body' chunk strategy='sentence' max tokens='32' ; INSERT INTO notes id, title, body VALUES 1, 'Certificate rotation', 'The replication certificate is not the one the HTTP API uses. Generate the new key on the node being rotated and sign it with the cluster authority. Update the paths and reload, one node at a time, confirming every peer reports as synced before you move on.' , 2, 'Disk pressure', 'When a data directory crosses eighty percent the merge scheduler stops compacting and the node starts refusing writes. Free space first, then trigger a manual OPTIMIZE. Adding a disk without draining the queue only postpones the problem.' , 3, 'Slow queries', 'Turn the query log on before guessing. Most investigations end at a single query nobody knew was being issued, usually one that sorts on an unindexed attribute over the whole table.' ; SELECT id, title, knn dist FROM notes WHERE knn chunks, 3, 'how do I replace an expiring certificate on every node' ; +------+----------------------+------------+ | id | title | knn dist | +------+----------------------+------------+ | 1 | Certificate rotation | 0.51039070 | | 2 | Disk pressure | 0.91703475 | | 3 | Slow queries | 1.02606630 | +------+----------------------+------------+ max tokens='32' is small on purpose here, so that these short notes actually split and you can see the multi-vector behaviour on a toy dataset. LENGTH on the vector column shows how each document was divided: SELECT id, title, LENGTH chunks AS n FROM notes ORDER BY n DESC LIMIT 5; +------+----------------------+------+ | id | title | n | +------+----------------------+------+ | 1 | Certificate rotation | 2 | | 2 | Disk pressure | 2 | | 3 | Slow queries | 2 | +------+----------------------+------+ Six vectors, three rows back. Search follows the rules described in Multiple vectors per document https://manual.manticoresearch.com/Searching/KNN Multiple-vectors-per-document : - A document matches if any of its vectors is near the query vector. - Manticore returns each match exactly once . knn dist is the distance to its closest chunk. - k counts documents , not vectors. knn chunks, 3, ... means three documents. - A document with no vectors is never returned. The same query over HTTP: POST /search { "table": "notes", "knn": { "field": "chunks", "query": "how do I replace an expiring certificate on every node", "k": 3 }, " source": "title" } ... { " id": 1, " score": 1, " knn dist": 0.51039070, " source": { "title": "Certificate rotation" } } ... Everything else on the KNN page keeps working as before: filtering, prefilter and postfilter strategies, quantization https://manticoresearch.com/blog/quantization/ , early termination https://manticoresearch.com/blog/knn-early-termination/ , and rescoring. Does it actually help? Numbers on our own manual We tested the feature on the Manticore English manual — 189 pages and about 298,000 words, ranging from a two-paragraph note to a 39,000-word changelog. The query set is generated mechanically, not hand-picked. For every page we took its section headings, kept only headings that are unique across the whole manual, and split them in two: - Deep-content queries 419 — headings that appear after the first ~1,200 characters of their page. That is a deliberately conservative line: the model's window is 512 tokens, roughly 2,000 characters, so a few of these still point at text truncate can partly see. The gap below is therefore an understatement, not an exaggeration. - Head-content queries 88 — headings inside the first ~1,200 characters. The control group: content truncate can already see. A query is a hit if KNN returns the page the heading came from, within the top k . Model: Xenova/all-MiniLM-L6-v2 384 dims, 512-token window running on Manticore's ONNX backend https://manticoresearch.com/blog/onnx-embeddings-speedup/ . Hardware: 32 threads. max tokens='256' , overlap tokens='32' for the multi-vector strategies. Quality numbers are deterministic for a given index; the timings are a single run per strategy on an otherwise idle box. Deep content — what chunking is for | Strategy | Vectors | Ingest | Index RAM | hit@1 | hit@5 | hit@10 | MRR | |---|---|---|---|---|---|---|---| | truncate | 189 | 21 s | 4.2 MB | 33.7% | 55.1% | 63.2% | 0.44 | | mean | 189 | 72 s | 4.2 MB | 43.9% | 65.2% | 74.7% | 0.54 | | fixed | 3,430 | 73 s | 9.5 MB | 56.3% | 81.1% | 86.2% | 0.68 | | recursive | 4,664 | 86 s | 11.7 MB | 58.7% | 83.3% | 89.5% | 0.70 | | sentence | 4,041 | 79 s | 10.6 MB | 55.4% | 83.5% | 89.0% | 0.68 | Chunking turns a coin flip into a working search. recall@5 goes from 55.1% to 83.3% , and the rank of the right answer improves just as much — MRR 0.44 → 0.70. Of the queries truncate could not answer in the top 5 at all, recursive recovers roughly two thirds. mean lands where you would expect: it recovers about a third of the gap for free, because it costs exactly nothing extra to store or search. Head content — the control group | Strategy | hit@1 | hit@5 | MRR | |---|---|---|---| | truncate | 65.9% | 86.4% | 0.74 | | mean | 59.1% | 83.0% | 0.69 | | fixed | 60.2% | 83.0% | 0.71 | | recursive | 58.0% | 86.4% | 0.70 | | sentence | 56.8% | 85.2% | 0.69 | For completeness, the control group is worth reading carefully. For content that the model could already see, truncate is still the most precise at rank 1 — 65.9% against 58.0% for recursive . A whole-document vector carries the page's overall topic, and when the query is about the page's opening subject, that context helps. By rank 5 the difference is gone: recursive matches truncate exactly at 86.4%. So the trade is a few points of top-1 precision on content near the beginning, in exchange for +28 points of recall on everything else. For a documentation search, a help center, or any RAG retriever that feeds 5–10 passages to an LLM, that is not a close call. Cost - Index RAM : 4.2 MB → 11.7 MB, about 2.5×, for ~25× as many vectors. Vectors are only part of what an RT table stores. The HNSW graph over those vectors also takes longer to build during chunk saves and OPTIMIZE , though Manticore builds it across all your cores https://manticoresearch.com/blog/knn-parallel-build/ . - Data load : 21 s → 86 s for 189 documents. Chunking means embedding the whole corpus instead of the first 380 words of each document, and the time scales with it. This is embedding cost, not chunking cost — the splitting itself is not measurable next to inference. - Query response time : 6.3 ms → 8.5 ms at p50. HNSW handles 4,664 vectors about as easily as 189 — see 2-pass HNSW, batched distances and AVX-512 https://manticoresearch.com/blog/knn-hnsw-performance/ for what carries that. If you use a paid embedding API, read that ingest number as a bill: chunking sends your whole corpus to the model instead of the head of each document, and you pay for every token of it. Local ONNX models have no per-token cost, which is a large part of why we made them fast https://manticoresearch.com/blog/onnx-embeddings-speedup/ . Recommendations for choosing a chunking strategy | Your data | Start with | |---|---| | Titles, names, short descriptions, tags, log lines | truncate | | Long but single-topic; or RAM is the hard limit; or the column is an existing float vector | mean | | Documentation, wikis, knowledge bases, articles, READMEs | recursive , max tokens 128–256 | | Support tickets, email, transcripts, legal text, reviews | sentence , max tokens 128–256 | | OCR, scraped HTML, machine transcripts, unstructured dumps | fixed , max tokens 256, plus overlap | | Chunks will be passed to an LLM as context | sentence , max tokens 384–512 | Overlap is deliberately absent from most of those: our sweep below could not measure a benefit from it on structured prose, and it costs vectors. Add it when a thought routinely straddles a boundary — unstructured transcripts, OCR, long narrative without paragraph breaks. How big should a chunk be? Chunk size is the setting that actually affects your results. The trade is direct: a smaller chunk is a sharper match on one idea, a larger chunk carries more context but dilutes each idea inside it. A paragraph buried in a long document only becomes findable once the chunk size is small enough to give it a vector of its own. We ran another test: recursive on the same 189-page manual, three chunk sizes × three overlap settings, and the same 419 deep queries. Two trends stand out: quality rises as chunks get smaller, while overlap adds cost without improving quality much. Note that the Y axis starts at 78%, not zero — the whole spread is about six points, so a zero-based axis would flatten it into a straight line. The numbers behind the chart: | max tokens | overlap tokens | Vectors | Index RAM | deep hit@5 | deep MRR | |---|---|---|---|---|---| | 128 | 0 | 8,256 | 17.3 MB | 85.2% | 0.718 | | 128 | 13 | 9,328 | 18.7 MB | 85.7% | 0.705 | | 128 | 32 | 11,623 | 22.6 MB | 85.2% | 0.694 | | 256 | 0 | 3,984 | 10.0 MB | 83.1% | 0.657 | | 256 | 26 | 4,525 | 10.9 MB | 84.5% | 0.681 | | 256 | 64 | 5,569 | 12.5 MB | 83.5% | 0.689 | | 512 | 0 | 1,973 | 6.9 MB | 80.2% | 0.655 | | 512 | 51 | 2,191 | 7.2 MB | 79.2% | 0.651 | | 512 | 128 | 2,666 | 8.0 MB | 79.7% | 0.660 | What we see: Smaller chunks win, consistently. Going from 512 to 128 tokens buys about five points of recall@5 80.2% → 85.2% and a large jump in ranking quality MRR 0.655 → 0.718 . It costs 4× the vectors and 2.5× the index RAM. Below 128 the chunks stop containing a whole thought, so this is not a slope you ride forever — but on long technical prose, 128–256 beat 512 every time. Overlap did essentially nothing for quality, and was not free. At 128 tokens, going from no overlap to 25% overlap moved recall@5 from 85.2% to 85.2% while adding 41% more vectors and 5 MB of RAM. The pattern holds at every size: the spread across overlap settings ±1.5 points is within the noise of a 419-query set, while the cost is not. This lines up with Chroma's chunking evaluation https://www.trychroma.com/research/evaluating-chunking , where plain recursive splitting at 200 tokens with no overlap scored 88.1% recall — within a few points of an LLM-driven splitter at 91.9% — and it is the opposite of the "always use 10–20% overlap" advice you will read in most RAG guides. The honest caveat: this is one corpus, one model, and queries that look like section headings. Overlap earns its keep when a single fact routinely straddles a boundary — long unbroken narrative, transcripts without structure — and recursive already snaps cuts to paragraph and sentence boundaries, which does much of the same work. So treat "start at 128–256 with no overlap, add overlap only if you can measure it helping" as the default, and check it on your own data with the recipe below. Comparing settings You do not have to guess, and you do not need two tables. A table can carry several model-backed vector columns , each with its own strategy, all filled from the same fields on the same INSERT : CREATE TABLE ab title text, body text, sent 256 float vector array knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title,body' chunk strategy='sentence' max tokens='256' overlap tokens='32', rec 128 float vector array knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title,body' chunk strategy='recursive' max tokens='128' overlap tokens='16' ; Load your corpus once, then run the same query against each column and compare. For example: SELECT id, LENGTH sent 256 AS sent chunks, LENGTH rec 128 AS rec chunks FROM ab; SELECT id, knn dist FROM ab WHERE knn sent 256, 5, 'how do I rotate the replication certificate' ; SELECT id, knn dist FROM ab WHERE knn rec 128, 5, 'how do I rotate the replication certificate' ; On a short runbook whose certificate section sits at the end, sentence /256 fits the whole document in a single chunk and answers at distance 0.515 ; recursive /128 splits it in two, isolates the certificate paragraph, and answers at 0.310 . Same row, same model, same query — only the chunk size differs. Build a dataset of real queries with answers you trust — even 50 is enough — and compare recall@5 across two or three columns, exactly as we did on the manual above. Then drop the losing column with ALTER TABLE ... DROP COLUMN and keep the winner. Recipes Documentation and help center search. Long Markdown pages, users asking questions in their own words. Chunk on structure and search across it: CREATE TABLE docs url string, title text, body text, chunks float vector array knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title,body' chunk strategy='recursive' max tokens='192' ; Note from='title,body' : the fields are joined before chunking, so the page title lands in the first chunk and gives it context. For a worked end-to-end example of this shape, see Vector search on GitHub https://manticoresearch.com/blog/github-semantic-search/ . Support tickets and email threads. A thread is a sequence of complete messages; cutting one mid-sentence loses the fact you need. Keep the chunk count bounded, because threads have no natural length limit: CREATE TABLE tickets ticket id bigint, customer string, status string, thread text, chunks float vector array knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='thread' chunk strategy='sentence' max tokens='256' overlap tokens='32' max chunks='64' ; SELECT ticket id, knn dist FROM tickets WHERE knn chunks, 10, 'customer was charged twice after upgrading' AND status = 'closed'; Filtering works exactly as it does for a single-vector column. Contracts and policy documents. Clause-level retrieval is the entire point — nobody wants "the contract" back, they want the indemnity clause. Smaller chunks, generous overlap: chunks float vector array knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='body' chunk strategy='sentence' max tokens='128' overlap tokens='32' Product catalog with long descriptions. One product is one topic, and catalogs are large, so pay nothing extra: embedding float vector knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='name,description' chunk strategy='mean' RAG: retrieval for an LLM. Whatever you retrieve gets pasted into a prompt, so chunks should read as prose — this is the retrieval half of conversational search https://manticoresearch.com/blog/conversational-search/ . Larger chunks, sentence boundaries, and ask for more of them: chunks float vector array knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title,body' chunk strategy='sentence' max tokens='512' overlap tokens='64' Adding chunking to a table you already have. Multi-vector columns can't be added by ALTER — existing rows have no vectors and there's no way to backfill them yet. A single-vector strategy can: ALTER TABLE docs ADD COLUMN v2 float vector knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title,body' chunk strategy='mean'; ALTER TABLE docs REBUILD EMBEDDINGS v2; For a multi-vector column, create the new table with the column in place and reindex into it. How other engines handle this Every vector engine now generates embeddings for you. Far fewer will split your document before doing it — and of those, most make you assemble it out of pipeline stages. | Engine | Embeds in-engine | Chunks in-engine | Strategies | One row per document at search time | |---|---|---|---|---| | Manticore Search | Yes — local + OpenAI / Voyage / Jina | Yes — chunk strategy on the vector column https://manual.manticoresearch.com/Searching/KNN Chunking-strategies | truncate, mean, fixed, recursive, sentence | Yes, native | | Elasticsearch | Yes — inference endpoints | Yes https://www.elastic.co/docs/explore-analyze/elastic-inference/inference-api chunking-settings | sentence default , word, recursive 9.1+ , none | Yes — semantic text hides the chunks | | OpenSearch | Yes — ML Commons | Yes — separate ingest processor https://docs.opensearch.org/latest/ingest-pipelines/processors/text-chunking/ | fixed token length, fixed char length, delimiter | Needs a nested field + nested query | | Vespa | Yes — built-in embedders | Yes — indexing expression https://docs.vespa.ai/en/rag/working-with-chunks.html | fixed-length, sentence, custom | Yes | | Azure AI Search | Yes — integrated vectorization | Yes — Split skill in a skillset https://learn.microsoft.com/en-us/azure/search/cognitive-search-skill-textsplit | pages chars , sentences | No — one row per chunk | | PostgreSQL + pgai | Yes — background worker | Yes https://github.com/timescale/pgai/blob/main/docs/vectorizer/api-reference.md | character, recursive character | No — separate table, join and dedupe | | Milvus / Zilliz | Yes — Function 2.6+ | No — app-side https://zilliz.com/learn/guide-to-chunking-strategies-for-rag | — | — | | Qdrant | Yes — Cloud Inference | No — app-side https://qdrant.tech/course/essentials/day-1/chunking-strategies/ | — | — | | Weaviate | Yes — vectorizer modules | No — app-side https://docs.weaviate.io/academy/py/standalone/chunking | — | — | | Meilisearch | Yes — embedders | No — app-side https://www.meilisearch.com/blog/rag-chunking-strategies | — | — | | Typesense | Yes | No — open request https://github.com/typesense/typesense/issues/1526 | — | — | | Apache Solr | Yes — LLM module https://solr.apache.org/guide/solr/latest/query-guide/text-to-vector.html 9.8+ | No | — | — | | Pinecone | Yes — integrated inference | No — app-side https://www.pinecone.io/learn/chunking-strategies/ | — | — | | MongoDB Atlas | Yes — Automated Embedding | No — app-side https://www.mongodb.com/resources/basics/chunking-explained | — | — | Every cell in the chunking column links to a source. A “yes” links to the feature's own documentation. An “app-side” links to that vendor's own guidance on chunking in your application — which is what they publish instead of an in-engine option. If we missed a feature, or one has shipped since, tell us https://github.com/manticoresoftware/manticoresearch/issues and we will fix it. Versions checked - 4 September 2026 The latest stable release of each product available that day: | Product | Version | |---|---| | Elasticsearch | 9.5.3 | | OpenSearch | 3.8.0 | | Vespa | 8.750.13 | | Azure AI Search | REST API 2026-04-01 | | PostgreSQL + pgai | extension 0.11.2 | | Milvus / Zilliz | 2.6.23 | | Qdrant | 1.19.0 | | Weaviate | 1.38.13 | | Meilisearch | 1.53.1 | | Typesense | 30.2 | | Apache Solr | 10.0.0 | | Pinecone, MongoDB Atlas | hosted services, no version to pin | Two things stand out. Chunking in-engine is still rare. Milvus, Qdrant, Weaviate, Pinecone, MongoDB Atlas, Typesense, Meilisearch and — since the 9.8 LLM module — Apache Solr will all run the embedding model for you, and every one of them will happily truncate your 4,000-word document without saying so. The splitting is your problem, in your application, in a language and a library that has no idea what tokenizer the model uses. Where chunking exists, the plumbing usually leaks. OpenSearch gets you there with a text chunking processor feeding a text embedding processor writing into a nested field, queried with a nested query and a score mode. Azure AI Search wants a skillset with a Split skill, an embedding skill and index projections — and returns one result row per chunk, so grouping back to documents is on you. pgai Vectorizer writes chunks to a second table, so every query is a join plus a DISTINCT ON . Elasticsearch's semantic text is genuinely close to Manticore's model: chunking settings on the inference endpoint, chunks hidden inside the field, one hit per document. Manticore does the same thing with less surface area: the strategy is an option on the column, the chunks are the column's value, and search returns documents. If you are weighing the whole stack rather than this one feature, we have written up the comparison with Elasticsearch https://manticoresearch.com/blog/manticore-alternative-to-elasticsearch/ and with Turbopuffer https://manticoresearch.com/blog/turbopuffer-vs-manticore/ too. What chunking does not fix Chunking solves one problem well — a document longer than the model's window is no longer half-invisible. It does not make retrieval perfect, and two known gaps are worth naming. A chunk does not know where it came from. Split a document and you get a paragraph that says "do one node at a time and confirm every peer reports as synced" with no indication of what is being rotated, or which product it belongs to. Anthropic's contextual retrieval https://www.anthropic.com/engineering/contextual-retrieval work put numbers on this: prepending a short, chunk-specific description of the surrounding document before embedding cut top-20 retrieval failures by 35%, and by 49% combined with a contextual BM25 index. Manticore does not do this for you. FROM joins its fields with a space before chunking, so listing title first puts the title at the head of the text that gets split — which means it lands in the first chunk and only that one. Every chunk after it is on its own: -- 'title' leads, so its words are in chunk 1; chunks 2..N never see them from='title,body' If you need every chunk to carry context, you have to build it into the stored text yourself before inserting — for example by repeating a short heading at the start of each section of body . There is no per-chunk prefix option today. Chunk boundaries are decided before the model sees the text. Manticore splits, then embeds each piece independently — the standard approach, and what every engine with in-engine chunking in the table above does. An alternative called late chunking https://jina.ai/news/late-chunking-in-long-context-embedding-models/ inverts it: run a long-context model over the whole document first, then pool the token embeddings into chunks, so each chunk vector carries context from the rest of the document. It needs a long-context model and more compute per document, and Manticore does not do it today. If your documents depend heavily on cross-paragraph context, it is worth knowing the option exists. Neither gap changes the basic result: for long documents, chunked retrieval beats truncated retrieval by a wide margin, and reaching it means adding chunk strategy to one column. Limits and gotchas max chunks discards text. Manticore merges overflow into the last kept chunk, then truncates it to the model's window. Nothing warns you. It's a guard rail for outliers, not a way to save memory across the board. Remote models chunk by bytes, not tokens. OpenAI, Voyage and Jina have no local tokenizer, so Manticore falls back to a deliberately conservative estimate of 3 bytes per token — a chunk lands under the provider's cap rather than over it. In practice max tokens='N' becomes an N × 3 -byte window. We measured it against a stub endpoint with a 3,599-byte document and the fixed strategy: | max tokens | Byte window | Chunks produced | |---|---|---| | 100 | 300 | 12 | | 200 | 600 | 6 | | 400 | 1,200 | 3 | English prose runs closer to 4 bytes per token, so on a remote model you get chunks roughly a quarter smaller than the number you asked for — set max tokens about 30% higher than you would for a local model to land in the same place. If exact boundaries matter, use a local model, where splitting is done on the model's real tokens. Multi-vector columns can't be added with ALTER . ALTER TABLE ... ADD COLUMN and ALTER TABLE ... REBUILD EMBEDDINGS on a model-backed float vector array are rejected. Recreate the table instead. Both work normally on a float vector , including with mean . Chunking applies to auto embeddings only. Vectors you insert yourself are stored exactly as given — Manticore never re-cuts data you supplied. chunk strategy without model name is a DDL error, on purpose. embeddings is a reserved word. EMBEDDINGS is a DDL keyword ALTER TABLE ... REBUILD EMBEDDINGS , so a column literally named embeddings is a syntax error unless escaped. Use escaping if you need that name. Queries are not chunked. A query is embedded whole, as a single vector. That is what you want: chunking exists to make a long document findable, not to split a fifteen-word question. The DDL tells you when a combination is wrong , at CREATE TABLE time rather than at the first insert: mysql CREATE TABLE t title text, v float vector ... chunk strategy='sentence' ; ERROR 1064: chunk strategy='sentence' produces several vectors per document and requires a float vector array attribute mysql ... chunk strategy='fixed' overlap tokens='32' ; ERROR 1064: overlap tokens requires an explicit non-zero max tokens mysql ... chunk strategy='paragraph' ; ERROR 1064: unknown chunk strategy 'paragraph'; expected truncate, mean, fixed, recursive or sentence mysql ... chunk strategy='truncate' max tokens='128' ; ERROR 1064: chunk strategy='truncate' ignores max tokens, overlap tokens and max chunks Try it The shortest path to a working chunked semantic search: DROP TABLE IF EXISTS docs; CREATE TABLE docs title text, content text, chunks float vector array knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title,content' chunk strategy='recursive' max tokens='256' overlap tokens='32' ; INSERT INTO docs id, title, content VALUES 1, 'Backup and restore runbook', 'Nightly backups run at 02:00 UTC ... ' ; SELECT id, title, knn dist FROM docs WHERE knn chunks, 5, 'how do I rotate the replication certificate' ; No model to download by hand, no splitter to pick, no pipeline to maintain. One column option, and the parts of your documents that used to be invisible start showing up in results. Full reference: Chunking strategies https://manual.manticoresearch.com/Searching/KNN Chunking-strategies and Multiple vectors per document https://manual.manticoresearch.com/Searching/KNN Multiple-vectors-per-document in the manual. Questions and bug reports on GitHub https://github.com/manticoresoftware/manticoresearch/issues .