Manticore Search 29.9.0: Chunked auto-embeddings and mmap columnar access

Manticore Search released version 29.9.0, adding chunked auto-embeddings that let long documents store multiple vectors instead of one, with five chunking strategies including truncate, mean, fixed, recursive, and sentence. The release also makes mmap the default access mode for columnar attributes, adds a MAX_INPUT_TOKENS option to cap local embedding work, and includes fixes across hybrid search, KNN, bulk ingestion, and grouped search. The update covers everything shipped after 29.0.2, from 29.0.3 through 29.9.0, with no release-wide mandatory data migration.

Manticore Search 29.9.0 /install/ has been released. The largest change is in auto-embeddings: long documents can now be split into searchable chunks, and a document can keep several vectors instead of compressing all of its content into one. This release also makes mmap the default access mode for columnar attributes, adds safer controls for embedding workloads, UTF-8 identifiers, better backup support, and fixes across hybrid search, KNN, bulk ingestion, grouped search, and schema changes. This post covers everything shipped after 29.0.2, from 29.0.3 through 29.9.0 . ❤️ We’d like to thank @tudorvasinca https://github.com/tudorvasinca for their work on PR 4857 https://github.com/manticoresoftware/manticoresearch/pull/4857 , PR 4859 https://github.com/manticoresoftware/manticoresearch/pull/4859 , and PR 4873 https://github.com/manticoresoftware/manticoresearch/pull/4873 . Upgrade notes There is no release-wide mandatory data migration. Existing tables and configuration can be upgraded normally, but a few fixes need follow-up if the earlier behavior already affected your data: - German sharp-s normalization is opt-in. Switching an existing table to lemmatize de v2 or lemmatize de v2 all changes indexed terms, so rebuild a plain table or replay documents into a new RT table. - Authenticated BACKUP introduces the backup authorization action. Before downgrading to 29.3.12 or earlier, remove any backup grants. - Columnar attributes now use mmap by default instead of buffered file reads. Set access columnar attrs='file' explicitly if you need to retain the previous access mode. Long documents can keep more than one embedding The old auto-embedding path produced one vector per document and truncated text that did not fit the model input window. That works for titles and short descriptions, but it means a relevant paragraph near the end of a long article may never reach the index. Manticore Search now supports five chunking strategies https://manual.manticoresearch.com/Searching/KNN Chunking-strategies : - truncate keeps the previous behavior and remains the default. - mean embeds every chunk and averages the results into one vector. - fixed splits text into fixed-size token windows. - recursive prefers paragraph, line, sentence, and space boundaries in that order. - sentence groups complete sentences up to the configured limit. The multi-vector strategies use float vector array https://manual.manticoresearch.com/Creating a table/Data types Float-vector-array . Each chunk competes independently during KNN search, but Manticore returns the document once and uses its closest chunk for knn dist : CREATE TABLE articles title text, content text, chunks float vector array knn type='hnsw' hnsw similarity='cosine' model name='Xenova/all-MiniLM-L6-v2' from='title,content' chunk strategy='sentence' max tokens='256' overlap tokens='32' ; INSERT INTO articles id, title, content VALUES 1, 'Rotating certificates', 'A long guide with many sections ...' ; SELECT id, knn dist FROM articles WHERE knn chunks, 5, 'how do I rotate a certificate' ; MAX TOKENS , OVERLAP TOKENS , and MAX CHUNKS control chunk size, shared context at boundaries, and the maximum vector count. Declare a model-backed float vector array when creating the table: adding one later with ALTER TABLE ... ADD COLUMN and rebuilding its embeddings are not supported yet. Put a ceiling on local embedding work Long-context models can make a single large input unexpectedly expensive, especially on CPU. The new MAX INPUT TOKENS https://manual.manticoresearch.com/Searching/KNN Auto-Embeddings-%28Recommended%29 column option caps how much of each input is sent to a local embedding model: ALTER TABLE articles MODIFY COLUMN chunks MAX INPUT TOKENS='512'; The change applies to embeddings generated afterward; existing vectors stay as they are. Set it during CREATE TABLE or change it later without re-embedding the table. A value of 0 , or leaving the option out, uses the model's own limit. This release also fixes two less visible problems around this path. Models no longer remain cached after an embedding column is modified, and configurations with different API TIMEOUT or MAX INPUT TOKENS values no longer collide and reuse the wrong cached model. UTF-8 identifiers and better German matching Table, field, and attribute names now follow one consistent safe UTF-8 identifier syntax https://manual.manticoresearch.com/Creating a table/Data types Table-and-field-name-syntax . RT, percolate, distributed, template, and plain tables can use localized names such as Chinese or Cyrillic identifiers across DDL, expressions, field selectors, and inferred source schemas. German AOT morphology also gains opt-in sharp-s normalization. With charset table=non cont,german and morphology=lemmatize de v2 or lemmatize de v2 all , forms such as Straße , Strasse , and STRAẞE match in ordinary whole-word searches. If index exact words=1 is enabled, exact-word queries can still distinguish the ß and ss forms. Better load testing and backups manticore-load https://github.com/manticoresoftware/manticore-load can now benchmark through the HTTP JSON API with --http . Writes use /bulk , searches use /search , and --table selects the target table. Its reports now include local searchd RSS during the run plus peak RSS, disk, and CPU statistics at the end, including aggregate monitoring for multi-command workloads. Manticore Backup https://github.com/manticoresoftware/manticoresearch-backup now works with authenticated Manticore Search installations through username/password or bearer-token credentials. SQL BACKUP https://manual.manticoresearch.com/Securing and compacting a table/Backup and restore General-syntax-of-BACKUP has a dedicated authorization action and checks read access to the selected tables. S3 backups and restores can also use the AWS SDK credential provider chain when static keys are not set. That includes IRSA, shared credentials, ECS task roles, and EC2 instance profiles. Temporary credentials can supply AWS SESSION TOKEN . Columnar attributes use mmap by default Manticore Search now defaults access columnar attrs https://manual.manticoresearch.com/Creating a table/Local tables/Plain and real-time table settings Accessing-table-files to mmap . The operating system maps and caches .spc columnar-attribute files on demand, without prereading the whole file at startup. The previous buffered path remains available with access columnar attrs='file' . ALTER TABLE also reopens replaced columnar storage with the configured access mode, so an altered table no longer falls back to a different reader than the one requested. Vector and grouped-search fixes Several fixes target queries that were valid but could return incomplete results or fail under a particular table layout: - Distributed and sharded KNN queries with a local shard no longer rescore merged 1-bit-quantized results twice, which could crash the coordinator or return the wrong nearest neighbor. Issue 4791 https://github.com/manticoresoftware/manticoresearch/issues/4791 - KNN queries with additional filters avoid a redundant knn dist prefilter when HNSW already excludes documents without vectors. PR 4861 https://github.com/manticoresoftware/manticoresearch/pull/4861 - LENGTH on a float vector array now reports the number of vectors rather than its internal storage-word count. PR 4879 https://github.com/manticoresoftware/manticoresearch/pull/4879 - Hybrid search with GROUP BY retains all buckets, including MVA groups, and follows both final and within-group ordering. Issue 4639 https://github.com/manticoresoftware/manticoresearch/issues/4639 - Hybrid filters on weight and expressions or aliases derived from it now run after fusion against the final text weight instead of being ignored. Weight-dependent filters inside OR trees remain unsupported and return an explicit error. Issue 4889 https://github.com/manticoresoftware/manticoresearch/issues/4889 - Multi-chunk RT grouping no longer risks duplicate groups, incorrect split counts, or a hang while ordering COUNT DISTINCT ... results. Issue 4856 https://github.com/manticoresoftware/manticoresearch/issues/4856 - Document-ID filters whose signed representation is negative now follow the lookup index's unsigned ordering. Issue 4774 https://github.com/manticoresoftware/manticoresearch/issues/4774 There are crash fixes here too: a second hybrid-search statement in a multi-statement request PR 4864 https://github.com/manticoresoftware/manticoresearch/pull/4864 , distributed JSON aggregation sorted by a string attribute Issue 4822 https://github.com/manticoresoftware/manticoresearch/issues/4822 , and dropping a table during auto-embedding precommit Issue 4860 https://github.com/manticoresoftware/manticoresearch/issues/4860 are all handled safely now. Bulk ingestion behaves predictably Elasticsearch-compatible / bulk requests now return HTTP 200 once a batch has been processed, while individual failures remain visible through errors: true and per-item statuses. Duplicate create actions return per-item 409 version conflict engine exception errors, including duplicates within the same batch. This prevents clients such as Fluent Bit from retrying writes that already succeeded. Fixed-length gzip-compressed /bulk bodies are also decoded correctly when they arrive across multiple socket reads. And if native bulk processing fails because the target table does not exist, the request can again reach Manticore's auto-schema fallback with valid NDJSON while preserving the expected bulk response envelope. More reliability fixes The rest of the release closes a broad set of operational and compatibility problems: - searchd --stopwait no longer hangs while a sharded table is being rebalanced after a node rejoins. Issue 3905 https://github.com/manticoresoftware/manticoresearch/issues/3905 - Compatible older binlogs replay safely during upgrade and completed RT chunks are published before clean shutdown. Issue 4808 https://github.com/manticoresoftware/manticoresearch/issues/4808 Fatal replay diagnostics also name the relevant recovery flag. Issue 4811 https://github.com/manticoresoftware/manticoresearch/issues/4811 - indexer creates missing parent directories for plain-table paths when the nearest existing parent is writable. Issue 4793 https://github.com/manticoresoftware/manticoresearch/issues/4793 - UUID document IDs no longer cause stored text fields to come back empty. Issue 4833 https://github.com/manticoresoftware/manticoresearch/issues/4833 - ALTER TABLE ... RENAME preserves hidden remote embedding API keys without exposing them in SHOW CREATE TABLE . Issue 4842 https://github.com/manticoresoftware/manticoresearch/issues/4842 - Sequel Ace 5.3.1+ compatibility probes work again. Issue 4828 https://github.com/manticoresoftware/manticoresearch/issues/4828 - JSON /search keeps distances for negated NEAR and proximity operators. Issue 4784 https://github.com/manticoresoftware/manticoresearch/issues/4784 - Internal string-sort helper columns no longer leak from LEFT JOIN results. Issue 4788 https://github.com/manticoresoftware/manticoresearch/issues/4788 - Malformed binary API SEARCH element counts are rejected instead of terminating searchd . PR 4790 https://github.com/manticoresoftware/manticoresearch/pull/4790 For the complete list, see the Version 29.9.0 changelog https://manual.manticoresearch.com/Changelog Version-29.9.0 . Get Manticore Search 29.9.0 Install or upgrade Manticore Search with the installation guide /install/ . Review the upgrade notes above if you use columnar attributes, German AOT morphology, or authenticated backups. Need help or want to connect? - Join our Slack https://slack.manticoresearch.com - Visit the Forum https://forum.manticoresearch.com - Report issues or suggest features on GitHub https://github.com/manticoresoftware/manticoresearch/issues - Email us at \ email protected\ /cdn-cgi/l/email-protection