Higher Vector Search for Long Documents: Chunking Infacet Manticore Search

Say you might be constructing search over your crew’s inner documentation — guides, runbooks, postmortems. You have a desk with auto embeddings
: you insert textual content, Manticore runs the mannequin and fills the vector column for you. (If that’s new to you, begin with vector search in Manticore
.) You load a 4,000-word doc. The insert succeeds. The search works. Everything appears to be like superb.

Except the mannequin you picked has a 512-token enter window, and that doc is about 5,000 tokens lengthy. The mannequin learn the primary 380 phrases and threw away the opposite 3,600. Nothing within the doc previous that time can ever be retrieved, and nothing wherever instructed you. The embedding could not signify the doc as a complete both.

Until now, you’d often cut up the doc into a number of items your self, create embeddings for each, after which work out find out how to mix the outcomes should you wished doc search relatively than chunk search. Manticore now handles this within the desk definition: add chunk_strategy to the vector column in CREATE TABLE, and Manticore splits every doc into chunks, embeds each chunk, and searches all of them:

DROP TABLE IF EXISTS docs;

CREATE TABLE docs (
  title textual content,
  content material textual content,
  chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,content material'
    chunk_strategy='sentence' max_tokens='256' overlap_tokens='32'
);

That is the entire characteristic. No ingest pipeline, no splitter library, no second desk for chunks, no GROUP BY to fold chunk hits again into paperwork.

TL;DR

• Five methods: truncate (the outdated default), imply, mounted, recursive, sentence. Set with chunk_strategy on a model-backed vector column.
• truncate and imply produce one vector per doc and work on a float_vector column. mounted, recursive and sentence produce many, in order that they want a float_vector_array
  column.
• A doc continues to be one search end result. Chunks compete individually, and Manticore returns the doc as soon as, with knn_dist() reporting the gap to its closest chunk. ok counts paperwork, not chunks.
• Tuning knobs: max_tokens (chunk measurement), overlap_tokens (shared tokens between neighbors), max_chunks (ceiling per doc).
• Measured on the Manticore handbook (189 pages, ~298k phrases): for content material buried previous the mannequin’s window, recall@5 went from 55.1% → 83.3% and MRR from 0.44 → 0.70, at ~2.5× the RAM and ~4× the ingest time.
• Queries are by no means chunked. A question is brief sufficient to embed as a complete; solely saved paperwork are cut up.

The drawback, proven with a small instance

Suppose you will have 4 paperwork:

1. Backup and restore runbook — about 700 phrases, roughly 900 tokens. Backup schedules, retention, restore drills, credentials, capability planning. The final part explains find out how to rotate the TLS certificates utilized by the replication port.
2. Monitoring and alerting information — unrelated.
3. Getting began with the CLI — unrelated.
4. TLS and certificates for the HTTP API — a brief web page that’s fully about certificates, and by no means mentions rotation or replication.

You can create the desk and add the paperwork utilizing the instructions under.

So, what we have now is: one desk, three vector columns with the identical supply textual content — one column per technique. A single INSERT fills all three, so the comparability circumstances are equivalent:

DROP TABLE IF EXISTS docs;
CREATE TABLE docs (
  title textual content,
  physique textual content,
  v_truncate float_vector       knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,physique',
  v_mean     float_vector       knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,physique' chunk_strategy='imply',
  v_sentence float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,physique'
    chunk_strategy='sentence' max_tokens='128' overlap_tokens='32'
);

INSERT INTO docs (id, title, physique) VALUES
  (1, 'Backup and restore runbook',
   'Nightly backups run at 02:00 UTC from the standby node. The job snapshots each desk listing, writes a manifest, and uploads the end result to object storage. Retention is thirty every day copies, twelve month-to-month copies, and one yearly copy. A restore drill runs on the primary Monday of every month towards a scratch cluster. The drill counts as handed solely when a full-text search over the restored knowledge returns the identical doc rely as manufacturing. Anything much less is handled as a failed drill and investigated the identical week. Before a restore, freeze the goal cluster in order that no writes land whereas recordsdata are being changed. Copy the manifest first and confirm its checksum. If the checksum doesn't match, cease: a partial restore is worse than no restore, as a result of the cluster will begin and silently serve half the corpus. After the recordsdata are in place, unfreeze and let replication catch up. Watch the queue depth. If it doesn't drain inside ten minutes, the node might be nonetheless studying from chilly storage and desires a warm-up cross earlier than it may possibly serve site visitors. Backup failures web page the on-call engineer. The three commonest causes are an expired object storage credential, a disk that stuffed up whereas the snapshot was being written, and a desk left frozen by a earlier failed run. All three are recoverable with out knowledge loss. Check the job log first, then the disk, then the freeze state of each desk. Capacity planning for backups is boring but it surely issues. A every day copy of the search cluster is roughly the dimensions of the information listing plus fifteen % for the manifest and metadata. Multiply by the retention rely, add the switch value, and you've got the month-to-month invoice. Most groups uncover too late that the yearly copies dominate the storage line. Object storage lifecycle guidelines do a lot of the retention work. Daily copies transfer to rare entry after seven days and expire after thirty. Monthly copies transfer to archive after sixty days. Yearly copies by no means expire robotically; deleting one is a handbook motion that requires a second approver. Credentials for the backup job reside within the secret supervisor and are issued to a task, to not an individual. The function can write new objects and listing the bucket. It can not delete, and it can not learn objects older than the present day. That final restriction is the most affordable defence towards a compromised backup runner turning into a knowledge exfiltration path. Documentation for every desk lives subsequent to its schema: what the desk is for, who owns it, how giant it's anticipated to get, and whether or not it may be rebuilt from an upstream supply. A desk that may be rebuilt doesn't want thirty every day copies. Roughly half of most clusters seems to be derived knowledge that no one had marked as derived. Verification is just not the identical because the job exiting zero. The job can succeed whereas producing an unusable copy: an empty desk, a truncated add, a manifest that references a file that was by no means written. The verification step reads the manifest again, checks each referenced object exists and matches its recorded measurement, and compares row counts on three sampled tables towards manufacturing. Rotating the replication TLS certificates is a separate process and the step individuals most frequently get incorrect. The certificates that secures the replication port is just not the identical because the one the HTTP API makes use of, and changing one doesn't substitute the opposite. Generate the brand new key and signing request on the node that can be rotated first, signal them with the cluster certificates authority, and place the recordsdata subsequent to the prevailing ones relatively than on high of them. Then replace the node configuration to level on the new paths and reload. Do one node at a time and ensure that the cluster reviews each peer as synced earlier than transferring on. A half-rotated cluster the place two nodes belief completely different authorities will hold accepting writes on either side and diverge quietly. When each node has been rotated, take away the outdated key materials and revoke the retired certificates on the authority.'),
  (2, 'Monitoring and alerting information',
   'Every node exports metrics over an HTTP endpoint {that a} scraper collects as soon as per fifteen seconds. The dashboards are grouped into 4 rows: site visitors, latency, saturation, and errors. Traffic is queries per second damaged down by desk. Latency is the ninety-fifth and ninety-ninth percentile of question time, measured server facet. Alerting is intentionally skinny. Paging alerts hearth on sustained error charge above one % for 5 minutes, on ninety-ninth percentile latency above two seconds for ten minutes, and on a node dropping out of the cluster. Everything else is a ticket, not a web page. Teams that web page on each anomaly cease studying pages inside a month. Log retention is fourteen days scorching and ninety days chilly. The question log data the question textual content, the desk, the match rely, and the elapsed time. Turning it on prices a couple of % of throughput and is nearly all the time price it, as a result of most efficiency investigations begin with a gradual question no one knew was being issued.'),
  (3, 'Getting began with the CLI',
   'The command line consumer connects over the MySQL wire protocol, so any MySQL consumer works and you do not want to put in something particular. Point it at port 9306 and also you get an interactive shell. The shell understands the same old conveniences: historical past, tab completion of desk names, and vertical output when a row is just too broad for the terminal. Start by itemizing tables, then have a look at one with SHOW CREATE TABLE. The output is the precise assertion that might recreate the desk, together with each possibility that was utilized implicitly, which makes it the quickest solution to discover out what a desk truly does relatively than what somebody documented two years in the past. Bulk loading from the shell is feasible however hardly ever what you need. For something above a couple of thousand rows, use the HTTP bulk endpoint or one of many log shipper integrations, each of which batch and retry for you.'),
  (4, 'TLS and certificates for the HTTP API',
   'The HTTP API may be served over TLS. You provide a certificates, a non-public key, and optionally a sequence file, and the listener begins talking HTTPS as an alternative of HTTP. Clients that current a certificates of their very own may be authenticated by it, which is the same old solution to lock an inner API down with out placing a password in each config file. Certificates for the HTTP API come from wherever your organisation will get certificates: a public authority, an inner authority, or an automatic issuer. The file format is PEM. Both the certificates and the important thing have to be readable by the consumer the server runs as, and the important thing should not be global community readable or the listener refuses to begin. Debugging TLS issues is usually about studying the handshake. A consumer that reviews an unknown authority is lacking the chain. A consumer that reviews a hostname mismatch is connecting by an tackle that's not within the certificates. A consumer that hangs is often speaking TLS to a plaintext port.');

Now ask a query whose reply lives within the runbook’s final part, as soon as per technique:

SELECT title, knn_dist() FROM docs
WHERE knn(v_truncate, 4, 'how do I rotate the TLS certificates used for replication');

SELECT title, knn_dist() FROM docs
WHERE knn(v_mean, 4, 'how do I rotate the TLS certificates used for replication');

SELECT title, knn_dist() FROM docs
WHERE knn(v_sentence, 4, 'how do I rotate the TLS certificates used for replication');

With truncate, the doc that really solutions the query loses to a decoy that merely appears to be like like it’s about certificates. The runbook’s single vector was constructed from its opening pages on backup schedules and restore drills, as a result of that’s all of the mannequin was allowed to learn.

With sentence chunking, the runbook is saved as 9 vectors as an alternative of 1:

SELECT id, title, LENGTH(v_sentence) AS chunks FROM docs ORDER BY id ASC;

+------+---------------------------------------+--------+
| id   | title                                 | chunks |
+------+---------------------------------------+--------+
|    1 | Backup and restore runbook            |      9 |
|    2 | Monitoring and alerting information         |      2 |
|    3 | Getting began with the CLI          |      2 |
|    4 | TLS and certificates for the HTTP API |      2 |
+------+---------------------------------------+--------+

One of these 9 is the certificate-rotation paragraph. It matches the question nearly precisely, so the doc wins by a large margin: 0.254 towards 0.700.

More concerning the chunking methods

The essential distinction is just not how the textual content is reduce. It is what a match means.

With one vector per doc, search asks: “is that this doc, as a complete, just like the question?” A single related paragraph is diluted by all the things round it, and a doc that covers 5 matters finally ends up probably not matching any of them.

With one vector per chunk, search asks: “does this doc include one thing comparable?” Each chunk competes by itself deserves, and Manticore returns the doc as soon as, scored by its finest chunk.

truncate — hold it when your paperwork are brief

title textual content,
v float_vector knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='title'

This is what you have already got; chunk_strategy='truncate' is the default and also you by no means have to write down it. It’s the suitable alternative — and the quickest, and the smallest — each time your textual content genuinely suits the mannequin’s window: product titles, brief descriptions, tags, chat messages, log strains, search queries, commit topics.

How a lot suits? More than most individuals assume, and fewer than they hope. all-MiniLM-L6-v2 takes 512 tokens, roughly 380 English phrases. text-embedding-3-small takes 8,192. If your Ninety fifth-percentile doc is comfortably beneath the restrict, cease studying and hold truncate.

When it hurts: something long-form. Documentation pages, knowledge-base articles, contracts, transcripts, e mail threads, wiki pages, README recordsdata, incident postmortems.

imply — one vector, however the entire doc

v float_vector knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='title,content material'
  chunk_strategy='imply'

Manticore splits the doc, embeds each chunk, and averages the chunk vectors right into a single normalized vector. Storage and search value are equivalent to truncate — one vector per doc, one HNSW node — however nothing is thrown away.

Use it when:

• You need the tail to rely however cannot afford extra vectors — a really giant corpus the place index RAM is the binding constraint.
• The column is a plain float_vector and you’ll’t change the sort (for instance you are including the column to an current desk with ALTER, which multi-vector methods do not assist).
• Your paperwork are about one factor, simply lengthy. A single product’s full description, one recipe, one job posting.

Do not use it when a doc covers a number of unrelated matters. Averaging a authorized contract’s indemnity clause with its fee phrases produces a vector that sits between them and is near neither. In our benchmark under, imply recovered a few third of the hole that chunking closes — an actual enchancment, and clearly not the identical factor.

mounted — predictable, least expensive to cause about

v float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='content material'
  chunk_strategy='mounted' max_tokens='256' overlap_tokens='32'

Cut each max_tokens tokens, it doesn’t matter what the textual content is doing at that time. Chunk rely is a straight operate of doc size, so index measurement is predictable earlier than you load something.

Use it when the textual content has no dependable construction to take advantage of: OCR output, scraped HTML that misplaced its paragraphs, machine transcripts with out punctuation, log dumps, minified content material. Also a superb default once you merely need the most affordable factor that stops truncation.

The value: a boundary can land mid-sentence, and a piece that begins in the course of a thought embeds badly. That is strictly what overlap_tokens is for — see under.

recursive — the perfect basic default for prose

v float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='title,content material'
  chunk_strategy='recursive' max_tokens='256' overlap_tokens='32'

Same token finances as mounted, however every reduce is pulled again to the closest pure boundary: a clean line first, then a line break, then a sentence finish, then an area. A piece stops the place the textual content stops, not the place the counter runs out. The boundary isn’t dragged again previous the midpoint of the chunk, so you do not get a stream of tiny fragments.

If you will have used LangChain’s RecursiveCharacterTextSplitter, this is identical thought, besides it runs contained in the database on the mannequin’s actual tokens as an alternative of characters, and there may be nothing to put in.

Use it for: Markdown and HTML documentation, wiki pages, data bases, weblog posts, README recordsdata, structured reviews — something written by a human in paragraphs. This scored highest on deep content material in our benchmark.

sentence — when a piece have to be an entire thought

v float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='content material'
  chunk_strategy='sentence' max_tokens='256' overlap_tokens='32'

Detects sentence boundaries with Unicode UAX #29
, then greedily packs entire sentences till the token finances is reached. A piece by no means begins or ends mid-sentence. A single sentence longer than the finances is cut up by the token window, as a final resort.

Use it for: assist tickets and e mail threads, chat and assembly transcripts, authorized and coverage textual content, information, buyer opinions, medical and scientific abstracts — something the place a fraction of a sentence adjustments or destroys the which means. It can also be the technique to select when chunks can be fed to an LLM afterwards, as a result of a piece that ends mid-clause reads badly in a immediate.

sentence is a bit more conservative than recursive: it produced fewer, cleaner chunks in our checks and scored about the identical on recall@5.

The three knobs

chunk_strategy  = truncate | imply | mounted | recursive | sentence
max_tokens      = chunk measurement in tokens; 0 (default) = the mannequin's personal restrict
overlap_tokens  = tokens shared between consecutive chunks; wants a non-zero max_tokens
max_chunks      = ceiling on vectors per doc; 0 (default) = limitless

max_tokens is capped at what the mannequin can truly settle for — ask for 4,096 on a 512-token mannequin and you continue to get 512, not an error. Smaller chunks imply sharper matches and extra vectors; bigger chunks imply extra context per vector and fewer of them. For English prose, 128–512 covers nearly each use case; we used 256 all through the benchmark.

overlap_tokens repeats the tail of every chunk on the head of the subsequent, so a sentence that straddles a boundary nonetheless seems intact someplace. 10–20% of max_tokens is the same old setting. Manticore ensures ahead progress: mounted and recursive cap the overlap at half the chunk measurement, and sentence re-seeds the subsequent chunk with at most overlap_tokens price of trailing entire sentences whereas all the time advancing by not less than one sentence. It requires an specific non-zero max_tokens — overlap towards “regardless of the mannequin’s restrict occurs to be” is not a significant setting, so Manticore rejects it.

max_chunks limits the influence of unusually giant paperwork. Without it, a 400-page PDF pasted into one row turns into 1000’s of HNSW nodes. With it, Manticore merges the overflow into the final saved chunk, then truncates it to the mannequin’s window when embedding:

-- a ~600-token doc, chunked at 64 tokens
chunk_strategy='mounted' max_tokens='64'                  -- 22 vectors
chunk_strategy='mounted' max_tokens='64' max_chunks='3'   --  3 vectors

Use it as a guard rail towards outliers, not as a solution to save reminiscence throughout the board.

What search appears to be like like

Nothing about your question adjustments from earlier than. There isn’t any chunk desk, no nested area, no be a part of, no GROUP BY. Here is the entire instance:

DROP TABLE IF EXISTS notes;
CREATE TABLE notes (
  title textual content,
  physique textual content,
  chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,physique'
    chunk_strategy='sentence' max_tokens='32'
);

INSERT INTO notes (id, title, physique) VALUES
  (1, 'Certificate rotation',
   'The replication certificates is just not the one the HTTP API makes use of. Generate the brand new key on the node being rotated and signal it with the cluster authority. Update the paths and reload, one node at a time, confirming each peer reviews as synced earlier than you progress on.'),
  (2, 'Disk stress',
   'When a knowledge listing crosses eighty % the merge scheduler stops compacting and the node begins refusing writes. Free house first, then set off a handbook OPTIMIZE. Adding a disk with out draining the queue solely postpones the issue.'),
  (3, 'Slow queries',
   'Turn the question go surfing earlier than guessing. Most investigations finish at a single question no one knew was being issued, often one which kinds on an unindexed attribute over the entire desk.');

SELECT id, title, knn_dist() FROM notes
WHERE knn(chunks, 3, 'how do I substitute an expiring certificates on each node');

+------+----------------------+------------+
| id   | title                | knn_dist() |
+------+----------------------+------------+
|    1 | Certificate rotation | 0.51039070 |
|    2 | Disk stress        | 0.91703475 |
|    3 | Slow queries         | 1.02606630 |
+------+----------------------+------------+

max_tokens="32" is small on goal right here, in order that these brief notes truly cut up and you’ll see the multi-vector behaviour on a toy dataset. LENGTH() on the vector column reveals how every doc was divided:

SELECT id, title, LENGTH(chunks) AS n FROM notes ORDER BY n DESC LIMIT 5;

+------+----------------------+------+
| id   | title                | n    |
+------+----------------------+------+
|    1 | Certificate rotation |    2 |
|    2 | Disk stress        |    2 |
|    3 | Slow queries         |    2 |
+------+----------------------+------+

Six vectors, three rows again. Search follows the principles described in Multiple vectors per document
:

• A doc matches if any of its vectors is close to the question vector.
• Manticore returns every match precisely as soon as. knn_dist() is the gap to its closest chunk.
• ok counts paperwork, not vectors. knn(chunks, 3, ...) means three paperwork.
• A doc with no vectors isn’t returned.

The similar question over HTTP:

POST /search
{
  "desk": "notes",
  "knn": {
    "area": "chunks",
    "question": "how do I substitute an expiring certificates on each node",
    "ok": 3
  },
  "_source": ["title"]
}

...
{
  "_id": 1,
  "_score": 1,
  "_knn_dist": 0.51039070,
  "_source": { "title": "Certificate rotation" }
}
...

Everything else on the KNN web page retains working as earlier than: filtering, prefilter and postfilter methods, quantization
, early termination
, and rescoring.

Does it truly assist? Numbers on our personal handbook

We examined the characteristic on the Manticore English handbook — 189 pages and about 298,000 phrases, starting from a two-paragraph observe to a 39,000-word changelog.

The question set is generated mechanically, not hand-picked. For each web page we took its part headings, saved solely headings which might be distinctive throughout the entire handbook, and cut up them in two:

• Deep-content queries (419) — headings that seem after the primary ~1,200 characters of their web page. That is a intentionally conservative line: the mannequin’s window is 512 tokens, roughly 2,000 characters, so a couple of of those nonetheless level at textual content truncate can partly see. The hole under is due to this fact an understatement, not an exaggeration.
• Head-content queries (88) — headings inside the primary ~1,200 characters. The management group: content material truncate can already see.

A question is a success if KNN returns the web page the heading got here from, throughout the high ok. Model: Xenova/all-MiniLM-L6-v2 (384 dims, 512-token window) working on Manticore’s ONNX backend
. Hardware: 32 threads. max_tokens="256", overlap_tokens="32" for the multi-vector methods. Quality numbers are deterministic for a given index; the timings are a single run per technique on an in any other case idle field.

Deep content material — what chunking is for

Chunking turns a coin flip right into a working search. recall@5 goes from 55.1% to 83.3%, and the rank of the suitable reply improves simply as a lot — MRR 0.44 → 0.70. Of the queries truncate couldn’t reply within the high 5 in any respect, recursive recovers roughly two thirds.

imply lands the place you’d count on: it recovers a few third of the hole at no cost, as a result of it prices precisely nothing additional to retailer or search.

Head content material — the management group

For completeness, the management group is price studying rigorously. For content material that the mannequin may already see, truncate continues to be essentially the most exact at rank 1 — 65.9% towards 58.0% for recursive. An entire-document vector carries the web page’s total subject, and when the question is concerning the web page’s opening topic, that context helps.

By rank 5 the distinction is gone: recursive matches truncate precisely at 86.4%. So the commerce is a couple of factors of top-1 precision on content material close to the start, in trade for +28 factors of recall on all the things else. For a documentation search, a assist middle, or any RAG retriever that feeds 5–10 passages to an LLM, that’s not a detailed name.

Cost

• Index RAM: 4.2 MB → 11.7 MB, about 2.5×, for ~25× as many vectors. Vectors are solely a part of what an RT desk shops. The HNSW graph over these vectors additionally takes longer to construct throughout chunk saves and OPTIMIZE, although Manticore builds it across all your cores
  .
• Data load: 21 s → 86 s for 189 paperwork. Chunking means embedding the entire corpus as an alternative of the primary 380 phrases of every doc, and the time scales with it. This is embedding value, not chunking value — the splitting itself is just not measurable subsequent to inference.
• Query response time: 6.3 ms → 8.5 ms at p50. HNSW handles 4,664 vectors about as simply as 189 — see 2-pass HNSW, batched distances and AVX-512
  for what carries that.

If you utilize a paid embedding API, learn that ingest quantity as a invoice: chunking sends your entire corpus to the mannequin as an alternative of the pinnacle of every doc, and also you pay for each token of it. Local ONNX fashions haven’t any per-token value, which is a big a part of why we made them fast
.

Recommendations for selecting a chunking technique

Overlap is intentionally absent from most of these: our sweep under couldn’t measure a profit from it on structured prose, and it prices vectors. Add it when a thought routinely straddles a boundary — unstructured transcripts, OCR, lengthy narrative with out paragraph breaks.

How massive ought to a piece be?

Chunk measurement is the setting that really impacts your outcomes. The commerce is direct: a smaller chunk is a sharper match on one thought, a bigger chunk carries extra context however dilutes every thought inside it. A paragraph buried in an extended doc solely turns into findable as soon as the chunk measurement is sufficiently small to offer it a vector of its personal.

We ran one other take a look at: recursive on the identical 189-page handbook, three chunk sizes × three overlap settings, and the identical 419 deep queries. Two traits stand out: high quality rises as chunks get smaller, whereas overlap provides value with out enhancing high quality a lot.

Note that the Y axis begins at 78%, not zero — the entire unfold is about six factors, so a zero-based axis would flatten it right into a straight line. The numbers behind the chart:

What we see:

Smaller chunks win, persistently. Going from 512 to 128 tokens buys about 5 factors of recall@5 (80.2% → 85.2%) and a big leap in rating high quality (MRR 0.655 → 0.718). It prices 4× the vectors and a pair of.5× the index RAM. Below 128 the chunks cease containing a complete thought, so this isn’t a slope you trip perpetually — however on lengthy technical prose, 128–256 beat 512 each time.

Overlap did primarily nothing for high quality, and was not free. At 128 tokens, going from no overlap to 25% overlap moved recall@5 from 85.2% to 85.2% whereas including 41% extra vectors and 5 MB of RAM. The sample holds at each measurement: the unfold throughout overlap settings (±1.5 factors) is throughout the noise of a 419-query set, whereas the associated fee is just not. This strains up with Chroma’s chunking evaluation
, the place plain recursive splitting at 200 tokens with no overlap scored 88.1% recall — inside a couple of factors of an LLM-driven splitter at 91.9% — and it’s the reverse of the “all the time use 10–20% overlap” recommendation you’ll learn in most RAG guides.

The trustworthy caveat: that is one corpus, one mannequin, and queries that appear to be part headings. Overlap earns its hold when a single truth routinely straddles a boundary — lengthy unbroken narrative, transcripts with out construction — and recursive already snaps cuts to paragraph and sentence boundaries, which does a lot of the identical work. So deal with “begin at 128–256 with no overlap, add overlap provided that you’ll be able to measure it serving to” because the default, and examine it by yourself knowledge with the recipe under.

Comparing settings

You wouldn’t have to guess, and you do not want two tables. A desk can carry a number of model-backed vector columns, every with its personal technique, all stuffed from the identical fields on the identical INSERT:

CREATE TABLE ab (
  title textual content,
  physique textual content,
  sent_256 float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,physique'
    chunk_strategy='sentence' max_tokens='256' overlap_tokens='32',
  rec_128 float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,physique'
    chunk_strategy='recursive' max_tokens='128' overlap_tokens='16'
);

Load your corpus as soon as, then run the identical question towards every column and evaluate. For instance:

SELECT id, LENGTH(sent_256) AS sent_chunks, LENGTH(rec_128) AS rec_chunks FROM ab;
SELECT id, knn_dist() FROM ab WHERE knn(sent_256, 5, 'how do I rotate the replication certificates');
SELECT id, knn_dist() FROM ab WHERE knn(rec_128,  5, 'how do I rotate the replication certificates');

On a brief runbook whose certificates part sits on the finish, sentence/256 suits the entire doc in a single chunk and solutions at distance 0.515; recursive/128 splits it in two, isolates the certificates paragraph, and solutions at 0.310. Same row, similar mannequin, similar question — solely the chunk measurement differs.

Build a dataset of actual queries with solutions you belief — even 50 is sufficient — and evaluate recall@5 throughout two or three columns, precisely as we did on the handbook above. Then drop the dropping column with ALTER TABLE ... DROP COLUMN and hold the winner.

Recipes

Documentation and assist middle search. Long Markdown pages, customers asking questions in their very own phrases. Chunk on construction and search throughout it:

CREATE TABLE docs (
  url string,
  title textual content,
  physique textual content,
  chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,physique'
    chunk_strategy='recursive' max_tokens='192'
);

Note from='title,physique': the fields are joined earlier than chunking, so the web page title lands within the first chunk and offers it context. For a labored end-to-end instance of this form, see Vector search on GitHub
.

Support tickets and e mail threads. A thread is a sequence of full messages; slicing one mid-sentence loses the very fact you want. Keep the chunk rely bounded, as a result of threads haven’t any pure size restrict:

CREATE TABLE tickets (
  ticket_id bigint,
  buyer string,
  standing string,
  thread textual content,
  chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='thread'
    chunk_strategy='sentence' max_tokens='256' overlap_tokens='32' max_chunks='64'
);

SELECT ticket_id, knn_dist() FROM tickets
WHERE knn(chunks, 10, 'buyer was charged twice after upgrading')
  AND standing = 'closed';

Filtering works precisely because it does for a single-vector column.

Contracts and coverage paperwork. Clause-level retrieval is the whole level — no one desires “the contract” again, they need the indemnity clause. Smaller chunks, beneficiant overlap:

chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='physique'
  chunk_strategy='sentence' max_tokens='128' overlap_tokens='32'

Product catalog with lengthy descriptions. One product is one subject, and catalogs are giant, so pay nothing additional:

embedding float_vector knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='identify,description'
  chunk_strategy='imply'

RAG: retrieval for an LLM. Whatever you retrieve will get pasted right into a immediate, so chunks ought to learn as prose — that is the retrieval half of conversational search
. Larger chunks, sentence boundaries, and ask for extra of them:

chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='title,physique'
  chunk_strategy='sentence' max_tokens='512' overlap_tokens='64'

Adding chunking to a desk you have already got. Multi-vector columns cannot be added by ALTER — current rows haven’t any vectors and there isn’t any solution to backfill them but. A single-vector technique can:

ALTER TABLE docs ADD COLUMN v2 float_vector knn_type='hnsw' hnsw_similarity='cosine'
  model_name='Xenova/all-MiniLM-L6-v2' from='title,physique' chunk_strategy='imply';

ALTER TABLE docs REBUILD EMBEDDINGS v2;

For a multi-vector column, create the brand new desk with the column in place and reindex into it.

How different engines deal with this

Every vector engine now generates embeddings for you. Far fewer will cut up your doc earlier than doing it — and of these, most make you assemble it out of pipeline levels.

Every cell within the chunking column hyperlinks to a supply. A “sure” hyperlinks to the characteristic’s personal documentation. An “app-side” hyperlinks to that vendor’s personal steerage on chunking in your utility — which is what they publish as an alternative of an in-engine possibility. If we missed a characteristic, or one has shipped since, tell us
and we’ll repair it.

Two issues stand out.

Chunking in-engine continues to be uncommon. Milvus, Qdrant, Weaviate, Pinecone, MongoDB Atlas, Typesense, Meilisearch and — because the 9.8 LLM module — Apache Solr will all run the embedding mannequin for you, and each one in all them will fortunately truncate your 4,000-word doc with out saying so. The splitting is your drawback, in your utility, in a language and a library that has no thought what tokenizer the mannequin makes use of.

Where chunking exists, the plumbing often leaks. OpenSearch will get you there with a text_chunking processor feeding a text_embedding processor writing right into a nested area, queried with a nested question and a rating mode. Azure AI Search desires a skillset with a Split talent, an embedding talent and index projections — and returns one end result row per chunk, so grouping again to paperwork is on you. pgai Vectorizer writes chunks to a second desk, so each question is a be a part of plus a DISTINCT ON. Elasticsearch’s semantic_text is genuinely near Manticore’s mannequin: chunking settings on the inference endpoint, chunks hidden inside the sphere, one hit per doc.

Manticore does the identical factor with much less floor space: the technique is an possibility on the column, the chunks are the column’s worth, and search returns paperwork. If you might be weighing the entire stack relatively than this one characteristic, we have now written up the comparability with Elasticsearch
and with Turbopuffer
too.

What chunking doesn’t repair

Chunking solves one drawback properly — a doc longer than the mannequin’s window is not half-invisible. It doesn’t make retrieval excellent, and two identified gaps are price naming.

A piece doesn’t know the place it got here from. Split a doc and also you get a paragraph that claims “do one node at a time and ensure each peer reviews as synced” with no indication of what is being rotated, or which product it belongs to. Anthropic’s contextual retrieval
work put numbers on this: prepending a brief, chunk-specific description of the encircling doc earlier than embedding reduce top-20 retrieval failures by 35%, and by 49% mixed with a contextual BM25 index.

Manticore doesn’t do that for you. FROM joins its fields with an area earlier than chunking, so itemizing title first places the title on the head of the textual content that will get cut up — which suggests it lands within the first chunk and solely that one. Every chunk after it’s by itself:

-- 'title' leads, so its phrases are in chunk 1; chunks 2..N by no means see them
from='title,physique'

If you want each chunk to hold context, you need to construct it into the saved textual content your self earlier than inserting — for instance by repeating a brief heading firstly of every part of physique. There isn’t any per-chunk prefix possibility at this time.

Chunk boundaries are determined earlier than the mannequin sees the textual content. Manticore splits, then embeds each bit independently — the usual method, and what each engine with in-engine chunking within the desk above does. An various known as late chunking
inverts it: run a long-context mannequin over the entire doc first, then pool the token embeddings into chunks, so every chunk vector carries context from the remainder of the doc. It wants a long-context mannequin and extra compute per doc, and Manticore doesn’t do it at this time. If your paperwork rely closely on cross-paragraph context, it’s price understanding the choice exists.

Neither hole adjustments the essential end result: for lengthy paperwork, chunked retrieval beats truncated retrieval by a large margin, and reaching it means including chunk_strategy to at least one column.

Limits and gotchas

max_chunks discards textual content. Manticore merges overflow into the final saved chunk, then truncates it to the mannequin’s window. Nothing warns you. It’s a guard rail for outliers, not a solution to save reminiscence throughout the board.

Remote fashions chunk by bytes, not tokens. OpenAI, Voyage and Jina haven’t any native tokenizer, so Manticore falls again to a intentionally conservative estimate of 3 bytes per token — a piece lands beneath the supplier’s cap relatively than over it. In observe max_tokens="N" turns into an N × 3-byte window. We measured it towards a stub endpoint with a 3,599-byte doc and the mounted technique:

English prose runs nearer to 4 bytes per token, so on a distant mannequin you get chunks roughly 1 / 4 smaller than the quantity you requested for — set max_tokens about 30% larger than you’d for a neighborhood mannequin to land in the identical place. If actual boundaries matter, use a neighborhood mannequin, the place splitting is finished on the mannequin’s actual tokens.

Multi-vector columns cannot be added with ALTER. ALTER TABLE ... ADD COLUMN and ALTER TABLE ... REBUILD EMBEDDINGS on a model-backed float_vector_array are rejected. Recreate the desk as an alternative. Both work usually on a float_vector, together with with imply.

Chunking applies to auto embeddings solely. Vectors you insert your self are saved precisely as given — Manticore by no means re-cuts knowledge you equipped. chunk_strategy with out model_name is a DDL error, on goal.

embeddings is a reserved phrase. EMBEDDINGS is a DDL key phrase (ALTER TABLE ... REBUILD EMBEDDINGS), so a column actually named embeddings is a syntax error except escaped. Use escaping should you want that identify.

Queries aren’t chunked. A question is embedded entire, as a single vector. That is what you need: chunking exists to make an extended doc findable, to not cut up a fifteen-word query.

The DDL tells you when a mixture is incorrect, at CREATE TABLE time relatively than on the first insert:

mysql> CREATE TABLE t (title textual content, v float_vector ... chunk_strategy='sentence');
ERROR 1064: chunk_strategy='sentence' produces a number of vectors per doc
            and requires a float_vector_array attribute

mysql> ... chunk_strategy='mounted' overlap_tokens='32');
ERROR 1064: overlap_tokens requires an specific non-zero max_tokens

mysql> ... chunk_strategy='paragraph');
ERROR 1064: unknown chunk_strategy 'paragraph'; anticipated truncate, imply, mounted,
            recursive or sentence

mysql> ... chunk_strategy='truncate' max_tokens='128');
ERROR 1064: chunk_strategy='truncate' ignores max_tokens, overlap_tokens and max_chunks

Try it

The shortest path to a working chunked semantic search:

DROP TABLE IF EXISTS docs;

CREATE TABLE docs (
  title textual content,
  content material textual content,
  chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine'
    model_name='Xenova/all-MiniLM-L6-v2' from='title,content material'
    chunk_strategy='recursive' max_tokens='256' overlap_tokens='32'
);

INSERT INTO docs (id, title, content material) VALUES
  (1, 'Backup and restore runbook', 'Nightly backups run at 02:00 UTC ... ');

SELECT id, title, knn_dist() FROM docs
WHERE knn(chunks, 5, 'how do I rotate the replication certificates');

No mannequin to obtain by hand, no splitter to select, no pipeline to take care of. One column possibility, and the elements of your paperwork that was invisible begin exhibiting up in outcomes.

Full reference: Chunking strategies
and Multiple vectors per document
within the handbook. Questions and bug reviews on GitHub
.