Your Chunker Is Deleting Facts. The Overlap You Need Is Exact.

01The failure that looks like a ranking problem

A user asks something your documentation plainly answers. The answer does not come back. You check the embedding model, try a different one, add a reranker, raise k from 5 to 20. Nothing helps, and the reason nothing helps is that the sentence containing the answer was cut in half three steps before any of that ran.

Chunking is usually described as a performance knob — smaller chunks are more precise, bigger ones carry more context. That framing hides the real property: a chunk is the atomic unit of what can be returned. If the span of text that answers a question is not wholly inside some single chunk, no amount of ranking skill will produce it, because the object you want was never one of the candidates.

This is worth separating from the usual retrieval failures, because it has a different cure. A fact that ranks 40th is a ranking problem. A fact that is split across two chunks is a representation problem, and it is invisible to every metric computed over your returned results.

02The arithmetic

Three numbers, all in tokens:

Symbol	Meaning
C	chunk size
V	overlap between consecutive chunks
L	length of the span that actually answers the question — the fact

Chunks start every S = C − V tokens — that is the stride — and chunk i covers [iS, iS + C). A fact occupying [x, x + L) is retrievable if some chunk contains all of it. Chunk i contains it when iS ≤ x and x + L ≤ iS + C, which rearranges to

x ∈ [ iS , iS + C − L ]

So each chunk rescues an interval of start positions of length C − L, and those intervals repeat every S tokens. Over one period, the share of start positions that are covered is (C − L)/S, capped at 1. Writing S out:

P(retrievable) = min ( 1, (C − L) / (C − V) ) for L ≤ C

Four consequences fall straight out, and the third is the one worth remembering.

Set	And you get
V = 0 (no overlap)	P = 1 − L/C. Loss is linear in how long your facts are.
L > C	P = 0. No overlap setting ever helps; the fact cannot fit in a chunk.
V ≥ L	P = 1 exactly. Every boundary is covered by its neighbour.
L → 0	P = 1. Single-token facts are never split, which is why keyword search never had this problem.

The third row is a clean rule and it is not the rule anyone uses. The overlap has to be at least as long as the longest fact you need to retrieve whole. It is an absolute length, not a percentage — so the ubiquitous “10% overlap” is really an unexamined claim that your facts are shorter than a tenth of a chunk. At C = 512, that is a bet that nothing you need is longer than 51 tokens, which is about two sentences.

03What it costs when you get it wrong

512-token chunks, no overlap — a common default:

Fact length	Retrievable	Split and lost
32 tokens (a sentence)	93.8%	6.3%
64 tokens	87.5%	12.5%
128 tokens (a short paragraph)	75.0%	25.0%
200 tokens	60.9%	39.1%
256 tokens (half the chunk)	50.0%	50.0%
384 tokens	25.0%	75.0%

A procedure, a table row with its header, a definition and its qualifying clause, a code block with its explanation — these run to a couple of hundred tokens routinely. At 200 tokens you are losing two in five of them, and the loss is invisible: the document is in the index, the embedding is fine, the chunk count looks right.

Why this never shows up in your eval

Retrieval evals are built from questions you wrote against documents you have. If you wrote the question by looking at a chunk, the answer is inside that chunk by construction, and the split facts are precisely the ones nobody wrote a question for. The failure mode selects itself out of the test set. The only way to see it is to go the other way: take the spans you need retrievable, and check whether each one sits inside a single chunk.

04The lab

The left panel is a document with its chunk windows drawn over it. The band underneath is every position a fact could start at: green where some chunk holds it whole, red where it straddles a boundary and sits in none. Start at the defaults — C = 512, V = 0, L = 256 — and the band is half red.

Drag overlap up and watch the red stripes narrow from both sides. They close completely the moment V reaches L, and the blue marker on the right-hand chart shows you hitting 100% exactly there — not asymptotically, exactly. Push past it and you buy nothing but index.

The shape of that green curve is the other thing worth noticing, and it runs against the usual intuition. Retrievability does not merely rise with overlap — it accelerates. The derivative is (C − L)/(C − V)², which grows as V grows, so the curve is convex: at C = 512 with 200-token facts, overlap around V = 192 is buying 2.56× as much coverage per token as overlap around V = 0. In most engineering trade-offs the last increment has the worst return. Here it has the best one, and the final stretch up to V = L is the most valuable overlap you will ever add, because it is what converts “mostly works” into “cannot fail”.

Overlap V	Index size	P for a 200-token fact
0	1.00×	60.9%
32	1.07×	65.0%
64	1.14×	69.6%
128	1.33×	81.3%
192	1.60×	97.5%
200	1.64×	100.0%

C = 512. Index size is chunks per document relative to no overlap, which is C/(C−V).

05The part that actually changes your config

Real corpora do not have one fact length. Take a spread with a median of 67 tokens, a 90th percentile of 174 and a 99th of 383 — short facts mostly, with a tail of long ones — and average the formula over it:

Chunk C	Overlap V	Facts retrievable	Index size
512	0	82.8%	1.00×
512	64	92.0%	1.14×
512	128	96.1%	1.33×
512	256	98.7%	2.00×
1024	0	91.4%	1.00×
1024	128	98.3%	1.14×
1024	256	99.5%	1.33×

Compare the two bolded-adjacent rows. 512 with 256 of overlap buys 98.7% for double the index. 1024 with 128 of overlap buys 98.3% for 1.14×. Essentially the same coverage, at a seventh of the extra storage.

The reason is in the cost formula. Index size is C/(C − V), which depends only on V as a fraction of C — while the coverage you need depends on V as an absolute length, because facts have absolute lengths. Doubling the chunk halves the relative price of the same protection. Overlap is cheaper in bigger chunks, and that is a reason to prefer bigger chunks that has nothing to do with context.

What this model does not tell you

It deliberately says nothing about whether bigger chunks rank worse. They generally do: one vector summarising 1024 tokens is a blurrier query target than one summarising 256, and that effect is real and works against everything above. I have not measured it here and I am not going to put a number on it that I have not measured. What this model gives you is the other half of the trade — the half that is exact, that nobody computes, and that sets a hard ceiling on recall before ranking gets a turn.

06What to do on Monday

Measure L, do not guess it. Take thirty questions your system should answer, highlight the minimal span that answers each, count the tokens. The 90th percentile of that is your V.
Check the facts you already know about. For each of those spans, ask whether it lies inside a single chunk. Any that do not are guaranteed failures and they will not appear in any recall number you currently compute.
Prefer raising C to raising V when both would work, since the same absolute protection costs proportionally less — then watch ranking quality, which is the thing that pushes back.
Chunk on structure where you can. Every boundary you place at a heading or a paragraph end is a boundary no fact straddles, which is the same effect as overlap at no index cost at all. The arithmetic here is the worst case: boundaries falling blind.

07Recap

A chunk is the atom of retrievability. A fact split across a boundary is not ranked badly — it is not a candidate.
P = min(1, (C−L)/(C−V)), which at V = 0 is just 1 − L/C.
512-token chunks with no overlap lose half of all 256-token facts and 39.1% of 200-token ones.
V ≥ L makes it exact. The overlap you need is an absolute length — your longest fact — not a percentage.
Overlap is cheaper in bigger chunks, because cost scales with V/C while protection scales with V.
Your eval cannot see this, because questions written from chunks are answerable from chunks by construction.

Every other part of a retrieval stack is a ranking problem, and ranking problems are gradual — a better model moves things up the list. The chunker is the one place where a decision made before any model runs puts a hard ceiling on what the rest of the system is allowed to find. It takes one line of arithmetic to know where that ceiling is.

Sources, and what is mine

Two Towers, One Index · The Unreachable Fifth, the other way an index quietly makes documents unfindable

There is nothing quoted in this post. The formula in §02 is derived from the definition of a sliding window, and I checked it against a brute-force simulation before writing a word of prose: lay chunks over a long document, drop 200,000 facts at uniformly random positions, count how many sit inside some single chunk. Closed form and simulation agree to 1.9×10⁻³ across eight (C, V, L) settings, and the four limiting cases — L = 0, L > C, V ≥ L, and V = 0 reducing to 1 − L/C — are exact. Every table cell comes from that same formula, and the lab evaluates it live, so the numbers in the prose are the numbers the page renders.

And it is a model. It assumes a fact is one contiguous span, that chunk boundaries fall blind rather than on structure, and that a fact's start position is uniform. Real chunkers split on paragraphs and headings, which helps for free; real answers are sometimes assembled from two distant spans, which this cannot represent at all; and real retrieval returns several chunks, so a fact split in two is occasionally recoverable when both halves come back together and the model stitches them. Each of those makes the true loss lower than the arithmetic here. The direction the model is wrong in is the safe one: it describes the worst case, and the worst case is what sets your ceiling.

The fact-length distribution in §05 is a log-normal I chose (median 67 tokens, p90 174, p99 383) because it has the right shape — mostly short, with a tail. It is an illustration, not a measurement of anybody's corpus, which is exactly why §06 tells you to measure your own.