Two Towers, One Index: How Retrieval Models and ANN Search Are Co-Designed

Picture the job a search box or a feed has to do. A user shows up. Somewhere in a catalog of a hundred million items is the handful they’d love. You have about 50 milliseconds to find them — and you pay that bill on every single request. Scoring all hundred million with a good model takes minutes. So the real question was never “what’s the most accurate model?” but “what shape of model lets you avoid running it a hundred million times per query?”

The answer has two parts, so tightly co-designed you can’t understand one without the other. The first is a model — the two-tower model (also called a dual encoder or bi-encoder). The second is a data structure — the approximate nearest neighbor (ANN) index. They look like two separate tools you happen to use together. They aren’t — they’re one system. Here it is, end to end — everything below is a detail of this one picture:

01The bargain: why a split score is the whole point

Here is the one idea everything else hangs on. A two-tower model scores how well a query q matches an item i as the dot product of two vectors, one per side — multiply them slot by slot, add up the results, and you get a single number for how aligned they are:

That separation is the sacrifice and the superpower. Because the item vector g(i) doesn’t depend on the query, you build it once, offline, for the whole catalog, and stash it in an index. At request time you run only the small query tower, then solve a pure geometry problem: which stored vectors are closest to this one? That question has a fast answer.

Now compare the most accurate thing you could build instead: a cross-encoder, which reads the query and item together and lets every layer compare them directly. It catches subtleties a dot product can’t. But its score depends on the pair, so nothing can be precomputed: every query means a fresh pass over every candidate.

So the cross-encoder doesn’t disappear — it moves. Real systems run in stages: a cheap two-tower model pulls a few hundred candidates out of a billion, then a heavy model reorders that shortlist. Two-tower is the first stage; the cross-encoder is what you can finally afford in the second. And fix one framing error: two-tower replaced candidate generation, not final ranking. Whatever the retriever misses, no ranker can rescue — so how much of the good stuff retrieval catches (its recall) sets the ceiling on everything.

02Where it came from, and why it won

The design has two ancestors. One is the Siamese network (Bromley, Guyon & LeCun, 1993) — twin networks that encode two inputs and compare the results, first used to verify signatures. The other is classic information retrieval. Their modern child is DSSM (Huang et al., Microsoft, 2013): push a query and a document through separate networks into the same space, score by similarity, and train on user clicks. It introduced two tricks that still echo: a hashing scheme shrinking a 500K-word vocabulary to ~30K inputs, and training against one clicked result plus a few random “wrong” ones (the seed of today’s in-batch negatives). One detail people miss: DSSM predates transformers entirely. It’s a plain feed-forward net, not BERT.

Why did this beat the matrix factorization that ruled the Netflix-Prize era? Because matrix factorization only memorizes: one vector per user-ID and item-ID it has already seen. Add a new item and it’s stuck until you retrain. A two-tower model instead reads features off an item and places it in the space, so it handles brand-new items on day one.

03Inside the towers: a few choices that matter

The towers themselves are ordinary networks — MLPs for recommendations, BERT encoders for text. But three knobs inside punch above their weight:

• Vector size is an accuracy-vs-cost dial. Typical is 64–1024 numbers per vector. Bigger lifts accuracy but costs more index memory and time — and compression can win much of it back, so size and index choice have to be decided together, never alone.
• Normalizing the vectors (scaling each to length 1) isn’t cosmetic. It makes the dot product behave like a clean angle comparison and keeps training stable. Skip it and a vector’s length starts secretly encoding popularity — sometimes wanted, often a bug.
• Temperature is a training-time setting that controls how sharply the model separates right from wrong answers. It changes how the model learns, not the final ranking — so it matters during training and never at serving time.

04Training against a billion classes

Here’s where the index reaches back into the model. The “correct” way to train a retriever is to treat every item as a class and ask the model to pick the right one — but that means scoring all 106–109 items for every training example, which becomes impractical somewhere past a few hundred thousand classes. So you approximate it: score the true item against a small handful of “wrong” ones (negatives) instead of all of them. It exists for the same reason ANN does: you simply cannot touch every item.

The cheapest source of negatives is the batch you’re already processing. In a batch of, say, 4,000 (query, item) pairs, each query treats the other 3,999 items as its negatives — thousands of nearly free negatives, since those item vectors are already computed. That’s why two-tower training uses giant batches. But there’s a catch: popular items show up as negatives constantly, so the model unfairly learns to push them down.

The fix is the logQ correction (Yi et al., 2019): nudge each negative’s score down by how often that item naturally appears, so common items aren’t penalized for being common. The clever part is that “how often” is estimated live, as the data streams by, because the catalog never stops changing. In a real YouTube A/B test, this bias correction meaningfully lifted the engagement win versus the naive version — a notable result at that scale.

05Negatives are the model

Past a certain point, how you pick negatives matters more than the architecture. A random negative is obviously wrong, so the model learns almost nothing from it. It learns from negatives that look plausible and aren’t — “hard” negatives. The baseline trick (from DPR, 2020) was to add one strong lexical near-match per example, which alone beat keyword search by a wide margin.

And that exposes the real hazard: false negatives. Dig hard enough for plausible wrong answers and you start dredging up items that are actually right but just weren’t labeled — and training on those as “wrong” poisons the model. Half the cleverness in modern negative mining is about being aggressive without crossing that line.

06The distance trap

You have to tell the index what “close” means. There are three common choices — cosine (angle only), dot product (angle and length), and Euclidean (straight-line distance) — and on normalized vectors they all rank results the same way. So far, harmless.

The trap is subtle. If you don’t normalize and use raw dot product, “closeness” stops behaving like real distance: a single item with a very long vector can end up looking like the best match for almost any query. That breaks the shortcuts most indexes rely on to skip work, and it quietly wrecks results.

07HNSW: the default index

Now the search itself: one query vector, a hundred million stored vectors, return the closest few hundred. Checking all of them is too slow for a 50 ms budget. The key insight behind ANN: you almost never need the exact closest items — return 98 of the true top 100 plus two near-misses and nobody notices, because the reranker sorts it out. You trade a little accuracy for a huge speedup.

The most popular index is HNSW (Malkov & Yashunin, 2016). Think of it as a navigable graph of vectors, with a few long-range “express” links layered on top of dense local ones. To search, you start at the top, ride the express links to get roughly close, then drop down for the last mile. It powers most of the vector databases you’ve heard of — FAISS, Qdrant, Weaviate, Milvus, pgvector, Elasticsearch and more.

It has exactly three knobs, and only one of them is something you tune live:

The price of all that speed is memory: the whole graph and all the vectors sit in RAM. That caps a single machine at roughly 100–200M vectors before you need disk-based methods or splitting across servers. HNSW hits 95–99% accuracy at sub-millisecond speeds, but squeezing out the last percent or two gets very expensive, and deleting items is awkward enough that heavy churn forces periodic rebuilds.

08IVF, compression, and the rest of the index zoo

When memory is the binding constraint, you switch from graphs to group-and-compress. Both ideas come from one 2011 paper (Jégou et al.), tested on a billion vectors.

IVF sorts vectors into clusters up front; at query time you only scan the few clusters nearest the query instead of everything. The dial here, nprobe, sets how many clusters to check — more clusters, more accuracy, more time. Its weakness is the boundary problem: a great match sitting just across a cluster line can be missed entirely.

Product Quantization (PQ) is the compressor. Instead of storing a full vector, it chops it into pieces and replaces each piece with a short code — turning a 3,000-byte vector into about 16 bytes, a ~190× shrink, so a billion vectors actually fit. Combine the two and you get IVF-PQ, the standard memory-saver.

Layered on top, the rest of the zoo is best read as a three-way trade between accuracy, speed, and memory:

09Buying interaction back

The single dot product threw away word-by-word matching. You can buy it back, at rising cost, and the production answer is to line the options up as a cascade rather than pick one:

ColBERT (2020) is the influential middle ground: it keeps a vector for every word and scores by matching each query word to its best partner in the document. That recovers much of a cross-encoder’s nuance while still letting you index documents ahead of time. Its cost isn’t compute — it’s storage, since one vector per word is a lot of vectors. Follow-ups (ColBERTv2, PLAID) mostly exist to shrink and speed up that index, and the idea has since jumped to images, too.

10Search, RAG, and a reality check

The same architecture powers text search and the retrieval step in RAG. The lineage runs through DPR (2020), and Sentence-BERT (2019) made the speed case stark: comparing 10,000 sentences pairwise drops from ~65 hours with a cross-encoder to ~5 seconds with a bi-encoder. Today’s embedding models — E5, BGE, GTE, OpenAI’s text-embedding-3, Cohere, Voyage — cluster closely on the public leaderboards.

But here’s the humbling part. The BEIR benchmark showed that on data they weren’t trained on, these dense models often generalize worse than plain old keyword search (BM25), which remains shockingly hard to beat out of the box.

11Running it in production

This is where most of the real work lives, and almost none of it is in the papers:

• You can’t just merge two indexes. HNSW graphs don’t glue together, so scaling out means splitting data across machines, asking each for its local top results, and merging — with extra fetched from each shard to cover the seams.
• Deletes rot the index quietly. Removing items in place leaves dangling links and dead ends that slowly erode accuracy, so production systems mark items deleted and clean up in a background pass; in batch-built indexes, a new write may not be searchable until the next indexing cycle.
• Filtering is the hardest part. “Vectors like this, but only in-stock and under $50” fights the index: filter first and you can break the graph; filter after and you may not have enough left. Modern engines (ACORN, Filtered-DiskANN) weave the filter into the search and pick a strategy per query.
• Hybrid search usually wins. Combining keyword search and vector search — typically by blending their rankings, not their raw scores — beats either alone, which is why most vector databases ship it by default.
• Measure accuracy in production. It drifts as you add, delete, and re-tune. The standard move is to occasionally run the slow exact search in the background and check how much the fast index agrees.

12Measuring it honestly

The standard benchmark (ANN-Benchmarks) makes one thing clear: there’s no single “best” index, only a trade-off curve between accuracy and speed. So “index A is faster than B” is meaningless unless you hold accuracy fixed and then compare speed. Accuracy, speed, and memory form a triangle — push one and another gives.

A couple of honest cautions: benchmarks usually measure a single query at a time, but production runs many at once, where memory pressure can flip the winner; and what matters is usually the worst-case latency (p99), not the average, because graph search has long tails.

13The full funnel

Every large system arranges the same pipeline: a cheap vector search narrows millions-to-billions of items down to a few hundred in well under 50 ms, then a feature-rich ranker reorders that shortlist. The famous write-ups all teach a piece of it:

Which lets us retire three myths in one breath. “Plug in a vector database and you’re done” — no; the hard parts are negatives, bias correction, and co-tuning retrieval with the ranker. “The index finds the true nearest neighbors” — no; it’s approximate on purpose, and teams dial the accuracy knob to hit a latency budget. “More candidates always help” — no; past what the ranker can reorder, extra candidates only add latency and noise.

14Same machine, different words: recommendations

Everything so far has used search vocabulary — query, document, relevance. Swap three words and you have a modern recommender system. This isn’t a loose analogy; it’s a literal substitution. In a feed, the two-tower model is the candidate-generation stage, and ANN over item embeddings is what fetches the few hundred things a heavy ranker then orders.

The scoring is identical — a dot product of a user vector and an item vector — so the same ANN index (ScaNN, HNSW, FAISS) precomputes every item embedding and returns candidates online. When you flip the demo near the top to Recommendations, nothing in the math changes; only the labels do. That’s the whole point.

What genuinely differs lives on the user side — and it’s where most of the research goes:

• The query has a memory. A search query is a static string; a user is a history. The user tower is often a Transformer over the sequence of recent actions (SASRec, BERT4Rec), so it captures short-term intent, not a frozen profile.
• The labels are implicit and biased. You don’t get clean relevance judgments — you get clicks, which are positive-only and skewed by whatever was already shown. A non-click isn’t a reliable “no.”
• Popularity bias is severe. Engagement follows a brutal power law, so the logQ correction from section 04 isn’t optional here — without it the model just recommends whatever is already popular.
• One vector may not be enough. A user has several interests at once, and a single embedding blurs them. Pinterest’s PinnerSage issues several user vectors and unions the results.
• Freshness is constant. New items and users arrive every second, so content-based item towers (not just learned IDs) let a brand-new item get a usable embedding immediately.

The exemplars are the same names from the funnel above — YouTube’s corrected two-tower retriever, Pinterest’s PinSage and PinnerSage, Meta’s Facebook-Search EBR, Alibaba’s Taobao MGDSPR — all running the identical retrieve-then-rank pattern over catalogs of millions to billions. (The main alternative, Alibaba’s tree-based TDM, drops the dot-product constraint to allow richer interaction at retrieval time — a reminder that two-tower + ANN is the dominant answer, not the only one.)

How this was made: the claims, papers, and figures here were assembled by a multi-agent research pass — 16 agents covering distinct facets (the model, training, the index families, real production systems), then a fact-checking stage that verified the numbers and citations and corrected two attribution errors, plus a focused pass on the recommendation-systems mapping and a clarity edit. The interactive demo above computes its exact ground truth every frame, so its recall and “missed” markers are real — but 2-D radically understates real high-dimensional difficulty. Figures tied to a specific paper come from that paper; “typical” ranges are practitioner conventions, not guarantees.