LLM Reranker Selection: Cross-Encoder vs Bi-Encoder

Tengence Team·2026-09-06
Search & Recommend
LLM Reranker Selection: Cross-Encoder vs Bi-Encoder

Summary: Reranking is not a binary choice but a two-stage pipeline: a bi-encoder recalls candidates and a cross-encoder reorders the short list. The real variables are candidate size, latency budget and language coverage.

When teams add reranking to RAG or site search, the first instinct is to swap in a stronger model. Precision improves, P95 climbs with it, and the change is eventually rolled back. This guide separates the architectural differences, the measured trade-offs and the production path, so the choice can be reviewed rather than guessed.

Key Takeaways

  • The official pattern is a pipeline, not a replacement: Sentence-Transformers' Retrieve & Re-rank tutorial recommends a Bi-Encoder for top-100 retrieval followed by a Cross-Encoder rerank of those candidates.
  • The accuracy gap comes from joint attention: a cross-encoder lets query and document tokens interact at every layer, capturing negation, modifiers and phrase-level semantics, while a bi-encoder compresses each side into a fixed-length vector.
  • A lightweight baseline is verifiable: cross-encoder/ms-marco-MiniLM-L6-v2 reports NDCG@10 = 74.30 on TREC Deep Learning 2019 and roughly 1,800 documents/sec on a V100.
  • Cost grows linearly with candidates: a cross-encoder cannot precompute document vectors, so N candidates require N joint forward passes; a bi-encoder builds a vector index offline and has near-constant online cost.
  • Recall is the ceiling: if the correct document is not among the top-100 candidates, no reranker can recover it. Fix tokenization, synonyms and multi-route recall before tuning the reranker.
  • Production needs an operations layer: pinned positions, blocks, direct hits, result diversification and weighting should stay independent of model scores, with reports and A/B validating every adjustment so a model upgrade cannot erase business rules.

1. The Answer First: A Two-Stage Pipeline, Not a Binary Choice

Comparing the two encoders as if they occupied the same slot is the most common framing error in selection. The official Retrieve & Re-rank tutorial from Sentence-Transformers states it plainly: Cross-Encoders achieve higher performance than Bi-Encoders, but they do not scale well for large datasets. The recommended pattern is to use an efficient Bi-Encoder to retrieve the top-100 most similar documents from a large corpus, then let a Cross-Encoder compute a score for every query–document pair among those 100 hits and reorder them. The reason is a completely different cost structure. A bi-encoder computes document vectors once, offline, and stores them in an index; online it needs only one query encoding plus an approximate nearest-neighbour lookup. A cross-encoder must concatenate the query with each candidate and run a forward pass, so cost scales with candidate count. "Which one should we use" therefore becomes "at which layer should we use each": the recall layer must cover millions of documents and can only use precomputable dual encoders, while the rerank layer sees tens to a few hundred candidates and can afford joint attention. Once these two layers are separated, discussions of precision, latency and budget share a premise instead of comparing models built for different jobs.

2. Why Cross-Encoders Are More Accurate but Cannot Precompute

The accuracy difference comes down to whether information meets during encoding. A Bi-Encoder embeds the query and the document separately into two fixed-length vectors and scores them with dot product or cosine similarity; query tokens and document tokens never "see" each other inside the model, so negations, numeric comparisons, multi-word phrases and fine-grained modifiers are lost in that compression — in exchange, document vectors can be computed once and reused. A Cross-Encoder feeds query and document as one sequence into a Transformer, where every query token attends to every document token, and outputs a relevance score for that pair rather than a reusable vector. The cross-encoder/ms-marco-MiniLM-L6-v2 model card on Hugging Face is a common lightweight baseline: it reports NDCG@10 = 74.30 on the passage ranking task of TREC Deep Learning 2019 and about 1,800 documents/sec on a V100. That throughput means reranking top-100 adds only modest GPU time, but pushing candidates to top-1000 turns latency into a product problem. Another limit is easy to overlook: most local cross-encoders truncate at 512 token, so long documents must be chunked first or the reranker sees only half the context.

3. Measured Trade-Offs: Accuracy, Latency and Cost

Dimension Bi-Encoder (dual tower) Cross-Encoder (joint encoding)
Input handling Query and document encoded separately Query and document concatenated, encoded jointly
Output One reusable vector per text One score per query–document pair
Precomputable document vectors Yes, index built offline No, forward pass required online
Online cost Near-constant after indexing Grows linearly with candidate count
Semantic precision Lossy compression, weaker than joint attention Captures negation and phrase-level matching
Typical role Recall top-100 from a large corpus Rerank a short list into top-10
Length constraint Bound by embedding model context Often truncates at 512 token

This division implies three hard engineering constraints. First, candidate size converts directly into latency budget: top-50 and top-100 are close on a lightweight model, but beyond that you need batching, distillation or a hosted API to amortize cost. Second, recall quality is the ceiling — if the right document never enters the candidate set, more reranking passes cannot help, and the fix belongs in query understanding and multi-route recall rather than a bigger reranker. Third, enterprise retrieval systems rarely want model scores to rule alone. The Elasticsearch Learning To Rank documentation describes attaching the ranking model as a rescore phase: inverted-index or vector retrieval narrows the candidates, then the model and engineered features reorder them, while business rules remain controllable at the query layer. That is also why "swap in a stronger model" so often fails to answer operations-side complaints.

4. Selection Criteria: Candidate Size, Latency, Languages

Work through four questions in order. How large is the candidate set: if it stays around top-10 and precision is paramount, a cross-encoder can score everything; if it serves site search over millions of documents, two stages are mandatory. What is the latency budget: estimate rerank overhead as candidate count multiplied by single-pass cost, then align it with the overall search P95 target instead of quoting the model card's throughput. Do you need multilingual coverage and long context: cross-border sites handle several languages at once, and product descriptions or help articles easily exceed 512 token, which calls for a multilingual reranker plus chunking, or a hosted service; the Cohere Rerank documentation describes an interface with 4,096 token of context and multilingual support, suited to teams that prefer not to host GPUs. Finally, what can operations sustain: self-hosting means owning versions, batch sizes, quantization and rollback, while a hosted API means governing data egress and per-call cost. Answer these four before returning to the comparison table, and most candidates eliminate themselves — rather than picking the top score on a leaderboard and shipping it.

5. Production: Embed Reranking in the Retrieval Chain and Operations Layer

An architecture that lasts places reranking inside a chain — query understanding, multi-route recall, coarse ranking, reranking, operations intervention, reporting and A/B — instead of letting it replace any step. Query understanding handles tokenization, spelling correction, synonyms, category prediction and term weighting; recall combines inverted index, multiple routes and AI-semantic retrieval so correct documents enter the candidate set; coarse ranking compresses the set to a few hundred; reranking then scores top-100 jointly and emits top-10. Operations intervention must stay independent of model scores: pinned positions, blocks, direct hits, diversification and weighting are deterministic business rules, and coupling them to the model means one upgrade can make campaign products vanish. TENGENCE Search, an enterprise-grade site search engine, makes exactly this kind of ranking and operational intervention configurable, with reports and A/B so each adjustment can be validated; it serves on-site retrieval and conversion, not external search-engine rankings or citations in AI answers. Returning to the two-stage pattern from sbert.net: secure recall coverage first, then let reranking lift top-of-list precision, check NDCG@10 on an offline evaluation set and conversion through online A/B. Only then is reranking a sustainable engineering improvement rather than a one-off model experiment.

FAQ

Q: If recall is poor, can adding a Cross-Encoder fix it?

A: No. Reranking only changes the order of candidates; it cannot materialize a document that was never retrieved. Check tokenization, synonyms, multi-route recall and chunking first, get the correct document into top-100 and then measure rerank gain.

Q: How large should the candidate set be?

A: top-100 is a common starting point, tuned against the latency budget. Cross-encoder cost grows linearly with candidates, so moving from top-100 to top-1000 noticeably raises latency. Use an offline evaluation set to find where NDCG@10 gains flatten.

Q: Can we treat 74.30 NDCG@10 as our own target?

A: No. It is one model's score on the public TREC Deep Learning 2019 dataset, useful only as a lightweight baseline reference. Recompute on your own labelled set and query logs, then judge relative lift.

Q: Self-hosted model or hosted rerank API — which one?

A: It depends on data compliance, language coverage and operations capacity. Self-hosting enables customization and private deployment but means owning versions, quantization and GPU capacity; a hosted API removes that work and supports long context, with data egress and per-call cost to evaluate.

Q: When model scores conflict with business rules, which wins?

A: Keep operations intervention in a separate layer after reranking. Pinned positions, blocks, direct hits and weighting are deterministic rules that a model upgrade should not erase; use reports and A/B to check whether the intervention truly improves conversion, and review it regularly.

Data Sources

Related Reading

Get Started

About TENGENCE

TENGENCE is an AI-era enterprise growth engine focused on public-domain acquisition and private-domain conversion. Its core products are TENGENCE GEO (GEO + SEO dual engine), TENGENCE Search and TENGENCE Insight (an intelligent data platform). For the reranker selection discussed here, TENGENCE Search serves enterprise on-site retrieval: query understanding, inverted-index and AI-semantic multi-route recall, coarse and fine ranking models, plus configurable interventions such as pinned positions, blocks, direct hits, diversification and weighting — with reports and A/B so every ranking adjustment is measurable.