Reranking Retrieval Results

Reranking retrieval results means adding a second, more accurate pass that reorders the candidates your initial search returned—and it's one of the highest-leverage upgrades you can make to a RAG system, typically lifting precision 15–40% over embeddings alone. This guide explains why fast vector search needs a reranker, how cross-encoders deliver that accuracy, the two-stage pipeline to build, which models to choose in 2026, and the mistakes that waste the latency you spend on it.

What reranking is and why retrieval needs it

Your first-stage retriever—a vector search over embeddings—is built for speed, not precision. It encodes the query into one vector, compares it against millions of stored document vectors, and returns the nearest matches in milliseconds. That speed comes from a compromise: the query and each document are embedded separately, so the model never directly compares the words in your question against the words in a candidate. It catches general semantic similarity but misses fine distinctions—negation, which specific term matched which, multi-hop relationships.

The result is a top-k list that's usually good but rarely optimally ordered. The genuinely best passage might sit at rank 7 instead of rank 1. Since you only feed the top few chunks to the LLM, that ordering matters enormously.

Reranking fixes the ordering. A reranker takes the query and a shortlist of candidates and scores each query-document pair directly, producing a far more accurate relevance order. You retrieve broadly and cheaply, then rerank precisely on the short list. This second pass is also one of the most reliable ways of reducing hallucinations with RAG: the model can only reason over the chunks it receives, so putting the truly relevant ones at the top gives it the right facts to work from.

How rerankers work: cross-encoders vs bi-encoders

The architectural difference between retrieval and reranking is the whole story.

A bi-encoder is the architecture behind embedding models. It runs the query and the document through the encoder independently, producing two vectors, and scores them by cosine similarity. Because document vectors can be computed once and stored, bi-encoders scale to millions of items—but the query and document never interact during encoding. (Our explainer on how embeddings represent meaning as vectors covers this side in depth.)

A cross-encoder—the standard reranker architecture—does the opposite. It feeds the query and document together through a transformer as a single input, so every query token can attend to every document token. It directly models term overlap, negation, and word-level relevance, which is why it's far more accurate. The catch is cost: a cross-encoder has to run a full forward pass for every query-document pair, so you can't run it over a whole corpus. You run it over a shortlist of perhaps 20–100 candidates the first stage already narrowed down.

Pointwise vs listwise

Most rerankers are pointwise: they score each document independently, then sort by score. Listwise rerankers (such as jina-reranker-v3 and LLM-based approaches) consider the candidates together and optimize their relative order—jina-reranker-v3, for instance, processes up to 64 documents in one long context window. Listwise methods can sharpen ordering when relative ranking matters, at higher compute cost.

You'll also encounter ColBERT and late-interaction models, which precompute token-level representations as a middle ground between bi- and cross-encoders. In practice for 2026, the standard bi-encoder-plus-cross-encoder pipeline is simpler and usually matches its quality, so ColBERT remains a niche choice.

The two-stage retrieval pipeline

Here's the pattern used by most production systems. Reranking is stage two.

1. Retrieve broadly. Run vector search and pull a generous candidate set—top 50 is a common starting point, not top 5. You want high recall here: the right answer must be somewhere in the set, even if poorly ranked.
2. (Optional) Fuse with keyword search. Run BM25 keyword search in parallel and merge the two lists with Reciprocal Rank Fusion (RRF), using the standard constant k=60. This catches exact terms—product codes, names—that pure vector search smooths over.
3. Rerank precisely. Pass the query and all candidates to the cross-encoder, which scores each pair and reorders them.
4. Truncate and generate. Keep the top 5–10 reranked chunks and send only those to the LLM.

A minimal reranking step with an open-source cross-encoder:

from sentence_transformers import CrossEncoder

# Stage 1 already returned ~50 candidate chunks for the query.
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")  # multilingual, Apache 2.0

query = "How do I rotate my API keys without downtime?"
pairs = [(query, chunk.text) for chunk in candidates]

scores = reranker.predict(pairs)                 # one relevance score per pair
ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])

top_chunks = [c for c, _ in ranked[:5]]          # send these 5 to the LLM

Hosted APIs collapse this further. Cohere's Rerank, for example, takes the query and the document list and returns them reordered with relevance scores between 0 and 1, integrating into an existing search system with minimal code.

Choosing a reranker

The reranker market in 2026 splits into hosted APIs (zero infrastructure, per-call cost) and open models you self-host (no per-call cost, but you run the GPU). The right pick depends on latency budget, language needs, data residency, and whether you'd rather pay in dollars or ops.

A few practical notes. Independent benchmarks in early 2026 have put Zerank and Cohere's newest Rerank near the top on quality, with Voyage Rerank 2.5 prized for balancing quality against roughly half the latency of some competitors—though leaderboard ranks shift, so verify on your own data. For a free, dependable baseline, bge-reranker-v2-m3 is hard to beat; if a pricier model doesn't clearly outperform it on your evaluation set, the extra cost or latency isn't justified. And when you need sub-20ms CPU reranking with no API dependency, FlashRank's small quantized models are purpose-built for it.

What latency to expect

Reranking adds a real but usually acceptable cost. A small MiniLM cross-encoder scores 50 candidates in roughly 100–300ms on CPU and under 50ms on a single GPU; FlashRank can hit sub-20ms on CPU. The heavier multilingual bge-reranker-v2-m3 runs around 350ms on CPU but drops to 50–100ms on a GPU—which is exactly why teams who test it CPU-only sometimes wrongly reject it. Hosted APIs like Cohere and Voyage tend to land in the few-hundred-millisecond range including network round-trip. Budget about 100ms on your own hardware as a working baseline.

Common mistakes and how to tune

Reranking too few candidates. A reranker can only reorder what stage one retrieved—it cannot recover a relevant chunk that never made the shortlist. If your first stage returns top 5 and reranks those 5, you've capped your ceiling at stage one's recall. Retrieve top 50 (or more), then rerank down to 5–10.

Reranking too many. Latency scales with candidate count, since the cross-encoder runs once per pair. Reranking 500 candidates to win a marginal quality gain can blow your latency budget. Find the smallest candidate set that captures the relevant chunks.

Testing only on CPU. As above, several strong open models look slow on CPU and competitive on GPU. Benchmark on the hardware you'll actually deploy.

Using raw scores as hard thresholds. Many rerankers' scores aren't calibrated across queries, so a fixed cutoff like "drop anything below 0.5" behaves inconsistently. Prefer relative ranking, or choose a model with calibrated scores if you need thresholding.

Not measuring whether it helps. Reranking is usually worth it, but not always—and which model wins depends on your corpus. Run an A/B test on a real query set and track precision and answer quality, the same discipline behind evaluating RAG quality end to end. A reranker can't fix bad inputs, either: if retrieval is weak because of poor chunking strategies or a mismatched index among your vector database options, fix those first. Reranking refines good retrieval; it doesn't replace it. For the full picture of how these stages connect, see our overview of how retrieval augmented generation works.

Frequently asked questions

What's the difference between a reranker and an embedding model? The embedding model (a bi-encoder) encodes query and documents separately for fast, broad retrieval. The reranker (a cross-encoder) processes the query and each candidate together for a slower but far more accurate relevance score. They're complementary stages, not alternatives.

How many documents should I rerank? Retrieve a generous set—often top 50—then rerank down to the top 5–10 you send the LLM. Too few caps your recall; too many wastes latency. Tune the candidate count against your own metrics.

Does reranking add a lot of latency? Usually around 100ms on your own GPU for a moderate candidate set, sub-20ms with lightweight models like FlashRank, and a few hundred milliseconds for hosted APIs including network time. For most applications that's an acceptable trade for the accuracy gain.

Do I need a GPU to run a reranker? Not necessarily. Small models (MiniLM, FlashRank) rerank fast on CPU. Larger multilingual models like bge-reranker-v2-m3 are much faster on a GPU, so test on your target hardware before deciding.

Can reranking fix a bad retriever? No. A reranker only reorders what the first stage returned. If the relevant chunk isn't in the candidate set, no reranking recovers it—improve chunking, embeddings, or retrieval recall first.

The takeaway

Reranking retrieval results is the cheapest reliable way to turn "good enough" retrieval into precise retrieval: retrieve broadly with a fast bi-encoder, then reorder the shortlist with an accurate cross-encoder before the LLM ever sees it. Your next step is to add a reranker to your existing pipeline—start with bge-reranker-v2-m3 or a Cohere/Voyage API trial—retrieve the top 50, rerank to the top 5, and A/B test the answer quality against your current setup. The gain is usually obvious within a handful of queries.