Reranker Benchmark

Core reranker metrics

A useful reranker benchmark combines ranking quality with latency and throughput under realistic candidate-set sizes.

nDCG@k

Measures ranking quality while giving more value to highly relevant results placed near the top.

Higher is better

MRR@k

Measures how early the first relevant result appears. Useful for search and question-answering workflows.

Higher is better

Recall@k

Measures how much of the relevant set survives retrieval and reranking at the chosen cutoff.

Higher is better

Latency

Measures reranking time for a defined query, document length and candidate-set size.

Lower is better

Throughput

Measures how many query-document pairs or requests can be processed per unit time.

Higher is better

Memory

Tracks accelerator or system memory required for the target batch size and sequence length.

Lower is better

Interactive benchmark comparison

Enter normalized quality metrics between 0 and 1, plus measured latency, throughput and memory. Adjust weighting to match your own production priorities.

CandidatenDCGMRRRecallLatency msThroughputMemory GB

Quality weight45%

Latency weight25%

Throughput weight20%

Memory weight10%

A reproducible reranker benchmark workflow

Comparisons are only useful when the dataset, candidate generation and runtime conditions are documented.

Define tasks

Select representative queries, documents, languages and domains.

Freeze retrieval

Use the same first-stage candidate set for every reranker.

Measure quality

Compute nDCG, MRR, Recall or task-specific ranking metrics.

Measure serving

Record latency, throughput, memory, batch size and hardware.

Version everything

Record model revision, tokenizer, runtime, precision and dataset version.

Benchmark design guidance

A reranker should be measured as part of a retrieval system, not in isolation from candidate generation and downstream use.

Variable	Why it matters	What to report
Candidate-set size	Reranking 20 documents is very different from reranking 1,000.	Top-N input candidates and final top-k output.
Document length	Longer inputs can increase latency and truncation risk.	Average / percentile token lengths and truncation policy.
Language	Multilingual quality can vary substantially.	Language distribution and per-language metrics.
Domain	General models may behave differently on legal, medical, code or enterprise data.	Dataset domain and relevance-labeling method.
Hardware	Operational performance depends on accelerator, CPU and runtime.	Device, precision, batch size and serving framework.
First-stage retriever	A reranker cannot recover documents that were never retrieved.	Retriever model, index settings and initial recall.
Downstream RAG quality	Better ranking should improve the final application, not only an offline metric.	Answer quality, citation quality or task success where relevant.