Reciprocal Rank Fusion in practice: what k actually controls

Every hybrid retrieval tutorial arrives at the same two-line conclusion: run a dense retriever and a lexical retriever, then combine them with Reciprocal Rank Fusion. It works, so nobody interrogates it.

Building MMU — a hybrid retrieval framework over legal and compliance documents — I had to interrogate it, because on this corpus the failure modes are expensive and the easy defaults are wrong often enough to matter.

This is what I found out about the parts the tutorials skip.

Why one retriever isn't enough

Legal text punishes both retrieval families in opposite, complementary ways.

Dense (BGE-M3)

Good at paraphrase. A query about "termination for convenience" finds the clause even when it never uses those words.

Bad at exact reference. §4(2)(b) is, to an embedding model, a short low-information string that looks a lot like §4(2)(c).

Lexical (BM25)

Good at exact reference. Finds §4(2)(b) every time, ranked first.

Bad at everything else. The paragraph three documents away that qualifies that clause shares almost no vocabulary with the query, so it never surfaces.

Picking one means accepting a known, permanent failure mode. On a corpus where missing a qualifying clause changes the answer, that's not a trade worth making silently — which is the argument for fusing rather than choosing.

The thing people get wrong: fusing scores

The intuitive move is to normalise both retrievers' scores and take a weighted sum:

\text{score}(d) = \alpha \cdot \tilde{s}_{\text{dense}}(d) + (1 - \alpha) \cdot \tilde{s}_{\text{bm25}}(d)

This is worse than it looks, for a reason that isn't obvious until you plot the distributions.

Cosine similarities from a dense model cluster tightly — on my corpus, almost everything relevant lands in a narrow band, because modern embedding models are trained to make everything moderately similar to everything else. BM25 scores are unbounded and long-tailed; they depend on term rarity and document length, so their scale shifts with the query.

Min-max normalising both to $[0, 1]$ doesn't fix this. It just stretches two differently-shaped distributions onto the same interval and pretends the result is comparable. The $\alpha$ you tune on one query set silently stops being right on another.

What RRF does instead

Reciprocal Rank Fusion throws the scores away and keeps only the ordering:

\text{RRF}(d) = \sum_{r \in R} \frac{1}{k + \text{rank}_r(d)}

where $R$ is the set of retrievers and $\text{rank}_r(d)$ is the 1-indexed position of document $d$ in retriever $r$ 's list.

Ranks are the one thing every retriever produces on a comparable scale. Rank 1 means the same thing coming out of BM25 as it does coming out of FAISS — "this retriever's best guess" — in a way that 0.83 and 14.2 never will.

rrf.py

from collections import defaultdict
 
def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60):
    """Fuse ranked document-id lists. `rankings` is one list per retriever,
    each ordered best-first."""
    scores: dict[str, float] = defaultdict(float)
 
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] += 1.0 / (k + rank)
 
    return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)

That's the whole algorithm. Its power is entirely in what it refuses to use.

What k actually controls

Almost every implementation hardcodes k = 60 and cites the original paper. Nobody explains what it does, so here it is.

$k$ sets how sharply rank position is discounted — in effect, how much a single retriever is allowed to dominate on its own confidence.

Look at the contribution of the top few ranks at two values of $k$ :

Rank	$k=10$	$k=60$
1	0.0909	0.0164
2	0.0833	0.0161
3	0.0769	0.0159
10	0.0500	0.0143
Ratio, rank 1 : rank 10	1.82×	1.15×

At $k=10$ , a document ranked first is worth nearly twice one ranked tenth. One retriever putting something at the top can carry it through fusion alone.

At $k=60$ , ranks 1 through 10 are nearly indistinguishable — a 15% spread. What now wins is agreement: a document that both retrievers place somewhere in their top ten beats a document that one retriever loves and the other doesn't rank at all.

Rather than ask you to take that on trust, here it is as something you can drag. Two ranked lists, one slider, and the fused ordering recomputed live — including the exact $k$ at which the top result changes hands.

Reciprocal Rank FusionRRF(d) = Σ 1 / (k + rank_r(d))

QUERY “termination for convenience — notice period”

BM25 · lexicalexact terms

1§4(2)(b) — Termination for convenience
2§4(2)(c) — Termination for cause
4§7 — Survival of obligations
8Notice periods — general obligations
9Definitions — “Convenience”
11Schedule 3 — Notice and cure periods

Dense · vectorparaphrase

6Termination: summary of obligations
7Schedule 3 — Notice and cure periods
9Notice periods — general obligations
12Definitions — “Convenience”
15§7 — Survival of obligations
18§4(2)(b) — Termination for convenience

k — rank discount60

1 — trusts confidencetrusts consensus — 200

Fused rankingscore · bm25/dense rank

1§4(2)(b) — Termination for convenience1/180.02921
2Notice periods — general obligations8/90.02920
3Schedule 3 — Notice and cure periods11/70.02901
4§7 — Survival of obligations4/150.02896
5Termination: summary of obligations14/60.02867
6§4(2)(c) — Termination for cause2/200.02863
7Definitions — “Convenience”9/120.02838
8Master agreement — recitals17/190.02565

Weight by rank — 1/(k+rank), normalised

rank 1rank 10

Rank 1 : rank 10

1.15×

Where RRF fails

Three failure modes I hit, none of which get mentioned in the tutorials:

It discards margin. If BM25 returns one overwhelming match and then junk, RRF sees "rank 1, rank 2, rank 3" and treats the junk as a respectable showing. The information that there was a cliff after the first result — which is real signal — is thrown away with the scores.

It rewards mediocre agreement. A document both retrievers rank around 8th can outrank a document one retriever ranks 1st. Sometimes that's the point. Sometimes you've promoted a document neither retriever actually thought was the answer.

It's sensitive to list depth. Documents outside a retriever's top- $n$ contribute nothing, so cutting BM25 at 20 instead of 100 changes fused results in ways that look like a ranking bug rather than a truncation artefact. Fix the depth per retriever and treat it as a real hyperparameter, not a resource limit.

Evaluating it honestly

The reason I care about all of this is that MMU is benchmarked across five custom metrics, and two of them are specifically designed to catch fusion going wrong:

Faithfulness — whether the generated answer is actually supported by the retrieved context, not merely consistent with it. Fusion that promotes plausible-but-unsupporting context degrades this without touching a single top-1 accuracy number.

Cross-document reasoning — whether the system combines evidence split across documents. This is where single-retriever setups quietly fail, and it's the metric that justifies the fusion machinery existing at all. If hybrid retrieval doesn't beat the best single retriever here, it isn't earning its complexity.

What I'd tell someone starting

Fuse ranks, not scores — the calibration problem is real and normalisation doesn't solve it. Then treat $k$ as a decision about how much you trust consensus over confidence, and actually sweep it rather than inheriting 60 from a paper written on a different corpus.

And measure the thing fusion is supposed to buy you. Top-1 accuracy will not tell you whether hybrid retrieval is working, because that's not the case hybrid retrieval exists to fix.

Why one retriever isn't enough#

The thing people get wrong: fusing scores#

What RRF does instead#

What k actually controls#

Where RRF fails#

Evaluating it honestly#

What I'd tell someone starting#