BM25, dense vector, RRF fusion, and cross-encoder rerank — measured honestly on distractor-augmented BEIR scifact. Defensible numbers. Open methodology. One decorator away from your own pipeline.
pip install trivium | Mode | nDCG@10 | R@10 | p95 ms |
|---|---|---|---|
| bm25 | 0.6569 | 0.7784 | 0.9 |
| vector (flat) | 0.6451 | 0.7833 | 0.1 |
| hybrid_rrf | 0.7016 | 0.8482 | 1.1 |
| hybrid_rerank | 0.6820 | 0.8062 | 167.3 |
Hybrid RRF +0.045 nDCG@10 over BM25 at 5K — and
+0.057 over pure vector. Reproduce with
trivium-benchmark --modes bm25,vector,hybrid_rrf,hybrid_rerank --scales 5000.
At 5K untouched scifact, hybrid RRF beats BM25 by +0.045 nDCG@10 and
pure vector by +0.057. At 100K with 95K scientific distractors, the
encoder hits its recall ceiling and BM25 ties. Every number comes from a single
trivium-benchmark invocation.
| Scale | Mode | nDCG@10 | R@10 | R@100 | p50 ms | p95 ms |
|---|---|---|---|---|---|---|
| 5K | bm25 | 0.6569 | 0.7784 | — | — | 0.9 |
| 5K | vector (flat) | 0.6451 | 0.7833 | — | — | 0.1 |
| 5K | hybrid_rrf | 0.7016 | 0.8482 | — | — | 1.1 |
| 5K | hybrid_rerank | 0.6820 | 0.8062 | — | — | 167.3 |
| 100K | bm25 | 0.3206 | 0.3793 | — | — | 6.5 |
| 100K | vector (IVFPQ+RFlat) | 0.1674 | 0.1971 | — | — | 0.05 |
| 100K | hybrid_rrf | 0.3232 | 0.3796 | — | — | 6.8 |
| 100K | hybrid_rerank | 0.3362 | 0.3871 | — | — | 187.7 |
Hybrid RRF pulls +0.045 nDCG@10 over BM25 — and +0.057 over pure vector. This is the headline: hybrid wins on the smallest test set, where the dense encoder has full recall and BM25 has full lexical precision.
At 100K the nDCG ceiling collapses for every method because MiniLM-L6 (22M params) can't pull gold documents out of 95K distractors. The exact dense ceiling is 0.2957; IVFPQ+RFlat drops to 0.1674. Hybrid RRF ties BM25; rerank adds a small +0.015 lift.
Cross-encoder rerank costs ~180 ms p95 per query — 5–10× the original estimate. Updating the published number is a finding. Use hybrid_rrf unless you need the last 0.015.
trivium ships with the things most benchmarks don't. A pre-flight that fails fast. A reproducibility manifest on every CSV row. A registry you can extend without forking.
Every CSV row carries Python, OS, NumPy, faiss, sentence-transformers, torch, bm25s, OMP threads, and git SHA. A BM25 pre-flight pins the implementation to the published Anserini baseline.
Fail fast at 0.62. The benchmark is honest.
Adding a new encoder, reranker, or pipeline mode is a single @register_pipeline call. The registry-driven runner has no elif chains — every mode goes through the same gate.
From a new model to a row in benchmark.csv.
pytrec_eval relevance grading, perf_counter_ns() per query, 20-query warmup, single-stream. The published 20–40 ms rerank number turned out to be 5–10× too low — we updated it.
If a number is wrong, we change it.
SCIDOCS + TREC-COVID + NFCorpus distractors prefixed by ID, deduped by SHA-256 and MinHash. The 100K curve is a true prefix of 1M so scale comparisons stay clean.
No closed-world optimism. No leakage.
OPQ + IVFPQ + RFlat with nlist = 4·sqrt(N), nprobe sweep over 8–256, k_factor = 4 for RFlat refinement. Every scale gets its own frontier; no single-knob tuning.
Best speed/recall is a sweep, not a setting.
Bm25, Faiss.Flat, Rrf, get_reranker — composable primitives with the same retriever ABC. The benchmark is the reference workload; the library is what you ship.
Write the pipeline once, run it everywhere.
Composable primitives that work the same way whether you're benchmarking against BEIR or running in production. Same retriever ABC, same fusion contract, same reranker interface.
list[SearchResult]. fuse() signature. @register_pipeline("name"); no elif chains. from trivium.retrieval.bm25 import Bm25
from trivium.retrieval.faiss import Faiss
from trivium.fusion.rrf import Rrf
from trivium.reranking.registry import get_reranker
# 1. Build a BM25 retriever
bm25 = Bm25()
bm25.add_documents(your_docs)
# 2. Build a dense retriever
vec = Faiss.Flat()
vec.add_documents(your_docs, vectors=your_vectors)
# 3. Search in parallel
bm25_results = bm25.search(queries, k=100)
vec_results = vec.search(query_vecs, k=100)
# 4. Fuse with Reciprocal Rank Fusion
fused = Rrf(k=60).fuse([bm25_results, vec_results], top_k=50)
candidates = [doc_lookup[h.doc_id] for h in fused]
# 5. Rerank with a cross-encoder (optional)
reranker = get_reranker("bge-large")
final = reranker.rerank(query_text, candidates, top_k=10)# A new pipeline mode is one decorator away.
from trivium.pipelines.registry import register_pipeline
@register_pipeline("hybrid_rrf_v2")
def hybrid_rrf_v2(query, docs, weights=(0.4, 0.6), k=60):
bm25_hits = bm25.search([query], k=100)[0]
vec_hits = vec.search([embed(query)], k=100)[0]
return weighted_rrf([bm25_hits, vec_hits], weights=weights, k=k)$ trivium-benchmark \
--modes bm25,vector,hybrid_rrf,hybrid_rerank \
--scales 5000,100000 \
--output results/benchmark.csv
[pre-flight] bm25 nDCG@10 = 0.6781 ✓ (target 0.6789 ± 0.03)
[5K] bm25 nDCG@10=0.6569 R@10=0.7784 p95=0.9 ms
[5K] vector (flat) nDCG@10=0.6451 R@10=0.7833 p95=0.1 ms
[5K] hybrid_rrf nDCG@10=0.7016 R@10=0.8482 p95=1.1 ms
[100K] hybrid_rrf nDCG@10=0.3232 R@10=0.3796 p95=6.8 ms
[100K] hybrid_rerank nDCG@10=0.3362 R@10=0.3871 p95=187.7 ms
trivium wires together a corpus, an embedder, retrievers, a fusion
strategy, and a reranker behind a single decorator. Every hyperparameter
lives in configs/default.yaml. Every CSV row records the
configuration that produced it.
BM25 (bm25s, Lucene defaults) and Faiss (Flat, IVFPQ, OPQ + RFlat) are
first-class retrievers. The retriever contract returns
list[SearchResult] with hits, scores, and latency.
Bm25(method="lucene", k1=0.9, b=0.4) Faiss.Flat(metric="ip") Faiss.IVFPQ(nlist, m=48, nbits=4)
RRF is the default. score(d) = Σ wᵢ / (k + rankᵢ(d)). Sweep
over k ∈ 200 and weights ∈ 0.3. The Pareto frontier lives in the CSV.
Rrf(k=60).fuse(ranker_lists, top_k=50) WeightedRrf(weights=(0.4, 0.6)) Custom fusion via registry Rerankers run on the fused top-50 and emit top-10. The MiniLM-L6 CE costs ~180 ms p95; BGE-large adds more lift; MonoT5-3B is a published ceiling we don't run by default.
get_reranker("minilm-ce") get_reranker("bge-large") get_reranker("monot5-3b") Choosing a fusion strategy for a real product. Need apples-to-apples numbers across BM25, vector, RRF, and rerank — at the scale that matters.
Validating a new encoder or reranker. Need a defensible reference workload that survives reviewer scrutiny — closed-world qrels, untouched scifact, reproducible manifests.
Want to publish numbers you can defend. Need a benchmark with pre-flight checks, golden anchors, and a methodology section that explains every knob in the YAML.
A good benchmark is defined as much by what it doesn't claim as by what it measures. trivium publishes its limits in the same place it publishes its numbers.
Every CSV row carries Python, OS, NumPy, faiss, sentence-transformers, torch, bm25s, OMP threads, and git SHA. The BM25 pre-flight pins the BM25 implementation to the published Anserini baseline (0.6789 ± 0.03).
E5-Mistral-7B (0.749), BGE-large (0.741), and MonoT5-3B rerank (0.777) are the published scifact ceilings. We don't run those models. The hybrid advantage we show is from MiniLM-L6 and a small CE reranker.
The vector index is RAM-resident. DiskBBQ's actual advantage is at 100M+ vectors where the index cannot fit in RAM. The DiskBBQ mode in trivium is an algorithmic analog.
Bm25, Faiss.Flat, Rrf, get_reranker — composable primitives. The benchmark is the reference workload; the public API is what you ship.
Corpus construction, embedding trade-offs, the IVFPQ + RFlat story,
the BM25 pre-flight invariant — every knob in configs/default.yaml
documented in the README.
Install trivium, prepare the BEIR corpus, run the benchmark, summarise the rows, and read the README. Total time on a laptop: about five minutes.
pip install trivium