The measurement is the product.
trivium makes the workload and the limits visible together. The goal is not a leaderboard number; it is a comparison another engineer can reproduce and challenge.
Corpus and distractors.
The 5K seed is the canonical BEIR scifact corpus: 5,183 documents, 300 test queries, and unchanged qrels. The larger scales add scientific distractors from SCIDOCS, TREC-COVID, and NFCorpus.
- Exact duplicates are removed with SHA-256 over title plus text.
- Near duplicates are filtered with MinHash LSH at threshold 0.85.
- Document IDs are namespaced as
scifact:anddistractor:. - Each scale is a seeded prefix, so comparisons do not change the corpus underneath the index.
Metrics and latency.
| Measure | Definition |
|---|---|
| nDCG@10 | Rank-aware relevance quality at the first ten results, evaluated with pytrec_eval. |
| Recall@10 / @100 | Whether the relevant document appears in the first 10 or 100 retrieved candidates. |
| p50 / p95 latency | Per-query wall-clock latency after warmup, reported as a distribution rather than a single estimate. |
| Pre-flight | BM25 on untouched 5K scifact compared with the published Anserini reference and its documented implementation drift. |
What this benchmark does not claim.
Not SOTA
Model ceilings
Default runs use MiniLM-L6. Larger encoders and rerankers may achieve higher published scores.
Not disk-aware
Memory boundary
The standard vector baseline is RAM-resident. DiskBBQ is an algorithmic analogue, not a 100M-vector production claim.