Getting started

From source checkout to benchmark CSV.

The first run downloads the BEIR corpus and model dependencies. Expect roughly 6 GB of corpus data and a few minutes on a modern laptop.

1. Install the checked-out source.

trivium is not being distributed through PyPI yet. Install the repository in editable mode so changes and registered components are immediately available.

git clone https://github.com/sachncs/trivium.git
cd trivium
python -m pip install -e ".[dev]"

Python 3.12 or newer is required. The development extra installs pytest, coverage, Ruff, mypy, and the wheel builder used by CI.

2. Prepare the corpus cache.

Preparation downloads the canonical scifact seed, adds scientific distractors, deduplicates documents, and writes the cached embeddings used by the benchmark.

trivium-prepare --max-scales 100000

The cache is written under data/cache/. It is safe to rerun preparation; validated cache files are reused.

3. Run a first comparison.

Start with the small, interpretable set of modes below. Add scales or tuning sweeps only after the pre-flight result is healthy.

trivium-benchmark \
  --modes bm25,vector,hybrid_rrf,hybrid_rerank \
  --scales 5000,100000 \
  --output results/benchmark.csv

Summarise the generated rows with:

trivium-summarise --csv results/benchmark.csv

4. Read the result responsibly.

Check	Why it matters
Pre-flight	Untouched 5K scifact BM25 should land near 0.6789 ± 0.03. A result below 0.62 is a failed run.
CSV row	Every row carries metrics, latency, configuration, runtime information, and the source commit.
Scale	100K adds 95K scientific distractors. A lower score there is part of the workload, not a hidden failure.