measure the method

Does memory
make it better?

Evaluate with paired runs: the same tasks with procedural guidance and without it. Keep the solver, model, task set, and scoring policy fixed so the graph is the variable you can inspect.

Start with a falsifiable question

Define the task distribution, success metric, cost budget, and acceptable variance before running the experiment. Record graph revision, model, prompt configuration, and seed with every result.

Run HotpotQA

pip install "methodos[eval]"
methodos eval hotpotqa --graph-id my-graph --n 200 --model gpt-4o-mini

The included harness compares with-procedural-graph and without-procedural-graph conditions and writes enough metadata to reproduce the run. Live provider tests are opt-in because they incur cost and depend on external availability.

Add a benchmark

1. Create a task loader returning stable query/answer records.
2. Implement a solver adapter using the public Solver protocol.
3. Run paired conditions against the same model and seed.
4. Persist scores, trajectories, graph revision, and configuration.
5. Add a CLI command and tests for the loader and scoring path.

Interpret results

Report mean and distribution, not only a winning aggregate. Examine regressions, rejected edits, token cost, latency, and failure categories. A graph that improves one benchmark while degrading another needs a narrower scope or a revised validation policy.