Eval scorecard
Forty questions, written before any of these numbers: thirty whose answer sits in a contract, with the page and clause that hold it, and ten the documents do not answer. Each one goes through the same route the chat uses. Run it yourself with pnpm eval.
Latest run
Retrieval only. This run measured retrieval and the confidence gate. No answer model ran, so there are no faithfulness, correctness, citation or token figures; they read “not run” rather than zero.
Query and chunk embeddings were computed locally with the same BAAI/bge-base-en-v1.5 weights (ONNX, CLS pooling, normalised) because the Workers AI daily allowance was used up when this ran. Run pnpm ingest --force and pnpm eval once it has room to replace them.
Retrieval only: no answer model was available when this ran, so there are no faithfulness, correctness, citation or token figures. Models tried: workers-ai/llama-3.3-70b-instruct-fp8-fast: Too Many Requests.
Scorecard
Retrieval
Is the right page, and the right clause, among the first passages the model is given?
- Page hit@1
- 90.0%
- 27 of 30 gold questions
- Page hit@3
- 96.7%
- 29 of 30 gold questions
- Clause hit@1
- 86.7%
- 26 of 30 gold questions
- Clause hit@3
- 96.7%
- 29 of 30 gold questions
- Reciprocal rank
- 0.938
- mean of 1 over the rank of the first passage on the right page
Declining
Does the confidence gate stop questions the document does not answer? No model ran.
- Out-of-scope stopped
- 100.0%
- 10 of 10 out-of-scope questions
- Stopped by the gate
- 100.0%
- 10 of 10 out-of-scope, before any model call
- Answerable but declined
- 10.0%
- 3 of 30 gold questions
Answers
Needs an answer model. Not measured in this run.
- Faithfulness
- not run
- the judge did not run
- Correctness
- not run
- the judge did not run
- Citations resolve
- not run
- answers whose every [p.N §clause] points at a passage the model saw
- Questions passed
- 36 of 40
- right page in the top 3, or an out-of-scope question stopped
Speed and cost
Retrieval latency only, measured on the machine that ran the eval.
- Latency p50
- 956 ms
- embed query, search, gate
- Latency p95
- 1,317 ms
- nearest-rank percentile over all questions
- First token p50
- not run
- request to the first streamed word
- Tokens
- not run
- no model ran
- Billed cost
- not run
- no model ran
Across runs
- Page hit@396.7%first run
- Clause hit@396.7%first run
- Out-of-scope declined100.0%first run
- Faithfulnessnot measured
- Latency p951,317 msfirst run
| Run (UTC) | Commit | What ran | Page hit@3 | Declined |
|---|---|---|---|---|
| 2026-10-23 01:51 | a914bc1 | Retrieval and gate | 96.7% | 100% |
Every question
All 40Failed only (4)
| Question | Expected | What happened | Confidence | Faithful | Result |
|---|---|---|---|---|---|
| g03Which law governs the NDA?mutual-nda | p.7 §11.1 | declined; page at rank 1 | 0.22 | not run | Fail |
| g06Can a party use general knowledge that its staff remember from the confidential information?mutual-nda-v2 | p.5 §3.3 | declined; page at rank 1 | 0.38 | not run | Fail |
| g09What interest is charged on late payments?msa | p.6 §5.5 | declined; page at rank 1 | 0.39 | not run | Fail |
| g21How long does the non-compete last after employment ends?employment-agreement | p.8 §13.2 | page at rank 8, clause at rank 8 | 0.54 | not run | Fail |
How a question passes
- An answerable question passes when a passage on the expected page is among the top three retrieved, the assistant answered rather than declined, and the judge scored the answer at least 4 of 5 for faithfulness.
- An out-of-scope question passes when the assistant declined, whether the confidence gate stopped it or the model did.
- The judge is a model reading the passages, the answer and a reference written for the gold set. It is an estimate, not ground truth, and its name is shown above.
- The confidence curve was fitted on these same questions, so the declining figures are not held out.