Eval scorecard
Forty questions, written before any of these numbers: thirty whose answer sits in a contract, with the page and clause that hold it, and ten the documents do not answer. Each one goes through the same route the chat uses. Run it yourself with pnpm eval.
Latest run
Retrieval only. This run measured retrieval and the confidence gate. No answer model ran, so there are no faithfulness, correctness, citation or token figures; they read “not run” rather than zero.
Query and chunk embeddings were computed locally with the same BAAI/bge-base-en-v1.5 weights (ONNX, CLS pooling, normalised) because the Workers AI daily allowance was used up when this ran. Run pnpm ingest --force and pnpm eval once it has room to replace them.
Retrieval only: no answer model was available when this ran, so there are no faithfulness, correctness, citation or token figures. Models tried: workers-ai/llama-3.3-70b-instruct-fp8-fast: Too Many Requests.
Scorecard
Retrieval
Is the right page, and the right clause, among the first passages the model is given?
- Page hit@1
- 90.0%
- 27 of 30 gold questions
- Page hit@3
- 96.7%
- 29 of 30 gold questions
- Clause hit@1
- 86.7%
- 26 of 30 gold questions
- Clause hit@3
- 96.7%
- 29 of 30 gold questions
- Reciprocal rank
- 0.938
- mean of 1 over the rank of the first passage on the right page
Declining
Does the confidence gate stop questions the document does not answer? No model ran.
- Out-of-scope stopped
- 100.0%
- 10 of 10 out-of-scope questions
- Stopped by the gate
- 100.0%
- 10 of 10 out-of-scope, before any model call
- Answerable but declined
- 10.0%
- 3 of 30 gold questions
Answers
Needs an answer model. Not measured in this run.
- Faithfulness
- not run
- the judge did not run
- Correctness
- not run
- the judge did not run
- Citations resolve
- not run
- answers whose every [p.N §clause] points at a passage the model saw
- Questions passed
- 36 of 40
- right page in the top 3, or an out-of-scope question stopped
Speed and cost
Retrieval latency only, measured on the machine that ran the eval.
- Latency p50
- 956 ms
- embed query, search, gate
- Latency p95
- 1,317 ms
- nearest-rank percentile over all questions
- First token p50
- not run
- request to the first streamed word
- Tokens
- not run
- no model ran
- Billed cost
- not run
- no model ran
Across runs
- Page hit@396.7%first run
- Clause hit@396.7%first run
- Out-of-scope declined100.0%first run
- Faithfulnessnot measured
- Latency p951,317 msfirst run
| Run (UTC) | Commit | What ran | Page hit@3 | Declined |
|---|---|---|---|---|
| 2026-10-23 01:51 | a914bc1 | Retrieval and gate | 96.7% | 100% |
Every question
All 40Failed only (4)
| Question | Expected | What happened | Confidence | Faithful | Result |
|---|---|---|---|---|---|
| g01How long does the Receiving Party have to notify the Disclosing Party of a suspected unauthorised disclosure?mutual-nda | p.4 §2.3 | page at rank 1, clause at rank 1 | 0.92 | not run | Pass |
| g02How long do the confidentiality obligations last after the agreement ends?mutual-nda | p.7 §9.2 | page at rank 1, clause at rank 1 | 0.91 | not run | Pass |
| g03Which law governs the NDA?mutual-nda | p.7 §11.1 | declined; page at rank 1 | 0.22 | not run | Fail |
| g04Within how many days must confidential information be returned or destroyed?mutual-nda | p.5 §5.1 | page at rank 1, clause at rank 1 | 0.92 | not run | Pass |
| g05In the second draft, how many hours does the receiving party have to report a suspected breach?mutual-nda-v2 | p.4 §2.3 | page at rank 1, clause at rank 1 | 0.84 | not run | Pass |
| g06Can a party use general knowledge that its staff remember from the confidential information?mutual-nda-v2 | p.5 §3.3 | declined; page at rank 1 | 0.38 | not run | Fail |
| g07What is the cap on each party's liability in a contract year?msa | p.9 §11.2 | page at rank 1, clause at rank 1 | 0.59 | not run | Pass |
| g08How long is the initial term of the Master Services Agreement?msa | p.4 §2.1 | page at rank 1, clause at rank 1 | 0.93 | not run | Pass |
| g09What interest is charged on late payments?msa | p.6 §5.5 | declined; page at rank 1 | 0.39 | not run | Fail |
| g10How much public liability insurance must the Supplier hold?msa | p.9 §12.1 | page at rank 1, clause at rank 1 | 0.91 | not run | Pass |
| g11Can the Customer terminate for convenience, and with how much notice?msa | p.10 §13.3 | page at rank 1, clause at rank 1 | 0.91 | not run | Pass |
| g12What monthly availability does the Provider commit to?saas-agreement | p.6 §5.1 | page at rank 1, clause at rank 1 | 0.73 | not run | Pass |
| g13What is the Provider's liability limit for a data security breach?saas-agreement | p.9 §12.3 | page at rank 1, clause at rank 1 | 0.91 | not run | Pass |
| g14How many Authorised Users and how much storage does the subscription include?saas-agreement | p.4 §2.2 | page at rank 1, clause at rank 1 | 0.81 | not run | Pass |
| g15How long does the Customer have to export its data after the agreement ends?saas-agreement | p.7 §6.5 | page at rank 1, clause at rank 1 | 0.91 | not run | Pass |
| g16By how much can the Provider increase the fees at renewal?saas-agreement | p.5 §4.4 | page at rank 1, clause at rank 1 | 0.80 | not run | Pass |
| g17How often is customer data backed up and for how long are backups kept?saas-agreement | p.6 §6.3 | page at rank 1, clause at rank 1 | 0.89 | not run | Pass |
| g18What is the first response time for a severity 1 support request?saas-agreement | p.14 §Sch2.2 | page at rank 1, clause at rank 1 | 0.78 | not run | Pass |
| g19What notice period applies to the Employee after probation?employment-agreement | p.9 §14.1 | page at rank 2, clause at rank 2 | 0.90 | not run | Pass |
| g20How many days of paid holiday does the Employee get each year?employment-agreement | p.6 §8.1 | page at rank 1, clause at rank 1 | 0.89 | not run | Pass |
| g21How long does the non-compete last after employment ends?employment-agreement | p.8 §13.2 | page at rank 8, clause at rank 8 | 0.54 | not run | Fail |
| g22What pension contribution does the employer make?employment-agreement | p.7 §10.1 | page at rank 1, clause at rank 1 | 0.80 | not run | Pass |
| g23How many weeks of full sick pay does the Employee get after probation?employment-agreement | p.7 §9.3 | page at rank 1, clause at rank 1 | 0.92 | not run | Pass |
| g24How long does the Firm keep client matter files?privacy-policy | p.4 §6.1 | page at rank 1, clause at rank 1 | 0.81 | not run | Pass |
| g25How quickly will the Firm respond to a request to exercise my rights?privacy-policy | p.5 §7.2 | page at rank 1, clause at rank 1 | 0.72 | not run | Pass |
| g26Does the Firm sell personal data?privacy-policy | p.3 §4.3 | page at rank 2, clause at rank 2 | 0.75 | not run | Pass |
| g27How long are CCTV images kept?privacy-policy | p.8 §13.3 | page at rank 1, clause at rank 1 | 0.92 | not run | Pass |
| g28How quickly must the Processor notify the Controller of a personal data breach?dpa | p.6 §9.1 | page at rank 1, clause at rank 1 | 0.93 | not run | Pass |
| g29How much notice must the Processor give before appointing a new sub-processor?dpa | p.5 §6.2 | page at rank 1, clause at rank 1 | 0.93 | not run | Pass |
| g30How often can the Controller audit the Processor?dpa | p.7 §11.3 | page at rank 1, clause at rank 2 | 0.93 | not run | Pass |
| o01What amount of liquidated damages is payable if the NDA is breached?mutual-nda | decline | declined by the gate | 0.18 | not run | Pass |
| o02Which arbitration institution decides disputes under the NDA?mutual-nda | decline | declined by the gate | 0.11 | not run | Pass |
| o03Does the MSA contain a most-favoured-customer pricing clause?msa | decline | declined by the gate | 0.14 | not run | Pass |
| o04What carbon emissions reduction target must the Supplier meet?msa | decline | declined by the gate | 0.11 | not run | Pass |
| o05Does the Provider offer a source code escrow arrangement?saas-agreement | decline | declined by the gate | 0.17 | not run | Pass |
| o06What cyber insurance cover must the Provider carry?saas-agreement | decline | declined by the gate | 0.13 | not run | Pass |
| o07How many stock options is the Employee granted?employment-agreement | decline | declined by the gate | 0.19 | not run | Pass |
| o08What relocation allowance is the Employee paid?employment-agreement | decline | declined by the gate | 0.26 | not run | Pass |
| o09Does the Firm use facial recognition at its offices?privacy-policy | decline | declined by the gate | 0.09 | not run | Pass |
| o10What service credits does the Processor owe for downtime?dpa | decline | declined by the gate | 0.40 | not run | Pass |
How a question passes
- An answerable question passes when a passage on the expected page is among the top three retrieved, the assistant answered rather than declined, and the judge scored the answer at least 4 of 5 for faithfulness.
- An out-of-scope question passes when the assistant declined, whether the confidence gate stopped it or the model did.
- The judge is a model reading the passages, the answer and a reference written for the gold set. It is an estimate, not ground truth, and its name is shown above.
- The confidence curve was fitted on these same questions, so the declining figures are not held out.