Verdict
Quality matched on 60 of 60 cases across 2 runs (93% to 100% at 95% confidence) and the candidate is cheaper. Latency is informational during calibration.
Cases evaluated
Evaluation complete
Quality matches
100%
Est. monthly savings
Projected from matching events observed over 30 days
Status
The experiment finished and its verdict is available.
Controlled replay compares both models from the same worker. Production latency is shown separately as context and is not subtracted from candidate replay latency.
claude-sonnet-4-6 and claude-haiku-4-5 received the same frozen prompts in alternating order.
Replayed source p50
1,419 ms
claude-sonnet-4-6
Replayed candidate p50
626 ms
claude-haiku-4-5
Paired median delta
-762 ms (-55%)
95% interval: -874 ms to -690 ms
Controlled p90
1,624 ms / 735 ms
Source / candidate
Production source context
919 ms p50, 2,499 ms p90
30 captured measurements from the user's environment
Controlled coverage
30 successful pairs
Only complete source/candidate pairs enter the comparison (provider-completion-v1)
Replay failures
The evaluation compares structured content canonically and reports presentation adherence as separate evidence unless strict formatting is enabled for the feature.
This run's encrypted evidence expires at the date shown above, 14 days after the run was created. The verdict summary and savings estimate remain in experiment history after that run-specific evidence is deleted.
Review the case results before opening sensitive prompt and response evidence. Opening a case is audited.
| Case | Result | Format | Evaluation confidence | Attempts | Production source | Replayed source | Candidate | Replay delta | Status | Case details |
|---|---|---|---|---|---|---|---|---|---|---|
| 11 | Equivalent | Met | 97% | 1 | 1,268 ms 44,229 in / 1,742 out | 1,670 ms 44,229 in / 1,742 out | 560 ms 42,460 in / 1,463 out | -1,110 ms (-66%) | Evaluated |
0 source, 0 candidate
Failures are excluded from percentiles and retained as reliability evidence
| View details |
| 12 | Equivalent | Met | 97% | 2 | 2,378 ms 101,196 in / 5,111 out | 1,207 ms 101,196 in / 5,111 out | 591 ms 97,148 in / 4,293 out | -616 ms (-51%) | Evaluated | View details |
| 13 | Equivalent | Met | 97% | 1 | 3,885 ms 216,723 in / 8,935 out | 1,254 ms 216,723 in / 8,935 out | 622 ms 208,054 in / 7,505 out | -632 ms (-50%) | Evaluated | View details |
| 14 | Equivalent | Met | 97% | 1 | 636 ms 39,806 in / 1,608 out | 1,301 ms 39,806 in / 1,608 out | 653 ms 38,214 in / 1,351 out | -648 ms (-50%) | Evaluated | View details |
| 15 | Equivalent | Met | 97% | 1 | 1,546 ms 60,859 in / 2,573 out | 1,348 ms 60,859 in / 2,573 out | 684 ms 58,425 in / 2,161 out | -664 ms (-49%) | Evaluated | View details |
| 16 | Equivalent | Met | 97% | 1 | 1,115 ms 114,996 in / 4,762 out | 1,395 ms 114,996 in / 4,762 out | 715 ms 110,396 in / 4,000 out | -680 ms (-49%) | Evaluated | Hide details |
Case 16 evidenceFull captured evidence for this scored comparison. EquivalentFormat metscored97% evaluation confidenceJudge: gemini-3.1-pro-preview Evaluation notesThe candidate preserved the original answer's policy, resolution, and customer-safe guidance. Format evidenceThe response preserved the expected support-answer structure. Captured prompt | ||||||||||
| 17 | Equivalent | Met | 97% | 1 | 635 ms 25,476 in / 1,099 out | 1,442 ms 25,476 in / 1,099 out | 746 ms 24,457 in / 923 out | -696 ms (-48%) | Evaluated | View details |
| 18 | Equivalent | Met | 97% | 1 | 1,674 ms 194,608 in / 8,265 out | 1,489 ms 194,608 in / 8,265 out | 517 ms 186,824 in / 6,943 out | -972 ms (-65%) | Evaluated | View details |
| 19 | Equivalent | Met | 97% | 1 | 599 ms 69,351 in / 3,002 out | 1,536 ms 69,351 in / 3,002 out | 548 ms 66,577 in / 2,522 out | -988 ms (-64%) | Evaluated | View details |
| 20 | Equivalent | Met | 97% | 1 | 213 ms 22,822 in / 1,019 out | 1,583 ms 22,822 in / 1,019 out | 579 ms 21,909 in / 856 out | -1,004 ms (-63%) | Evaluated | View details |
Northstar support case 16: resolve the ticket-summary request with the account policy and cited help-center context.
claude-sonnet-4-6
Resolved ticket-summary case 16 with the correct policy, next step, and customer-safe explanation.
claude-haiku-4-5
Cheaper candidate preserved the policy, action, and explanation for ticket-summary case 16.