Verdict
Quality matched on 60 of 60 cases across 2 runs (93% to 100% at 95% confidence) and the candidate is cheaper. Latency is informational during calibration.
Cases evaluated
Evaluation complete
Quality matches
100%
Est. monthly savings
Projected from matching events observed over 30 days
Status
The experiment finished and its verdict is available.
Controlled replay compares both models from the same worker. Production latency is shown separately as context and is not subtracted from candidate replay latency.
claude-sonnet-4-6 and claude-haiku-4-5 received the same frozen prompts in alternating order.
Replayed source p50
1,419 ms
claude-sonnet-4-6
Replayed candidate p50
626 ms
claude-haiku-4-5
Paired median delta
-762 ms (-55%)
95% interval: -874 ms to -693 ms
Controlled p90
1,624 ms / 735 ms
Source / candidate
Production source context
919 ms p50, 2,499 ms p90
30 captured measurements from the user's environment
Controlled coverage
30 successful pairs
Only complete source/candidate pairs enter the comparison (provider-completion-v1)
Replay failures
The evaluation compares structured content canonically and reports presentation adherence as separate evidence unless strict formatting is enabled for the feature.
This run's encrypted evidence expires at the date shown above, 14 days after the run was created. The verdict summary and savings estimate remain in experiment history after that run-specific evidence is deleted.
Review the case results before opening sensitive prompt and response evidence. Opening a case is audited.
| Case | Result | Format | Evaluation confidence | Attempts | Production source | Replayed source | Candidate | Replay delta | Status | Case details |
|---|---|---|---|---|---|---|---|---|---|---|
| 21 | Equivalent | Met | 97% | 1 | 1,329 ms 43,345 in / 1,787 out | 1,630 ms 43,345 in / 1,787 out | 610 ms 41,611 in / 1,501 out | -1,020 ms (-63%) | Evaluated |
0 source, 0 candidate
Failures are excluded from percentiles and retained as reliability evidence
| View details |
| 22 | Equivalent | Met | 97% | 1 | 2,499 ms 98,896 in / 4,182 out | 1,677 ms 98,896 in / 4,182 out | 641 ms 94,940 in / 3,513 out | -1,036 ms (-62%) | Evaluated | View details |
| 23 | Equivalent | Met | 97% | 2 | 4,097 ms 212,300 in / 9,159 out | 1,214 ms 212,300 in / 9,159 out | 672 ms 203,808 in / 7,694 out | -542 ms (-45%) | Evaluated | View details |
| 24 | Equivalent | Met | 97% | 1 | 1,637 ms 59,444 in / 2,645 out | 1,261 ms 59,444 in / 2,645 out | 703 ms 57,066 in / 2,222 out | -558 ms (-44%) | Evaluated | View details |
| 25 | Equivalent | Met | 97% | 1 | 675 ms 24,945 in / 1,126 out | 1,308 ms 24,945 in / 1,126 out | 734 ms 23,947 in / 946 out | -574 ms (-44%) | Evaluated | View details |
| 26 | Equivalent | Met | 97% | 1 | 521 ms 37,152 in / 1,966 out | 1,355 ms 37,152 in / 1,966 out | 765 ms 35,666 in / 1,651 out | -590 ms (-44%) | Evaluated | View details |
| 27 | Equivalent | Met | 97% | 1 | 884 ms 108,096 in / 4,646 out | 1,402 ms 108,096 in / 4,646 out | 536 ms 103,772 in / 3,903 out | -866 ms (-62%) | Evaluated | View details |
| 28 | Equivalent | Met | 97% | 1 | 1,270 ms 181,340 in / 8,042 out | 1,449 ms 181,340 in / 8,042 out | 567 ms 174,086 in / 6,755 out | -882 ms (-61%) | Evaluated | View details |
| 29 | Equivalent | Met | 97% | 1 | 426 ms 65,105 in / 2,931 out | 1,496 ms 65,105 in / 2,931 out | 598 ms 62,501 in / 2,462 out | -898 ms (-60%) | Evaluated | View details |
| 30 | Equivalent | Met | 97% | 1 | 137 ms 21,230 in / 992 out | 1,543 ms 21,230 in / 992 out | 629 ms 20,381 in / 833 out | -914 ms (-59%) | Evaluated | Hide details |
Case 30 evidenceFull captured evidence for this scored comparison. EquivalentFormat metscored97% evaluation confidenceJudge: gemini-3.1-flash-lite (fallback) Evaluation notesThe candidate preserved the original answer's policy, resolution, and customer-safe guidance. Format evidenceThe response preserved the expected support-answer structure. Captured prompt | ||||||||||
Northstar support case 130: resolve the ticket-summary request with the account policy and cited help-center context.
claude-sonnet-4-6
Resolved ticket-summary case 130 with the correct policy, next step, and customer-safe explanation.
claude-haiku-4-5
Cheaper candidate preserved the policy, action, and explanation for ticket-summary case 130.