Verdict
Quality matched on 100% of cases and the candidate is cheaper. Latency is informational during calibration.
Cases evaluated
Evaluation complete
Quality matches
100%
Est. monthly savings
Projected from matching events observed over 30 days
Status
The experiment finished and its verdict is available.
Controlled replay compares both models from the same worker. Production latency is shown separately as context and is not subtracted from candidate replay latency.
claude-sonnet-4-6 and claude-haiku-4-5 received the same frozen prompts in alternating order.
Replayed source p50
1,419 ms
claude-sonnet-4-6
Replayed candidate p50
626 ms
claude-haiku-4-5
Paired median delta
-762 ms (-55%)
95% interval: -874 ms to -690 ms
Controlled p90
1,624 ms / 735 ms
Source / candidate
Production source context
1,364 ms p50, 3,576 ms p90
30 captured measurements from the user's environment
Controlled coverage
30 successful pairs
Only complete source/candidate pairs enter the comparison (provider-completion-v1)
The evaluation compares structured content canonically and reports presentation adherence as separate evidence unless strict formatting is enabled for the feature.
This run's encrypted evidence expires at the date shown above, 14 days after the run was created. The verdict summary and savings estimate remain in experiment history after that run-specific evidence is deleted.
Review the case results before opening sensitive prompt and response evidence. Opening a case is audited.
| Case | Result | Format | Evaluation confidence | Attempts | Production source | Replayed source | Candidate | Replay delta | Status | Case details |
|---|---|---|---|---|---|---|---|---|---|---|
| 11 | Equivalent | Met | 97% | 1 | 873 ms 40,691 in / 1,832 out | 1,670 ms 40,691 in / 1,832 out | 560 ms 39,063 in / 1,539 out | -1,110 ms (-66%) | Evaluated |
Replay failures
0 source, 0 candidate
Failures are excluded from percentiles and retained as reliability evidence
| View details |
| 12 | Equivalent | Met | 97% | 2 | 1,329 ms 43,345 in / 1,787 out | 1,207 ms 43,345 in / 1,787 out | 591 ms 41,611 in / 1,501 out | -616 ms (-51%) | Evaluated | View details |
| 13 | Equivalent | Met | 97% | 1 | 1,588 ms 91,997 in / 4,298 out | 1,254 ms 91,997 in / 4,298 out | 622 ms 88,317 in / 3,610 out | -632 ms (-50%) | Evaluated | View details |
| 14 | Equivalent | Met | 97% | 1 | 2,499 ms 98,896 in / 4,182 out | 1,301 ms 98,896 in / 4,182 out | 653 ms 94,940 in / 3,513 out | -648 ms (-50%) | Evaluated | View details |
| 15 | Equivalent | Met | 97% | 1 | 2,502 ms 199,031 in / 9,382 out | 1,348 ms 199,031 in / 9,382 out | 684 ms 191,070 in / 7,881 out | -664 ms (-49%) | Evaluated | View details |
| 16 | Equivalent | Met | 97% | 1 | 4,097 ms 212,300 in / 9,159 out | 1,395 ms 212,300 in / 9,159 out | 715 ms 203,808 in / 7,694 out | -680 ms (-49%) | Evaluated | View details |
| 17 | Equivalent | Met | 97% | 1 | 954 ms 70,767 in / 2,716 out | 1,442 ms 70,767 in / 2,716 out | 746 ms 67,936 in / 2,281 out | -696 ms (-48%) | Evaluated | View details |
| 18 | Equivalent | Met | 97% | 1 | 1,637 ms 59,444 in / 2,645 out | 1,489 ms 59,444 in / 2,645 out | 517 ms 57,066 in / 2,222 out | -972 ms (-65%) | Evaluated | View details |
| 19 | Equivalent | Met | 97% | 1 | 371 ms 23,353 in / 1,153 out | 1,536 ms 23,353 in / 1,153 out | 548 ms 22,419 in / 969 out | -988 ms (-64%) | Evaluated | Hide details |
Case 19 evidenceFull captured evidence for this scored comparison. EquivalentFormat metscored97% evaluation confidenceJudge: gemini-3.1-pro-preview Evaluation notesThe candidate preserved the original answer's policy, resolution, and customer-safe guidance. Format evidenceThe response preserved the expected support-answer structure. Captured prompt | ||||||||||
| 20 | Equivalent | Met | 97% | 1 | 675 ms 24,945 in / 1,126 out | 1,583 ms 24,945 in / 1,126 out | 579 ms 23,947 in / 946 out | -1,004 ms (-63%) | Evaluated | View details |
Northstar support case 19: resolve the ticket-summary request with the account policy and cited help-center context.
claude-sonnet-4-6
Resolved ticket-summary case 19 with the correct policy, next step, and customer-safe explanation.
claude-haiku-4-5
Cheaper candidate preserved the policy, action, and explanation for ticket-summary case 19.