Verdict
Quality matched on 30 of 30 cases (88% to 100% at 95% confidence). A switch needs at least 90% at 95% confidence, so the next run's 30 new cases will be combined with these.
Cases evaluated
Evaluation complete
Quality matches
100%
Est. monthly savings
Projected from matching events observed over 30 days
Status
The experiment finished and its verdict is available.
Controlled replay compares both models from the same worker. Production latency is shown separately as context and is not subtracted from candidate replay latency.
us.anthropic.claude-sonnet-4-5-20250929-v1:0 and us.amazon.nova-pro-v1:0 received the same frozen prompts in alternating order.
Replayed source p50
2,309 ms
us.anthropic.claude-sonnet-4-5-20250929-v1:0
Replayed candidate p50
1,136 ms
us.amazon.nova-pro-v1:0
Paired median delta
-1,142 ms (-50%)
95% interval: -1,254 ms to -1,073 ms
Controlled p90
2,514 ms / 1,245 ms
Source / candidate
Production source context
2,579 ms p50, 2,899 ms p90
30 captured measurements from the user's environment
Controlled coverage
30 successful pairs
Only complete source/candidate pairs enter the comparison (provider-completion-v1)
The evaluation compares structured content canonically and reports presentation adherence as separate evidence unless strict formatting is enabled for the feature.
This run's encrypted evidence expires at the date shown above, 14 days after the run was created. The verdict summary and savings estimate remain in experiment history after that run-specific evidence is deleted.
Review the case results before opening sensitive prompt and response evidence. Opening a case is audited.
| Case | Result | Format | Evaluation confidence | Attempts | Production source | Replayed source | Candidate | Replay delta | Status | Case details |
|---|---|---|---|---|---|---|---|---|---|---|
| 11 | Equivalent | Met | 97% | 1 | 2,510 ms 27,280 in / 616 out | 2,560 ms 27,280 in / 616 out | 1,070 ms 26,189 in / 517 out | -1,490 ms (-58%) | Evaluated |
Replay failures
0 source, 0 candidate
Failures are excluded from percentiles and retained as reliability evidence
| View details |
| 12 | Equivalent | Met | 97% | 2 | 2,299 ms 29,120 in / 1,008 out | 2,097 ms 29,120 in / 1,008 out | 1,101 ms 27,955 in / 847 out | -996 ms (-47%) | Evaluated | View details |
| 13 | Equivalent | Met | 97% | 1 | 2,895 ms 35,700 in / 2,040 out | 2,144 ms 35,700 in / 2,040 out | 1,132 ms 34,272 in / 1,714 out | -1,012 ms (-47%) | Evaluated | View details |
| 14 | Equivalent | Met | 97% | 1 | 2,684 ms 41,420 in / 1,199 out | 2,191 ms 41,420 in / 1,199 out | 1,163 ms 39,763 in / 1,007 out | -1,028 ms (-47%) | Evaluated | View details |
| 15 | Equivalent | Met | 97% | 1 | 2,473 ms 31,000 in / 700 out | 2,238 ms 31,000 in / 700 out | 1,194 ms 29,760 in / 588 out | -1,044 ms (-47%) | Evaluated | View details |
| 16 | Equivalent | Met | 97% | 1 | 2,262 ms 23,660 in / 819 out | 2,285 ms 23,660 in / 819 out | 1,225 ms 22,714 in / 688 out | -1,060 ms (-46%) | Evaluated | View details |
| 17 | Equivalent | Met | 97% | 1 | 2,858 ms 40,740 in / 2,328 out | 2,332 ms 40,740 in / 2,328 out | 1,256 ms 39,110 in / 1,956 out | -1,076 ms (-46%) | Evaluated | Hide details |
Case 17 evidenceFull captured evidence for this scored comparison. EquivalentFormat metscored97% evaluation confidenceJudge: gemini-3.1-pro-preview Evaluation notesThe candidate preserved the original answer's policy, resolution, and customer-safe guidance. Format evidenceThe response preserved the expected support-answer structure. Captured prompt | ||||||||||
| 18 | Equivalent | Met | 97% | 1 | 2,647 ms 33,440 in / 968 out | 2,379 ms 33,440 in / 968 out | 1,027 ms 32,102 in / 813 out | -1,352 ms (-57%) | Evaluated | View details |
| 19 | Equivalent | Met | 97% | 1 | 2,436 ms 34,720 in / 784 out | 2,426 ms 34,720 in / 784 out | 1,058 ms 33,331 in / 659 out | -1,368 ms (-56%) | Evaluated | View details |
| 20 | Equivalent | Met | 97% | 1 | 2,225 ms 26,780 in / 927 out | 2,473 ms 26,780 in / 927 out | 1,089 ms 25,709 in / 779 out | -1,384 ms (-56%) | Evaluated | View details |
Northstar support case 17: resolve the escalation-agent request with the account policy and cited help-center context.
us.anthropic.claude-sonnet-4-5-20250929-v1:0
Resolved escalation-agent case 17 with the correct policy, next step, and customer-safe explanation.
us.amazon.nova-pro-v1:0
Cheaper candidate preserved the policy, action, and explanation for escalation-agent case 17.