Verdict
Quality matched on 30 of 30 cases (88% to 100% at 95% confidence). A switch needs at least 90% at 95% confidence, so the next run's 30 new cases will be combined with these.
Cases evaluated
Evaluation complete
Quality matches
100%
Est. monthly savings
Projected from matching events observed over 30 days
Status
The experiment finished and its verdict is available.
Controlled replay compares both models from the same worker. Production latency is shown separately as context and is not subtracted from candidate replay latency.
us.anthropic.claude-sonnet-4-5-20250929-v1:0 and us.amazon.nova-pro-v1:0 received the same frozen prompts in alternating order.
Replayed source p50
2,309 ms
us.anthropic.claude-sonnet-4-5-20250929-v1:0
Replayed candidate p50
1,136 ms
us.amazon.nova-pro-v1:0
Paired median delta
-1,142 ms (-50%)
95% interval: -1,254 ms to -1,073 ms
Controlled p90
2,514 ms / 1,245 ms
Source / candidate
Production source context
2,579 ms p50, 2,899 ms p90
30 captured measurements from the user's environment
Controlled coverage
30 successful pairs
Only complete source/candidate pairs enter the comparison (provider-completion-v1)
The evaluation compares structured content canonically and reports presentation adherence as separate evidence unless strict formatting is enabled for the feature.
This run's encrypted evidence expires at the date shown above, 14 days after the run was created. The verdict summary and savings estimate remain in experiment history after that run-specific evidence is deleted.
Review the case results before opening sensitive prompt and response evidence. Opening a case is audited.
| Case | Result | Format | Evaluation confidence | Attempts | Production source | Replayed source | Candidate | Replay delta | Status | Case details |
|---|---|---|---|---|---|---|---|---|---|---|
| 21 | Equivalent | Met | 97% | 1 | 2,821 ms 45,780 in / 2,616 out | 2,520 ms 45,780 in / 2,616 out | 1,120 ms 43,949 in / 2,197 out | -1,400 ms (-56%) | Evaluated |
Replay failures
0 source, 0 candidate
Failures are excluded from percentiles and retained as reliability evidence
| View details |
| 22 | Equivalent | Met | 97% | 1 | 2,610 ms 38,000 in / 1,100 out | 2,567 ms 38,000 in / 1,100 out | 1,151 ms 36,480 in / 924 out | -1,416 ms (-55%) | Evaluated | View details |
| 23 | Equivalent | Met | 97% | 2 | 2,399 ms 28,210 in / 637 out | 2,104 ms 28,210 in / 637 out | 1,182 ms 27,082 in / 535 out | -922 ms (-44%) | Evaluated | View details |
| 24 | Equivalent | Met | 97% | 1 | 2,188 ms 29,900 in / 1,035 out | 2,151 ms 29,900 in / 1,035 out | 1,213 ms 28,704 in / 869 out | -938 ms (-44%) | Evaluated | View details |
| 25 | Equivalent | Met | 97% | 1 | 2,784 ms 36,960 in / 2,112 out | 2,198 ms 36,960 in / 2,112 out | 1,244 ms 35,482 in / 1,774 out | -954 ms (-43%) | Evaluated | View details |
| 26 | Equivalent | Met | 97% | 1 | 2,573 ms 42,560 in / 1,232 out | 2,245 ms 42,560 in / 1,232 out | 1,275 ms 40,858 in / 1,035 out | -970 ms (-43%) | Evaluated | View details |
| 27 | Equivalent | Met | 97% | 1 | 2,362 ms 31,930 in / 721 out | 2,292 ms 31,930 in / 721 out | 1,046 ms 30,653 in / 606 out | -1,246 ms (-54%) | Evaluated | Hide details |
Case 27 evidenceFull captured evidence for this scored comparison. EquivalentFormat metscored97% evaluation confidenceJudge: gemini-3.1-pro-preview Evaluation notesThe candidate preserved the original answer's policy, resolution, and customer-safe guidance. Format evidenceThe response preserved the expected support-answer structure. Captured prompt | ||||||||||
| 28 | Equivalent | Met | 97% | 1 | 2,151 ms 24,440 in / 846 out | 2,339 ms 24,440 in / 846 out | 1,077 ms 23,462 in / 711 out | -1,262 ms (-54%) | Evaluated | View details |
| 29 | Equivalent | Met | 97% | 1 | 2,747 ms 42,000 in / 2,400 out | 2,386 ms 42,000 in / 2,400 out | 1,108 ms 40,320 in / 2,016 out | -1,278 ms (-54%) | Evaluated | View details |
| 30 | Equivalent | Met | 97% | 1 | 2,536 ms 34,580 in / 1,001 out | 2,433 ms 34,580 in / 1,001 out | 1,139 ms 33,197 in / 841 out | -1,294 ms (-53%) | Evaluated | View details |
Northstar support case 27: resolve the escalation-agent request with the account policy and cited help-center context.
us.anthropic.claude-sonnet-4-5-20250929-v1:0
Resolved escalation-agent case 27 with the correct policy, next step, and customer-safe explanation.
us.amazon.nova-pro-v1:0
Cheaper candidate preserved the policy, action, and explanation for escalation-agent case 27.