Verdict
Quality matched on 30 of 30 cases (88% to 100% at 95% confidence). A switch needs at least 90% at 95% confidence, so the next run's 30 new cases will be combined with these.
Cases evaluated
Evaluation complete
Quality matches
100%
Est. monthly savings
Projected from matching events observed over 30 days
Status
The experiment finished and its verdict is available.
Controlled replay compares both models from the same worker. Production latency is shown separately as context and is not subtracted from candidate replay latency.
us.anthropic.claude-sonnet-4-5-20250929-v1:0 and us.amazon.nova-pro-v1:0 received the same frozen prompts in alternating order.
Replayed source p50
2,309 ms
us.anthropic.claude-sonnet-4-5-20250929-v1:0
Replayed candidate p50
1,136 ms
us.amazon.nova-pro-v1:0
Paired median delta
-1,142 ms (-50%)
95% interval: -1,254 ms to -1,073 ms
Controlled p90
2,514 ms / 1,245 ms
Source / candidate
Production source context
2,579 ms p50, 2,899 ms p90
30 captured measurements from the user's environment
Controlled coverage
30 successful pairs
Only complete source/candidate pairs enter the comparison (provider-completion-v1)
The evaluation compares structured content canonically and reports presentation adherence as separate evidence unless strict formatting is enabled for the feature.
This run's encrypted evidence expires at the date shown above, 14 days after the run was created. The verdict summary and savings estimate remain in experiment history after that run-specific evidence is deleted.
Review the case results before opening sensitive prompt and response evidence. Opening a case is audited.
| Case | Result | Format | Evaluation confidence | Attempts | Production source | Replayed source | Candidate | Replay delta | Status | Case details |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Equivalent | Met | 97% | 2 | 3,006 ms 48,300 in / 2,760 out | 2,090 ms 48,300 in / 2,760 out | 1,020 ms 46,368 in / 2,318 out | -1,070 ms (-51%) | Evaluated |
Replay failures
0 source, 0 candidate
Failures are excluded from percentiles and retained as reliability evidence
| View details |
| 2 | Equivalent | Met | 97% | 1 | 2,795 ms 40,280 in / 1,166 out | 2,137 ms 40,280 in / 1,166 out | 1,051 ms 38,669 in / 979 out | -1,086 ms (-51%) | Evaluated | View details |
| 3 | Equivalent | Met | 97% | 1 | 2,584 ms 30,070 in / 679 out | 2,184 ms 30,070 in / 679 out | 1,082 ms 28,867 in / 570 out | -1,102 ms (-50%) | Evaluated | View details |
| 4 | Equivalent | Met | 97% | 1 | 2,373 ms 22,880 in / 792 out | 2,231 ms 22,880 in / 792 out | 1,113 ms 21,965 in / 665 out | -1,118 ms (-50%) | Evaluated | View details |
| 5 | Equivalent | Met | 97% | 1 | 2,969 ms 39,480 in / 2,256 out | 2,278 ms 39,480 in / 2,256 out | 1,144 ms 37,901 in / 1,895 out | -1,134 ms (-50%) | Evaluated | Hide details |
Case 5 evidenceFull captured evidence for this scored comparison. EquivalentFormat metscored97% evaluation confidenceJudge: gemini-3.1-pro-preview Evaluation notesThe candidate preserved the original answer's policy, resolution, and customer-safe guidance. Format evidenceThe response preserved the expected support-answer structure. Captured prompt | ||||||||||
| 6 | Equivalent | Met | 97% | 1 | 2,758 ms 32,300 in / 935 out | 2,325 ms 32,300 in / 935 out | 1,175 ms 31,008 in / 785 out | -1,150 ms (-49%) | Evaluated | View details |
| 7 | Equivalent | Met | 97% | 1 | 2,547 ms 33,790 in / 763 out | 2,372 ms 33,790 in / 763 out | 1,206 ms 32,438 in / 641 out | -1,166 ms (-49%) | Evaluated | View details |
| 8 | Equivalent | Met | 97% | 1 | 2,336 ms 26,000 in / 900 out | 2,419 ms 26,000 in / 900 out | 1,237 ms 24,960 in / 756 out | -1,182 ms (-49%) | Evaluated | View details |
| 9 | Equivalent | Met | 97% | 1 | 2,932 ms 44,520 in / 2,544 out | 2,466 ms 44,520 in / 2,544 out | 1,268 ms 42,739 in / 2,137 out | -1,198 ms (-49%) | Evaluated | View details |
| 10 | Equivalent | Met | 97% | 1 | 2,721 ms 36,860 in / 1,067 out | 2,513 ms 36,860 in / 1,067 out | 1,039 ms 35,386 in / 896 out | -1,474 ms (-59%) | Evaluated | View details |
Northstar support case 5: resolve the escalation-agent request with the account policy and cited help-center context.
us.anthropic.claude-sonnet-4-5-20250929-v1:0
Resolved escalation-agent case 5 with the correct policy, next step, and customer-safe explanation.
us.amazon.nova-pro-v1:0
Cheaper candidate preserved the policy, action, and explanation for escalation-agent case 5.