Verdict
Quality matched on 28 of 30 cases (78% to 99% at 95% confidence). Review the mismatches before changing models.
Cases evaluated
Evaluation complete
Quality matches
93%
Est. monthly savings
Projected from matching events observed over 30 days
Status
The experiment finished and its verdict is available.
Controlled replay compares both models from the same worker. Production latency is shown separately as context and is not subtracted from candidate replay latency.
gpt-5.5 and gpt-4o received the same frozen prompts in alternating order.
Replayed source p50
2,079 ms
gpt-5.5
Replayed candidate p50
946 ms
gpt-4o
Paired median delta
-1,102 ms (-54%)
95% interval: -1,222 ms to -1,033 ms
Controlled p90
2,284 ms / 1,055 ms
Source / candidate
Production source context
1,076 ms p50, 2,723 ms p90
30 captured measurements from the user's environment
Controlled coverage
30 successful pairs
Only complete source/candidate pairs enter the comparison (provider-completion-v1)
Replay failures
The evaluation compares structured content canonically and reports presentation adherence as separate evidence unless strict formatting is enabled for the feature.
This run's encrypted evidence expires at the date shown above, 14 days after the run was created. The verdict summary and savings estimate remain in experiment history after that run-specific evidence is deleted.
Review the case results before opening sensitive prompt and response evidence. Opening a case is audited.
| Case | Result | Format | Evaluation confidence | Attempts | Production source | Replayed source | Candidate | Replay delta | Status | Case details |
|---|---|---|---|---|---|---|---|---|---|---|
| 11 | Equivalent | Met | 97% | 1 | 1,268 ms 54,058 in / 8,712 out | 2,330 ms 54,058 in / 8,712 out | 880 ms 51,896 in / 7,318 out | -1,450 ms (-62%) | Evaluated |
0 source, 0 candidate
Failures are excluded from percentiles and retained as reliability evidence
| View details |
| 12 | Equivalent | Met | 97% | 2 | 2,378 ms 123,684 in / 25,555 out | 1,867 ms 123,684 in / 25,555 out | 911 ms 118,737 in / 21,466 out | -956 ms (-51%) | Evaluated | View details |
| 13 | Equivalent | Met | 97% | 1 | 3,885 ms 264,884 in / 44,676 out | 1,914 ms 264,884 in / 44,676 out | 942 ms 254,289 in / 37,528 out | -972 ms (-51%) | Evaluated | View details |
| 14 | Equivalent | Met | 97% | 1 | 1,546 ms 74,384 in / 12,867 out | 1,961 ms 74,384 in / 12,867 out | 973 ms 71,409 in / 10,808 out | -988 ms (-50%) | Evaluated | Hide details |
Case 14 evidenceFull captured evidence for this scored comparison. EquivalentFormat metscored97% evaluation confidenceJudge: gemini-3.1-pro-preview Evaluation notesThe candidate preserved the original answer's policy, resolution, and customer-safe guidance. Format evidenceThe response preserved the expected support-answer structure. Captured prompt | ||||||||||
| 15 | Equivalent | Met | 97% | 1 | 635 ms 31,137 in / 5,495 out | 2,008 ms 31,137 in / 5,495 out | 1,004 ms 29,892 in / 4,616 out | -1,004 ms (-50%) | Evaluated | View details |
| 16 | Equivalent | Met | 97% | 1 | 925 ms 45,409 in / 8,265 out | 2,055 ms 45,409 in / 8,265 out | 1,035 ms 43,593 in / 6,943 out | -1,020 ms (-50%) | Evaluated | View details |
| 17 | Equivalent | Met | 97% | 1 | 1,692 ms 132,117 in / 24,393 out | 2,102 ms 132,117 in / 24,393 out | 1,066 ms 126,832 in / 20,490 out | -1,036 ms (-49%) | Evaluated | View details |
| 18 | Equivalent | Met | 97% | 1 | 2,684 ms 221,637 in / 42,442 out | 2,149 ms 221,637 in / 42,442 out | 837 ms 212,772 in / 35,651 out | -1,312 ms (-61%) | Evaluated | View details |
| 19 | Equivalent | Met | 97% | 1 | 1,032 ms 79,573 in / 15,369 out | 2,196 ms 79,573 in / 15,369 out | 868 ms 76,390 in / 12,910 out | -1,328 ms (-60%) | Evaluated | View details |
| 20 | Equivalent | Met | 97% | 1 | 406 ms 25,948 in / 5,227 out | 2,243 ms 25,948 in / 5,227 out | 899 ms 24,910 in / 4,391 out | -1,344 ms (-60%) | Evaluated | View details |
Northstar support case 14: resolve the conversation-recap request with the account policy and cited help-center context.
gpt-5.5
Resolved conversation-recap case 14 with the correct policy, next step, and customer-safe explanation.
gpt-4o
Cheaper candidate preserved the policy, action, and explanation for conversation-recap case 14.