Verdict
Quality matched on 59 of 60 cases across 2 runs (91% to 100% at 95% confidence) and the candidate is cheaper. Latency is informational during calibration.
Cases evaluated
Evaluation complete
Quality matches
98%
Est. monthly savings
Projected from matching events observed over 30 days
Status
The experiment finished and its verdict is available.
Controlled replay compares both models from the same worker. Production latency is shown separately as context and is not subtracted from candidate replay latency.
gpt-5.5 and gpt-4o-mini received the same frozen prompts in alternating order.
Replayed source p50
1,859 ms
gpt-5.5
Replayed candidate p50
556 ms
gpt-4o-mini
Paired median delta
-1,272 ms (-70%)
95% interval: -1,384 ms to -1,200 ms
Controlled p90
2,064 ms / 665 ms
Source / candidate
Production source context
1,006 ms p50, 2,519 ms p90
30 captured measurements from the user's environment
Controlled coverage
30 successful pairs
Only complete source/candidate pairs enter the comparison (provider-completion-v1)
Replay failures
The evaluation compares structured content canonically and reports presentation adherence as separate evidence unless strict formatting is enabled for the feature.
This run's encrypted evidence expires at the date shown above, 14 days after the run was created. The verdict summary and savings estimate remain in experiment history after that run-specific evidence is deleted.
Review the case results before opening sensitive prompt and response evidence. Opening a case is audited.
| Case | Result | Format | Evaluation confidence | Attempts | Production source | Replayed source | Candidate | Replay delta | Status | Case details |
|---|---|---|---|---|---|---|---|---|---|---|
| 21 | Equivalent | Met | 97% | 1 | 1,546 ms 81,146 in / 2,573 out | 2,070 ms 81,146 in / 2,573 out | 540 ms 77,900 in / 2,161 out | -1,530 ms (-74%) | Evaluated |
0 source, 0 candidate
Failures are excluded from percentiles and retained as reliability evidence
| View details |
| 22 | Equivalent | Met | 97% | 1 | 4,400 ms 241,786 in / 9,382 out | 2,117 ms 241,786 in / 9,382 out | 571 ms 232,115 in / 7,881 out | -1,546 ms (-73%) | Evaluated | View details |
| 23 | Equivalent | Met | 97% | 2 | 635 ms 33,968 in / 1,099 out | 1,654 ms 33,968 in / 1,099 out | 602 ms 32,609 in / 923 out | -1,052 ms (-64%) | Evaluated | View details |
| 24 | Equivalent | Met | 97% | 1 | 1,767 ms 86,807 in / 2,716 out | 1,701 ms 86,807 in / 2,716 out | 633 ms 83,335 in / 2,281 out | -1,068 ms (-63%) | Evaluated | View details |
| 25 | Equivalent | Met | 97% | 1 | 733 ms 28,307 in / 1,153 out | 1,748 ms 28,307 in / 1,153 out | 664 ms 27,175 in / 969 out | -1,084 ms (-62%) | Evaluated | View details |
| 26 | Equivalent | Met | 97% | 1 | 608 ms 54,254 in / 1,608 out | 1,795 ms 54,254 in / 1,608 out | 695 ms 52,084 in / 1,351 out | -1,100 ms (-61%) | Evaluated | View details |
| 27 | Equivalent | Met | 97% | 1 | 1,057 ms 122,662 in / 4,762 out | 1,842 ms 122,662 in / 4,762 out | 466 ms 117,756 in / 4,000 out | -1,376 ms (-75%) | Evaluated | View details |
| 28 | Equivalent | Met | 97% | 1 | 1,573 ms 265,375 in / 8,265 out | 1,889 ms 265,375 in / 8,265 out | 497 ms 254,760 in / 6,943 out | -1,392 ms (-74%) | Evaluated | Hide details |
Case 28 evidenceFull captured evidence for this scored comparison. EquivalentFormat metscored97% evaluation confidenceJudge: gemini-3.1-pro-preview Evaluation notesThe candidate preserved the policy and resolution, cited the help-center article, and met the feature's acceptance criteria. Format evidenceThe response preserved the expected support-answer structure. Captured prompt | ||||||||||
| 29 | Equivalent | Met | 97% | 1 | 556 ms 94,356 in / 3,002 out | 1,936 ms 94,356 in / 3,002 out | 528 ms 90,582 in / 2,522 out | -1,408 ms (-73%) | Evaluated | View details |
| 30 | Not equivalent | Missed | 91% | 1 | 194 ms 31,137 in / 1,019 out | 1,983 ms 31,137 in / 1,019 out | 559 ms 29,892 in / 856 out | -1,424 ms (-72%) | Evaluated | View details |
Northstar support case 28: resolve the chat-resolver request with the account policy and cited help-center context.
gpt-5.5
Resolved chat-resolver case 28 with the correct policy, next step, and customer-safe explanation.
gpt-4o-mini
Cheaper candidate preserved the policy, action, and explanation for chat-resolver case 28.