25%
candidate, 75% source control
Session rollout: running at 25% candidate.
Cohorts are monotonic. Lower buckets stay on the candidate as traffic increases.
Target verification
0%
Canary
5%
Progressive
10%
Progressive
25%
Progressive
50%
Progressive
95%
Complete
100%
Eligible assignments
100
100 outcomes
Telemetry complete
100.0%
Policy age p95: 52,800 ms. Enforcement overhead p95: 10 ms.
Treatment recovered
1
0 unrecovered failures
Stable-key assignments
100.0%
Stable cohorts observed.
| Arm | Assigned | Attempted | Succeeded | Failed | Recovered | Error rate | p50 latency | p95 latency | Actual spend |
|---|---|---|---|---|---|---|---|---|---|
| Candidate (claude-haiku-4-5) | 25 | 25 | 24 | 0 | 1 | 4.0% | 624 ms | 750 ms | $0.20954 |
| Source control (claude-sonnet-4-6) | 75 | 75 | 74 | 1 | 0 | 1.3% | 1,396 ms | 1,582 ms | $2.14 |
Actual rollout spend
$2.35
Source control $2.14, candidate $0.18102, source fallback $0.028518
Estimated source baseline
$0.70966
95% interval: $0.7021 to $0.71721
Estimated avoided spend
$0.50012
Control-normalized estimate, not exact savings
Evaluation spend
$0
Rollout quality refreshes
Stage duration
Hold this stage for at least 60 minutes.
Treatment samples
Collect at least 20 treatment outcomes.
Control samples
Collect at least 20 control outcomes.
Telemetry completeness
Wait for correlated post-call outcomes before promotion.
Policy freshness
At least 95% of rollout assignments must use policy no older than two minutes.
Error regression
Candidate failure and recovery rate must remain within the configured control delta.
Latency regression
Candidate p95 latency must remain within the configured control delta.
Cost regression
Candidate average request cost must remain within the configured control delta.
Quality evidence
Use current green evidence for this exact source and candidate pair: at least 90% equivalence at 95% confidence across its retained runs.
Replay budget
Keep replay budget available for refreshed quality evidence.
Provider compatibility
Source and candidate must use the same supported provider path.
Evaluation budget
Rollout quality refreshes must remain within the disclosed evaluation budget.
Every week, TrAIce replays 30 recent live ticket-summary requests on claude-sonnet-4-6 and asks the judge whether claude-haiku-4-5 answered at least as well. A check below 90% raises an alert. Checks never change the rollout.
Since the swap
116 of 120 matched
Holding up96.7%, 95% interval 91% to 99%, across 4 checks
Next check
Nov 13, 2026
Checks run weekly, a day after the swap starts.
Cost
About $0.415 per check
The original model runs on your connected key; the judge is included. $0.42 of $12.50 used this month; checks may use half of the monthly experiment budget.
| Checked | Candidate share | Matched | 95% interval | Result | Cost | Cases |
|---|---|---|---|---|---|---|
| Nov 6, 2026 | 10% | 30 of 30 | 88% to 100% | Passed | $0.42 | |
| Oct 30, 2026 | 10% | 30 of 30 | 88% to 100% | Passed | $0.39 | |
| Oct 23, 2026 | 5% | 26 of 30 | 70% to 95% | Below 90% | $0.44 | |
Promoted
10% to 25% candidate, revision 4.
Promoted
5% to 10% candidate, revision 3.
Promoted
0% to 5% candidate, revision 2.
Created
Started to 0% candidate, revision 1.
| Oct 16, 2026 |
| 5% |
| 30 of 30 |
| 88% to 100% |
| Passed |
| $0.41 |