StagingDB checkingCommit: 5c5ec74eLast migration: checkingApp: staging.runtraice.comDB server: Cloud SQL via Unix socketLogical DB: traiceAuth: Staging OAuth or emailEmail: Brevo

Experiments

Workspace

escalation-agent

us.anthropic.claude-sonnet-4-5-20250929-v1:0 → us.anthropic.claude-haiku-4-5-20251001-v1:0

Verdict

Gather more evidence

Quality matched on 29 of 30 cases (83% to 100% at 95% confidence). A switch needs at least 90% at 95% confidence, so the next run's 30 new cases will be combined with these.

Done

Judge: gemini-3.1-pro-preview (30)Run cost: $0.95001

Cases evaluated

30 of 30

Evaluation complete

Quality matches

29 of 30

97%

Est. monthly savings

$1,360/mo· 66.7% lower

Projected from matching events observed over 30 days

Status

Done

The experiment finished and its verdict is available.

Latency evidence

Controlled replay compares both models from the same worker. Production latency is shown separately as context and is not subtracted from candidate replay latency.

Informational during calibration

Latency not tested

This run did not have one explicit funding path that could execute both the source and candidate replay legs. Its quality and cost evidence remains valid.

This run re-checks the kept test set escalation-agent on us.anthropic.claude-sonnet-4-5-20250929-v1:0 (version 1). Its result stays on the test set and does not change swap recommendations.

Sample window: Oct 29, 08:58 PM to Nov 5, 08:58 PM
Candidate funding: Connected AWS Bedrock key
Candidate provider: AWS Bedrock
Judge used: gemini-3.1-pro-preview (30)
Evaluation criteria: Candidate quality and usefulness
JSON schema: No schema for this feature
Format policy: Presentation measured separately
Case evidence retained until: Nov 22, 12:23 AM

The evaluation compares structured content canonically and reports presentation adherence as separate evidence unless strict formatting is enabled for the feature.

This run's encrypted evidence expires at the date shown above, 14 days after the run was created. The verdict summary and savings estimate remain in experiment history after that run-specific evidence is deleted.

Case summary

Review the case results before opening sensitive prompt and response evidence. Opening a case is audited.

Case	Result	Format	Evaluation confidence	Attempts	Production source	Replayed source	Candidate	Replay delta	Status	Case details
11	Equivalent	Met	96%	0	2,510 ms 27,280 in / 616 out	Unavailable	1,530 ms 26,462 in / 542 out	Unavailable	Evaluated

Experiments

Workspace

Experiment results

Experiments

Experiment results

escalation-agent

Latency evidence

Case summary

Case 20 evidence

Evaluation notes

Captured prompt

Original response

Candidate response

20	Equivalent	Met	96%	0	2,225 ms 26,780 in / 927 out	Unavailable	1,491 ms 25,977 in / 816 out	Unavailable	Evaluated	Hide details
Case 20 evidence Full captured evidence for this scored comparison. EquivalentFormat metscored96% evaluation confidenceJudge: gemini-3.1-pro-preview Evaluation notes The candidate kept the escalation decision, the refund limit, and the next step. Captured prompt Northstar support case 20: resolve the escalation-agent request with the account policy and cited help-center context.