Sample readiness
The experiment finished and its verdict is available.
These checks apply to every experiment for chat-resolver, whatever the source model. Changes apply to new runs and start a fresh evidence pool, so earlier runs are not mixed with runs under different rules.
Added to the judge's instructions. A candidate that violates them is judged not equivalent.
Checked without the judge. A candidate that fails it counts as not equivalent and skips the judge call. If the original output fails it too, the schema is likely wrong, so the judge decides and the miss is only reported.
Test whether chat-resolver answers as well with a shorter prompt: less conversation history, or a shorter system prompt that the judge model proposes. TrAIce replays 30 captured requests on gpt-5.5 with the shorter prompt, and the judge compares each answer with the one your application got. Nothing changes in your traffic.
This demo is read-only. Create a workspace to run a trial.
Send the last 4 messages of history
Across 2 trials keeping the last 4 messages, 59 of 60 answers were as useful as the full prompt's (91% to 100% at 95%). Requests sent 44% fewer input tokens. At this workload's uncached input over the last 30 days, that saves about $5,291 a month.
| Started | Change |
|---|
Once a week, TrAIce groups up to 500 of this workload's sampled requests by what they ask for and shows what each group costs. The judge model on Google Vertex AI names each group from a few of its requests. Names are kept only as long as the samples they came from (at most 14 days); the counts and costs stay.
Analysed Nov 6, 2026 from 500 sampled requests, for $0.031 of replay budget. Next analysis around Nov 13, 2026.
| Topic | Requests | Cost | Per request | Input tokens | Output tokens | Errors | Retried |
|---|---|---|---|---|---|---|---|
| Refund and credit requestsCustomers asking whether they can get a refund or account credit. | 34%170 | 39% | $1.65 | 386,424 | 12,682 | 2% | 12% |
| Order status and deliveryQuestions about where an order is and when it will arrive. | 27% |
These complete, unexpired prompt and response pairs can be selected for an experiment. Opening sample content is audited.
| Case | Captured | Prompt tokens | Response tokens | Production latency | Expires | Availability | Sample details |
|---|---|---|---|---|---|---|---|
| 31 | Nov 8, 08:32 PM |
| As useful |
|---|
| 95% interval |
|---|
| Result |
|---|
| Input tokens |
|---|
| Saving per month |
|---|
| Cost |
|---|
| Cases |
|---|
| Nov 5, 2026 | Last 4 messages | 29 of 30 | 83% to 100% | Completed | 44% fewer | About $5,291 | $1.02 | |
| Oct 30, 2026 | Last 4 messages | 30 of 30 | 88% to 100% | Completed | 44% fewer | About $5,334 | $0.97 |
| 26% |
| $1.38 |
| 322,540 |
| 10,633 |
| 2% |
| 13% |
| Password and sign-in helpUsers locked out or unable to reset a password or sign in. | 18%90 | 16% | $1.29 | 307,006 | 9,386 | 4% | 12% |
| Plan changes and billingUpgrading, downgrading, or understanding a charge on an invoice. | 13%65 | 12% | $1.28 | 303,921 | 9,338 | 2% | 12% |
| Bug reportsReports of an error message or a feature not working as expected. | 8%40 | 7% | $1.27 | 301,498 | 9,236 | 5% | 13% |
Shares are of the 500 sampled requests analysed. Per request, tokens, errors and retries are averages over each topic's requests.
| 58972 |
| 1787 |
| 1,300 ms |
| Nov 22, 08:58 PM |
| Fresh for next run |
| View sample |
| 32 | Nov 8, 08:31 PM | 134928 | 4182 | 2,442 ms | Nov 22, 08:58 PM | Fresh for next run | Close |
Sample case 32 evidenceCaptured prompt and source response available for the next experiment. fresh Captured promptNorthstar support case 32: resolve the chat-resolver request with the account policy and cited help-center context. Original responsegpt-5.5 Resolved chat-resolver case 32 with the correct policy, next step, and customer-safe explanation. | |||||||
| 33 | Nov 8, 08:30 PM | 288964 | 9159 | 3,996 ms | Nov 22, 08:58 PM | Fresh for next run | View sample |
| 34 | Nov 8, 08:29 PM | 81146 | 2645 | 1,594 ms | Nov 22, 08:58 PM | Fresh for next run | View sample |
| 35 | Nov 8, 08:28 PM | 33968 | 1126 | 656 ms | Nov 22, 08:58 PM | Fresh for next run | View sample |
| 36 | Nov 8, 08:25 PM | 50716 | 1966 | 492 ms | Nov 22, 08:58 PM | Fresh for next run | View sample |