People label cases from experiments, swap quality checks, and shorter prompt trials blind, then compare with the judge. Agreement tells you how far to trust its verdicts. The queue shows disagreements, fallback-judge verdicts, and low-confidence verdicts first, so this figure understates agreement on typical traffic. Label at least 50 cases for a usable estimate. Rubric equivalence-v5.
10 more labels needed before this estimate is usable.
Would a user of this feature accept the answer under test in place of the reference answer? Judge the meaning and the facts, not the wording. The judge's verdict, the models, and the kind of run appear after you decide.
Northstar support case 30: resolve the conversation-recap request with the account policy and cited help-center context.
Candidate answer for conversation-recap case 30 omitted the required approval caveat.
Resolved conversation-recap case 30 with the correct policy, next step, and customer-safe explanation.
Ask another owner or admin to label this case, and optionally the next cases in their queue. Assigned cases come first for them, and an assignment is complete when they save a label.
Open assignments: Maya Chen, 1 case.
Labels on cases the JSON schema check decided without the judge are not counted. Groups with few labels have wide intervals.
| Feature | Agreement | 95% interval | Judge passed, people rejected |
|---|---|---|---|
| chat-resolver | 15 of 15 (100%) | 79% to 100% | 0 of 1 |
| conversation-recap | 14 of 15 (93%) | 70% to 99% | 0 of 1 |
| ticket-summary | 10 of 10 (100%) | 72% to 100% | 0 of 0 |
| Judge model | Agreement | 95% interval | Judge passed, people rejected |
|---|---|---|---|
| gemini-3.1-pro-preview | 37 of 37 (100%) | 90% to 100% | 0 of 1 |
| gemini-3.1-flash-lite (fallback) | 2 of 3 (67%) | 20% to 94% | 0 of 1 |
| Kind of run | Agreement | 95% interval | Judge passed, people rejected |
|---|---|---|---|
| Experiment | 39 of 40 (98%) | 87% to 100% | 0 of 2 |