01Side by side
| Compared on | AI judge | Rule-based scoring |
|---|---|---|
| Decides | Whether a conversation passed, by reading it | Whether recorded actions broke a rule |
| Repeatable | Within a tolerance; answers vary | Exactly |
| Cost per conversation | A model call, or several | Effectively nothing |
| Good at | Honesty, tone, overall conduct | Prices, refunds, payments, data, budgets |
| Known weaknesses | Position, length and self-preference biases | Cannot judge what is not written as a rule |
| Can be gamed by the agent | If the agent’s words can address the judge | No: words do not change the record |
02When each is the better choice
AI judge
- Honesty: did the agent say anything untrue?
- Tone and firmness, where policy is about manner.
- Conversations the rules could not decide.
Rule-based scoring
- Any rule with a number: windows, limits, floors, approved amounts, budgets.
- Anything that must be compared across versions.
- The headline score.
03Using both
Let rules decide first. Use a judge only where they cannot, on a sample large enough for a range, from a different vendor than the agent’s model, and never to turn a rule failure into a pass. Check the judge against people before trusting it.
Research supports the caution: Zheng et al. (2023) documented position, verbosity and self-enhancement biases in AI judges, and Panickssery et al. (2024) found that models can recognise their own text and tend to rate it higher.
04What Duelpoint does
Duelpoint scores rules first, from a referee that is plain code. An AI judge from a different vendor than your agent’s model is used only for honesty, tone and conduct, on a sample, can only lower a score or send a case to a person, and is checked against human labels before its findings lose their beta label. See how we score.
Questions
Can an agent talk an AI judge into a better score?
It can try. Safeguards include passing the agent’s words to the judge only as quoted data, and excluding any conversation that addresses the scorer directly.
See where your agent leaks money. Tested against your own rules, with the money at stake.