01Ten criteria
| Criterion | What good looks like |
|---|---|
| Your rules | Tests against your refund policy, prices and approved amounts, not a generic standard. |
| Outcomes | Scores what the agent did (refunds, prices, payments, data), not how it sounded. |
| Persistence | Multi-turn characters that push, bluff and adapt, not single prompts. |
| Honest customers | Measures over-refusal as well as breaches. |
| Ranges | Reports rates with likely ranges and enough conversations to be useful. |
| Cost | Shows the money at stake for each failure. |
| Evidence | The conversations and the line where the agent gave way, exportable. |
| Safety | Runs only against test copies, never live payments or customers. |
| Re-testing | Re-runs the same test after fixes and on every change, and compares fairly. |
| Independence | Attack scripts you cannot tune your agent against, from someone who did not build it. |
02Red flags
- A single score with no range or sample size.
- Pass or fail judged only by another model reading the transcript.
- No honest customers in the test.
- Asking to connect your live agent.
- Attack scripts you can read and tune against.
03Questions for a demo
- Show me a failure: what did the agent do, what did it cost, and where did it give way?
- How many conversations is this rate based on, and what is its range?
- How do you score whether a refund was within policy?
- How do you measure refusals of honest customers?
- What happens when I change the agent and re-test?
Building it yourself instead? See build or buy. For your agent’s vendor, see the questions in the enterprise guide.
Questions
Is a general LLM evaluation platform enough?
It is a good fit for answer quality and regressions. For agents with money authority, check whether it can run multi-turn adversarial conversations against your own rules, score outcomes rather than wording, and report rates with ranges.
See where your agent leaks money. Tested against your own rules, with the money at stake.