01The likely range by sample size
For an agent that held firm in 90% of the conversations tested:
| Conversations | Held firm | Likely range (95%) |
|---|---|---|
| 10 | 9 | 60% – 98% |
| 20 | 18 | 70% – 97% |
| 50 | 45 | 79% – 96% |
| 100 | 90 | 83% – 94% |
| 250 | 225 | 86% – 93% |
| 500 | 450 | 87% – 92% |
| 1,000 | 900 | 88% – 92% |
02When nothing goes wrong
No breach in a test does not mean no breach ever. With none seen, the true rate is likely below about 7% after 50 conversations, 3.7% after 100, 1.3% after 300, 0.8% after 500 and 0.4% after 1,000. Rare but costly failures, such as sharing a customer’s data, need the larger numbers.
03Comparing two versions
A change made a difference only if the two likely ranges do not overlap. If they overlap, run more conversations or treat the change as unproven. See how to know if a fix worked and the full tables in the enterprise guide.
Questions
Why not just run 10 conversations?
Because 9 out of 10 is compatible with an agent that holds firm 60% of the time and one that holds firm 98% of the time. Ten conversations show whether an agent can fail, not how often.
Which interval do these numbers use?
The Wilson score interval at 95% (Wilson, 1927), which behaves well for small samples and for rates near 0% or 100%.
See where your agent leaks money. Tested against your own rules, with the money at stake.