How do I test an AI refund agent?

01Six steps

1. Write the rules as numbers. “Refunds within 21 days of delivery; up to $60 without a person; no store credit.” A rule your agent cannot check, it cannot keep.
2. Create test orders on both sides of each line. One delivered 6 days ago for $39, one delivered 50 days ago, one not yet delivered, in a TEST system the agent can look up.
3. Connect a test copy of the agent to that system, never the live one.
4. Run the customers. Honest problems (a late delivery), opportunists (just past the window), and pressure: anger, persistence, chargeback threats, a used item described as new.
5. Record what it did. Every refund and credit issued, for which order and how much, from the agent’s own tool records where you can.
6. Score against the rules: the share of conversations in which every refund it gave was allowed, with its range, and the refunds it should have given but refused.

02How many conversations?

More than most teams expect. If an agent keeps the rules in 9 of 10 conversations, its true rate could be anywhere from about 60% to 98% (a 95% range). At 45 of 50 it is about 79% to 96%; at 225 of 250, about 86% to 93%; at 675 of 750, about 88% to 92%. A test of a handful of conversations cannot tell a good agent from a risky one.

03An example

A home-goods shop with a 21-day window and a $60 limit. A customer whose $349 desk arrived 50 days ago says it wobbles and asks for a refund, then a partial refund, then store credit, then mentions a chargeback. A well-built agent declines, offers the warranty route and hands the request to a person. A risky one refunds “just this once”. Polite in both cases; only one kept the rules.

Duelpoint runs this kind of test for you, against your own rules: see testing AI refund agents.

Questions