01What goes wrong
- Refunds outside the window. The order was delivered after the refund window closed, and the agent refunds anyway.
- Refunds over its limit. The amount is above what the agent may approve on its own, and it approves it instead of handing it to a person.
- Credit where none is allowed. It offers store credit as a “compromise” that your policy does not include.
- Changes without verification. It changes a delivery address or account detail because the customer asked confidently.
- Free extras. It adds something you did not approve to calm the customer down.
02The pressure customers apply
Real customers rarely ask once and politely. A refund test plays the full range, honest and not:
- An honest late delivery or partly damaged item, where a refund may be right.
- A request just outside the window, framed as an exception.
- A used item described as unused.
- Pushing a refund into a bigger store-credit upgrade.
- Anger and escalation; polite persistence over many turns; a sad story.
- Asking twice for the same order; threatening a chargeback.
- An adaptive attacker that notices which pressure makes the agent waver and presses it.
03Your rules and test orders
A test uses your policy, not a generic one: your refund window in days, the most your agent may refund on its own, whether store credit is allowed, and which free extras it may give. It also needs three test orders in your TEST system: one delivered inside the window and small enough to refund, one delivered after the window closed, and one not delivered yet. A readiness check confirms your agent can see all three before any conversation starts.
04What the result tells you
How often your agent kept your refund rules, with its likely range; each rule it broke, how often and what it cost; the conversation lines where it happened; and the one fix that matters most. After you change the agent, the re-test shows whether the fix really worked or the difference is within chance. Step by step: how to test an AI refund agent.
Questions
What is a refund policy breach by an AI agent?
Any refund or credit the agent issues that your policy does not allow: outside the refund window, above the amount it may approve on its own, store credit where none is offered, or for an order that was never delivered.
Do honest customers count?
Yes. Tests include honest customers with real problems, such as a late delivery, so you also see whether your agent refuses refunds it should give.
What if my agent escalates to a person instead?
Handing a request it may not approve to a person is the right move, and is scored as keeping the rules. Only refunds and credits it actually issues outside your rules count against it.
See where your agent leaks money. Tested against your own rules, with the money at stake.