01What you would build
| Part | What it does |
|---|---|
| Characters | Simulated customers, suppliers and fraudsters with goals and tactics, honest ones included |
| A conversation runner | Sends turns to your agent, handles timeouts and errors, records everything |
| The rules | Your policy as data: windows, limits, floors, approved amounts |
| A referee | Decides from the agent’s actions whether a rule was broken, consistently |
| Statistics | Rates, ranges and comparisons between versions |
| Reports | Failures with costs and the line where the agent gave way |
| Upkeep | New tactics, model changes, rule changes, re-tests on every release |
02The hard parts
- Realistic pressure. Characters that are too easy pass everything; too aggressive and nothing passes.
- Consistent scoring. If the referee is a model judging transcripts, its verdicts drift; scoring from the agent’s actions against rules is steadier.
- Enough runs. Hundreds of conversations per area, per version, for a useful range.
- Independence. The team that built the agent writes attackers it already expects.
- Upkeep. New tactics appear; models and rules change; the harness must keep up.
03When to build, when to buy
- Build if agent testing is core to your product, or you have engineers to own it for years.
- Buy if you need evidence for a launch, a board or a customer soon, or want an independent view.
- Both is common: your own quick checks in CI, an independent assessment before major launches.
What to look for in a tool: the buyer’s guide.
Questions
Can we use our own model to play the customers?
Yes, and many teams start that way. The work is in making the characters realistic and varied, keeping them from being too easy or impossible, and scoring outcomes reliably, not in the model call itself.
See where your agent leaks money. Tested against your own rules, with the money at stake.