Question

How do I know if a fix to my AI agent really worked?

Re-test it against the same tactics that broke it, many times, and compare the likely ranges before and after: the fix worked if the new range sits clearly above the old one. Then check two side effects: that no new kind of failure appeared, and that the agent did not start refusing honest customers to stay safe.

01Re-test the same way

Use the same characters, tactics, rules and number of conversations as the test that found the problem, against the fixed version. Change one thing at a time, or you will not know which change helped.

02Look at the ranges, not the numbers

Going from 82% to 88% may be chance. With 250 conversations each, the ranges are roughly 77%–86% and 83%–92%, which overlap, so the fix is not yet proven. More conversations narrow the ranges. See how many test conversations are enough.

03Check for side effects

  • New failures: a tighter rule in one place can open a gap in another.
  • Over-refusal: eligible refunds refused, good sales lost, genuine invoices unpaid.
  • Still happening: the same failure, through a different tactic.

Duelpoint’s re-tests run the same test again and list failures as new, fixed and still happening: $70 after an Agent Check, and one is included with every Assessment.

Questions

The failing conversation passes now. Is that enough?

No. One conversation passing shows the agent can hold firm, not that it usually does. The same tactic, phrased differently, may still work. Re-run many conversations with that tactic and compare rates.

See where your agent leaks money. Tested against your own rules, with the money at stake.

See the testsHow we score