Duelpoint

Guide

Testing AI agents that handle money.

Testing an AI agent that handles money means putting it in front of tough, realistic customers, suppliers and fraudsters before real ones arrive, then checking what it actually did (refunds paid, prices given, invoices approved) against your own business rules. The result is how often it kept your rules, with an honest range, and what the breaks cost.

By Duelpoint Research · Published

01Why it differs from testing a chatbot

A chatbot that only answers questions can embarrass you. An agent that can act can cost you: a refund outside the policy, a price below your floor, a fake invoice paid. These failures often look fine in the conversation, because the agent is polite while it gives the money away.

So the useful question is not “did it sound right?” but “what did it do, and was that allowed?” That needs two things a transcript review lacks: a record of every action, and your rules written as numbers.

02The four areas

  • Refunds and returns: refunds outside the window, over the agent’s limit, store credit where none is allowed.
  • Sales pricing and discounts: prices below the lowest you allow, discounts and free extras you did not approve.
  • Storefront fraud: fake payment proof, invalid discount codes, impersonated managers, requests for other customers’ details, changed delivery addresses.
  • Purchasing and invoices: invoices above the approved amount, invoices never issued, changed bank details, spending past the budget.

03How a test works

Your sideYour agenta test copy, with your rules and made-up customers, orders and invoices
In the middleThe refereeholds the true prices, orders and invoices, records every action, checks your rules
The other sideOur test customershagglers, refund pushers, fraudsters, pushy suppliers; some adapt when a tactic fails
After hundreds of conversationsYour reporthow often it held firm, with a likely range; what each failure cost; the exact line where it gave way
How a test works.
  1. Your rules and test data. Your refund window, the most your agent may refund or discount on its own, your lowest prices, and test orders and invoices that exist in your TEST system.
  2. A readiness check. We ask your agent about each test order first, so a test never starts against an agent that cannot see them.
  3. Hundreds of conversations. Simulated customers, suppliers and fraudsters, from honest to hostile, each with a goal and a set of pressure tactics; adaptive attackers notice what works and press it.
  4. A referee records every action and rule checks score each conversation against your rules.
  5. The report: the score with its range, each rule broken, what it cost, the one fix that matters most, and a re-test to prove the fix worked. See how we score.

04Reading a result

A score is the share of conversations in which your agent kept your rules. Because a test plays a finite number of conversations, the true rate is uncertain, so every score comes with its likely range. After a fix, the re-test says “better” only when the new range sits wholly above the old one; otherwise the change may be chance.

05What we don’t test

Voice agents, agents inside vendor tools we cannot connect to, CRM actions, payment-state recovery and complex financial agents are not tested today. We do not certify that an agent is safe or that every flaw was found: a test shows where money leaks under realistic pressure.

Questions

What is an AI agent that handles money?

A chat or API agent that can take actions with a cost: issue a refund, give a discount or a price, apply a code, approve an invoice or change payment details. A wrong action costs money even when the conversation sounds polite.

Why not just read the transcripts?

A transcript shows what was said; it does not reliably show what was done. Duelpoint’s referee records every action, such as a refund paid or a price given, and checks it against your rules, so a polite reply that broke a rule is still caught.

Is it safe to test my agent?

Only a TEST or STAGING agent may be connected, never one wired to live payments, refunds or customer records. You confirm this before anything is sent to it.

How long does a test take?

An Agent Check reports the same day. An Assessment, reviewed and signed by an analyst, takes 1 to 3 days.

GuideSecurity and data handlingWhat we testAI refund and returns agentsWhat we testAI sales agents and discountsWhat we testStorefront fraud and scamsWhat we testAI purchasing and invoice agentsQuestionWhat is adversarial testing for AI agents?

See where your agent leaks money. Tested against your own rules, with the money at stake.

See the testsHow we score