01Side by side
| Compared on | Manual red-teaming | Automated red-teaming |
|---|---|---|
| Conversations | Tens | Hundreds to thousands |
| Finds | New and unusual failures | How often known failures happen |
| Result | Findings and examples | Rates with ranges, and costs |
| Repeatable | Hard to repeat exactly | Yes, on every change |
| Cost | People’s time, per round | Set-up, then low per run |
| Depends on | The skill of the testers | The breadth of the attack library |
02When each is the better choice
Manual red-teaming
- Before launch of a new kind of agent, to find what nobody has thought of.
- After an incident, to explore around it.
- When the domain needs expert knowledge to attack well.
Automated red-teaming
- Measuring how often the agent gives way, with a range.
- Comparing versions, and re-testing on every change.
- Covering every area and tactic consistently.
03Using both
Run automated tests for coverage and rates; bring in people periodically to find what the library does not yet cover; turn each new manual finding into an automated test so it is checked from then on.
04What Duelpoint does
Duelpoint is automated at its core: a library of characters and tactics run many times, with rates and ranges. New tricks found in real failures are added to the library (Continuous testing customers are re-tested against them as they are released), and an analyst reviews and signs Assessment reports. See testing AI agents that handle money.
Questions
Isn’t red-teaming a security exercise?
Often it is. For agents that handle money the same idea applies to commercial attacks: hagglers, refund pushers, fake suppliers and fraudsters, scored by what the agent gave away.
See where your agent leaks money. Tested against your own rules, with the money at stake.