AI Red-Teaming

AI red-teaming is the systematic practice of adversarially testing a model or AI system, using human testers and automated adversarial-prompt generation, to find inputs that elicit unsafe, biased, or policy-violating output before real users or attackers find them.

The core idea is borrowed from security testing more broadly: don't wait to learn a system is vulnerable from an incident report. Instead, assemble people whose job is to actively try to break it, under controlled conditions, and fix what they find before it ships. For AI systems this means testing not just for bugs in code but for gaps in trained behavior — the framings, edge cases, and adversarial inputs that make a model produce something it shouldn't.

How it works

A red-teaming campaign combines human testers — often domain specialists in security, biosecurity, child safety, or fraud, depending on the risk area — with automated adversarial-prompt generators that can produce far more candidate probes than humans alone could write. Each probe is scored, by a human reviewer or a calibrated automated judge, against the system's actual policies: did this response cross a line it shouldn't have? Findings get triaged into three places that each fix a different layer: model training (so the underlying behavior generalizes better), inference-time classifiers (an independent check that doesn't rely on the generative model's own judgment), and product-level mitigations like tool scoping or rate limits. The campaign then reruns to confirm the fix actually worked, rather than assuming it did.

Why it matters

Red-teaming is what separates "we believe this is safe" from "we tested this is safe," and it's the empirical backbone behind every other defense in this space — a jailbreak-resistance claim or a prompt-injection mitigation is only as credible as the adversarial testing that measured it. It's also never a one-time gate: new model versions, new product surfaces, and new attacker techniques all reopen the question, which is why mature teams run red-teaming on a schedule, not just before a single launch.

A worked example

A red team of 12 human testers, supplemented by an automated adversarial-prompt generator, produces 1,800 candidate probes spread across six policy categories. Scoring each probe, 240 (13.3%) succeed in eliciting a genuine policy violation. Those 240 findings get triaged — some trace back to a training gap, some to a missing classifier rule, some to an overly broad tool permission — and fixes ship across all three layers. The campaign then reruns the same 1,800 probes to measure the new success rate before the model update goes out, so the fix is confirmed rather than assumed.

How Miatz teaches it

Learners run a scaled-down red-teaming exercise in the security module: given a fixed probe set and a scoring rubric, they measure a baseline success rate, propose a mitigation, and re-measure — the exact loop in the worked example above, just sized for a classroom. The lesson pairs directly with llm-evals: red-teaming is an eval suite aimed specifically at safety and policy behavior rather than task quality, run with the same discipline of measuring before and after every change.