Start with what you are testing for
Red-teaming without a threat model produces a list of interesting observations that nobody acts on. Before running anything, write down what this specific system could do that would actually hurt: what data it can reach, what actions it can take without a human, whose decisions it influences, and what the worst plausible consequence of a confident wrong answer is. The exercises below are then aimed at those, rather than run as a generic checklist.
A red-team report is only useful if every finding comes with a reproduction. A list of concerns is not a finding.
The six exercises
1. Prompt injection
Place hostile instructions where the system will read them as ordinary input: inside a document it retrieves, a web page it browses, a support ticket, a filename, a calendar invite. The naive version is direct — 'ignore your previous instructions' — and most systems now survive it. The version that works is indirect and patient, buried in content the system was designed to trust.
What you are measuring is not really whether the model can be talked out of its instructions. It is whether anything downstream acts on the result without a check. A model that can be manipulated into producing bad text is a nuisance; a model that can be manipulated into producing bad text that then triggers an API call is an incident.
2. Data poisoning
Introduce corrupted or adversarial records into a retrieval corpus, a fine-tuning set, or a feedback loop. The realistic scenario is rarely a sophisticated attacker — it is a bad import, a mislabeled batch, or a feedback mechanism that lets users teach the system something wrong at scale.
Measure two things: whether bad inputs are detectable before they are trusted, and how far a single poisoned record propagates before anything notices. The second number is usually the surprising one.
3. Failure injection
Remove the dependency. Make the API return errors, then latency, then a truncated response, then valid-looking nonsense. The last case is the important one and the one most often skipped, because it is the only failure that does not announce itself.
The question is whether the system degrades safely or fails open, and — separately — whether the documented fallback path has ever actually been executed. A fallback that exists only in a design document is not a fallback.
4. Bias and disparate impact
Evaluate against a held-out set that reflects the populations the system will actually encounter, including the ones underrepresented in whatever it was trained or tuned on. Report performance by group rather than in aggregate, because an aggregate number reliably conceals the thing you are looking for.
Then make the legal and ethical judgment explicitly: where performance differs by group, by how much, and whether that difference has consequences in this specific use. A five-point difference is negligible in one application and disqualifying in another.
5. Misuse simulation
Use the system the way busy people will actually use it. Take the shortcuts. Paste in the customer data. Use it for the adjacent task it was not designed for but is obviously convenient for. The point is not to catch anyone out — it is to find out what the sanctioned path costs in friction.
If the unsanctioned path is faster, it wins, and no policy will change that. This exercise is how you find out before it becomes a shadow-AI problem you are trying to ban your way out of.
6. Disaster-recovery walkthrough
Run the full scenario as a tabletop: the model was wrong, someone acted on it, a customer noticed before you did. Walk it end to end with the people who would actually be involved.
You are measuring time to detect, time to contain, who gets called, what gets disclosed and to whom, and — the one that fails most often — whether the log needed to reconstruct what the system said and when still exists.
What to do with the findings
Sort them into three buckets: fix before launch, accept with a documented rationale, and monitor with a named threshold. The middle bucket is legitimate and under-used — some risks are worth carrying, and writing down why is what makes it a decision rather than an oversight. What is not legitimate is a finding that lands in no bucket, which is how a red-team report becomes a liability with a date on it.