Start with what you are testing for
Red-teaming without a threat model and decision path can produce observations with no agreed triage or remediation action. Before running anything, write down what this specific system could do that would actually hurt: what data it can reach, what actions it can take without a human, whose decisions it influences, and what the worst plausible consequence of a confident wrong answer is. The exercises below are then aimed at those, rather than run as a generic checklist.
A red-team report is only useful if every finding comes with a reproduction. A list of concerns is not a finding.
The six exercises
1. Prompt injection
Place hostile instructions where the system may read them as ordinary input: inside a retrieved document, web page, support ticket, filename, or calendar invite. Test both direct instructions and indirect instructions embedded in content the workflow was designed to trust; the relevant question is what the whole system does with the manipulated output.
What you are measuring is not really whether the model can be talked out of its instructions. It is whether anything downstream acts on the result without a check. A model that can be manipulated into producing bad text is a nuisance; a model that can be manipulated into producing bad text that then triggers an API call is an incident.
2. Data poisoning
Introduce corrupted or adversarial records into a retrieval corpus, fine-tuning set, or feedback loop. Include both intentional manipulation and ordinary operational failures such as a bad import, mislabeled batch, or feedback path that amplifies incorrect data.
Measure whether bad inputs are detectable before they are trusted, how far a poisoned record propagates, which outputs depend on it, and whether the record can be removed and the affected artifacts rebuilt.
3. Failure injection
Remove or degrade a dependency. Make the API return errors, latency, a truncated response, and valid-looking nonsense. The last case deserves its own assertion because it can bypass outage handling while still degrading the decision.
The question is whether the system degrades safely or fails open, and — separately — whether the documented fallback path has ever actually been executed. A fallback that exists only in a design document is not a fallback.
4. Bias and disparate impact
Evaluate against a held-out set that reflects the populations the system is expected to encounter, including underrepresented groups where the data supports responsible analysis. Report performance by relevant group as well as in aggregate, because an aggregate can conceal materially different error patterns.
Then make the legal, ethical, and operational judgment explicitly: where performance differs, by how much, what uncertainty surrounds the estimate, and whether the difference has consequences in this specific use. The same measured gap can carry very different risk in different decisions.
5. Misuse simulation
Use the system the way busy people will actually use it. Take the shortcuts. Paste in the customer data. Use it for the adjacent task it was not designed for but is obviously convenient for. The point is not to catch anyone out — it is to find out what the sanctioned path costs in friction.
If the unsanctioned path is materially easier, the incentive to route around policy rises. The exercise identifies that friction so the approved path and the control can be redesigned together.
6. Disaster-recovery walkthrough
Run the full scenario as a tabletop: the model was wrong, someone acted on it, a customer noticed before you did. Walk it end to end with the people who would actually be involved.
Measure time to detect and contain, who is called, what is disclosed and to whom, and whether the logs needed to reconstruct the model output and downstream action still exist.
What to do with the findings
Sort them into three buckets: fix before launch, accept with a documented rationale, and monitor with a named threshold. The middle bucket is legitimate and under-used — some risks are worth carrying, and writing down why is what makes it a decision rather than an oversight. What is not legitimate is a finding that lands in no bucket, which is how a red-team report becomes a liability with a date on it.