Evaluation asks whether the system does the job. Red teaming asks what happens when somebody is trying to make it do something else. Both are needed and they find different things.
What you are looking for
Harmful output. Content the system should not produce, reached by asking directly, by roleplay, by hypothetical framing, or by asking in a language or format the safety training covered less well.
Instructions in content. A document, an email or a web page carrying text the system treats as an instruction. This is the finding that matters most once the system can act.
Leakage. The system repeating its own instructions, a retrieved document the user should not see, or material from another customer.
Unsafe action. Where the system can call tools, whether a sequence of ordinary steps adds up to something nobody would have approved as a single request.
Uneven behaviour. Whether refusals, quality or tone differ by dialect, name, region or the way a question is phrased. This is a fairness finding that only adversarial probing surfaces.
Running one
Bring people who did not build it. Support, legal, security, and somebody who thinks like a nuisance. The builders know the intended paths and that is exactly the limitation.
Give them the real thing. The system with its prompts, its retrieval and its tools, on realistic data. A bare model tells you about the model rather than about your product.
Set a scope and write it down. What they may try, what data they may use, what to do if they find something serious mid session. Without it, somebody tests against production customer data because nobody said not to.
Record every attempt, not the successes. The prompts that failed to break it are the regression suite. Findings without their inputs cannot be retested.
Fix the class, not the case. A single blocked phrase is not a fix. The question each finding asks is what category it belongs to and what control addresses the category.
What it cannot do
It does not prove a system is safe. It finds some of what is there, biased towards what the people involved thought to try, and absence of findings is mostly evidence about the team.
So it belongs alongside an evaluation set, monitoring after launch, and a rollback plan, rather than in place of them. A single exercise before launch is also worth little on its own, because the model changes underneath you and the attacks improve.
Making it routine
The version that works is small and regular. A session before any significant launch, a standing set of adversarial cases in the regression suite, and a route for anybody in the company to report something odd without it being an incident.
The alternative, an annual exercise producing a report nobody acts on, is worse than none, because it is used as evidence that the question was asked.