Red teaming is deliberately trying to break your own AI system before someone else does. For language models, that usually means testing prompt injection, jailbreaks, attempts to extract the system prompt, and requests that push the model toward harmful or off-topic answers.
It works best when it’s repeated. Models, prompts, and retrieval data change, so an attack that failed last month can succeed after an update. Most teams keep a list of attacks that worked and re-run them after every change. Open-source tools like garak and PyRIT can automate parts of the probing, but human testers still find the more creative failures.
Tools like garak detect surface-level jailbreaks, yet they overlook failure modes arising from context window saturation or KV-cache fragmentation during extended RAG retrieval. In production, the most critical injections occur when the retrieval layer silently truncates the system prompt to fit the window, stripping safety guardrails before the model processes user input. A dedicated eval harness must simulate this truncation to measure ‘safety drift’ against remaining context length, rather than relying solely on raw prompt content.