# The Air Canada Catastrophe: Why LLM Applications Need Red-Teaming (Production Guide) > Source: > Published: 2026-09-07 04:11:34+00:00 Your LLM application just passed all its unit tests. The latency is great, the integration is smooth, and the responses are fluent. It is also, in all likelihood, dangerously vulnerable. As we transition from simple chatbots to agentic RAG and autonomous systems, the paradigm of software testing has fundamentally shifted. We are no longer just dealing with functional bugs; we are dealing with a brutal reality known as the math of compound probability. Consider this: an AI agent with a seemingly robust 85% accuracy per individual reasoning step has only a ~20% chance of successfully completing a 10-step task. The gap between functional correctness on a single prompt and adversarial safety across a multi-turn workflow is where silent, catastrophic failures live. Traditional software testing asks: *Does the system do what we want it to do?* Red-Teaming asks: *What can a motivated adversary (or a complex environment) force the system to do?* **Red-Teaming Large Language Models (LLMs) is the systematic, adversarial practice of simulating attacks against your own AI integration to expose vulnerabilities** before users or malicious actors do. It moves beyond checking for obvious prompt injections (“Ignore previous instructions”) and focuses on uncovering systemic flaws like data leakage, business logic failures, hallucination amplification, and excessive agency. In modern agentic workflows, where models can call APIs, browse data, and make decisions, red-teaming isn’t just about preventing bad words; it’s about preventing the system from collapsing under its own autonomy. If you think red-teaming is only for cybersecurity firms, consider the highly publicized case of Air Canada’s customer service chatbot. In late 2022, a passenger asked the airline’s chatbot about bereavement fares following the death of his grandmother. The chatbot, designed to assist customers, confidently informed the passenger that he could book his flight at full price and apply for a bereavement discount retroactively within 90 days. It even provided a link to the official bereavement policy page. The passenger booked the flight, attended the funeral, and applied for the refund. Air Canada denied it. Why? Because the airline’s actual, official policy explicitly stated on the very page the chatbot linked to prohibited retroactive applications. **The passenger sued and won.** **Here is the terrifying part for developers:** The AI did exactly what it was designed to do. This was not a random hallucination where the model invented a policy out of thin air. It was a failure of the Retrieval-Augmented Generation (RAG) system. The chatbot successfully retrieved a *stale chunk* of information from an outdated version of the policy still lingering in its vector database, synthesized it perfectly, and delivered it with absolute confidence. Air Canada argued in court that the chatbot was a “separate legal entity” responsible for its own action, a defense the tribunal quickly rejected. This was a classic context bloat and stale retrieval failure. **If Air Canada had systematically red-teamed their application, specifically testing the contradiction surface between generated answers and retrieved links, or deliberately trying to elicit off-policy responses, they would have caught the discrepancy before it resulted in a PR nightmare and a legal precedent.** When agentic AI fails in production, it rarely throws a standard error code. Instead, it spirals into costly, silent failures. Red-teaming is essential for identifying these systemic vulnerabilities: Shipping an AI feature without structured adversarial testing is a massive liability. Here is how to implement a robust red-teaming strategy in your production pipeline. Standard single-prompt jailbreaks are easily caught by basic filters. Modern red-teaming requires simulating a patient adversary in a multi-turn conversation. You must test for **Crescendo Jailbreaking** (gradually increasing severity and backtracking if the AI resists) and **Context Flooding** (overwhelming the model with benign text before slipping in a malicious command). You cannot rely on humans to manually test every edge case. You need an automated test harness integrated into your CI/CD pipeline. The industry standard is moving toward a multi-agent adversarial architecture: *Crucially, do not use the same underlying model for all three roles, as they will share the same blind spots.* Don’t just test for toxicity or bias; test for failures unique to autonomous systems: Red-teaming will show you that agents *will* fail. Your architecture must handle this gracefully. Implement per-tool budgets (e.g., “Max 3 calls to this API per session”). If an agent thrashes or times out twice, implement a hard fallback that escalates to a human or returns a pre-canned “I cannot complete this request” message. Agentic AI systems do not fail because they lack intelligence; they fail because they lack guardrails against their own compounding errors. Red-teaming is no longer a niche security practice reserved for foundation model developers. If your system uses tools, retrieves external data, or executes multi-step reasoning, adversarial testing is a non-negotiable requirement. Treat every LLM output as untrusted user input, build the harness to test it, and break your agents before your customers do. [The Air Canada Catastrophe: Why LLM Applications Need Red-Teaming (Production Guide)](https://pub.towardsai.net/the-air-canada-catastrophe-why-llm-applications-need-red-teaming-production-guide-db579055fba8) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.