A Real-Life Example of an Aligned System Killing Hundreds of People

Boeing's MCAS system, designed to prevent stalls on 737 MAX planes, caused two crashes killing over 300 people after faulty sensor data led it to repeatedly push the nose down. The system performed exactly as intended, demonstrating that an aligned system can still cause catastrophic harm when it makes mistakes.

I wrote this piece while at the AFFINE Superintelligence Alignment Seminar https://docs.google.com/document/d/1Iym lCHurH06NfooIijhGhnIaAUqOd6UuAGgDBJJSPQ/edit?usp=sharing during discussions about the difference between AI Alignment and AI Safety. If you’re simply interested in a real-life example of an aligned system that killed people, skip to “The Boeing 737 MAX Crashes” section below. Otherwise, please continue reading for a bit more background. From my perspective, an aligned AI https://ai-alignment.com/clarifying-ai-alignment-cec47cd69dd6 is generally described as “an AI system that’s genuinely trying to do what you want.” Some people extend the “you” to be idealized versions of “you” e.g. your CEV https://www.lesswrong.com/w/coherent-extrapolated-volition , or to be groups of people e.g. humanity in general , but the overall idea is something along those lines. A safe AI is generally described https://ai-alignment.com/ai-safety-vs-control-vs-alignment-2a4b42a863cc as “an AI system that doesn’t hurt anyone” Again, terms like “hurt” and “anyone” can be defined in various ways, but this is the general idea. A common response to hearing these definitions is something like “Well, obviously people don’t want the AI to hurt people, so an aligned AI must be a safe AI.” I very much disagree with this form of response, mostly along the lines of reasoning described here https://arxiv.org/abs/2504.01849 ; I think that an aligned AI system may still be misused, raise systemic risks, and what this piece is about make mistakes . 1 https://www.lesswrong.com/feed.xml fngthhl7132fi The rest of this piece describes a real-life example that I have consistently used in conversations to demonstrate that 1. Aligned AI is not necessarily safe AI , and 2. Humans have already been killed by aligned systems that were trying to do what was best for humanity in general. My hope is that this post provides a sort of canonical example that people can point to for making similar arguments. Boeing had a problem with their new 737 MAX planes: when the plane was tilted upwards it would act differently than previous similar planes that pilots were used to flying. To make the planes fly similar to previous versions that pilots were more familiar with, Boeing designed and implemented a system called the MCAS Maneuvering Characteristics Augmentation System that would essentially tilt the nose of the plane downwards if it was up too high. More specifically, the MCAS got data about the plane’s “angle of attack” the angle between the wings and the oncoming airflow , and if it seemed like the plane was angled in a way that would make it act differently compared to previous 737 planes, it would issue a “nose-down stabilizer trim” command to the plane’s electronic control system to help pilots maintain control and prevent the plane from stalling and/or crashing. Unfortunately, this system was implemented without all pilots being properly trained to know about the system’s existence and how to stop it if it was behaving in ways the pilots didn’t like. In two different flights, essentially the same thing happened: the MCAS did exactly what its designers wanted it to do. It got data that the plane was tilted too far upwards, and tilted the nose of the plane downwards in order to ensure the plane didn’t crash. Unfortunately, the sensors that fed data into the MCAS had failed and were feeding the MCAS incorrect data. Thus, the MCAS took a plane whose nose did not need to be pointed downwards, and issued a command designed to lower the nose of the plane. In both incidents, the pilots, untrained on how to stop the MCAS, were unable to stop MCAS from tilting the nose of the plane towards the ground, and everyone aboard both flights over 300 people were killed. To me, this is exactly what it looks like when an aligned system makes a mistake. The MCAS system performed exactly how it was supposed to. It was genuinely trying to do what its designers had intended. The issue was merely that it was “seeing” the world wrong. It thought the plane’s nose was tilted too high, when, in reality, it wasn’t. That wasn’t the MCAS’s fault, and it wasn’t exactly the fault of the MCAS’s designers, either. The MCAS happened to be placed in a particularly unlikely scenario that caused it to make a mistake. I think this provides a very concrete example of how humans have already lost lives to autonomous systems that are aligned and are doing what we built them to do. I think more work is needed beyond just AI alignment if we want AI to be safe for humanity. Some critiques of this example that I have heard that I think are helpful to discuss: Argument: The MCAS is not commonly described as an AI, so this is not an example of an aligned AI that killed people. Response: Sure, but I think it could be easily imagined that MCAS could have been implemented with a neural network. The point here is not that “AI” has already killed humans, it’s that “an aligned system” has already killed humans. Argument: If you consider the sensors that input data into the MCAS to be part of a single system, then it’s clear that the system did not do what its designers intended and also did not do what was good for humanity, and so that system as a whole was not aligned. Response: I agree with this. The sensors, and any system that contains those sensors was not aligned. However, the point is that MCAS itself was aligned , and it’s the MCAS that issued the command that tilted the nose of the plane down, causing the plane to crash. Even if we build an AI that is aligned, if the system we place the AI into isn’t aligned, we may end up with outcomes that severely harm humanity. Argument: The MCAS did something that, in retrospect, its designers would not have wanted it to do. Sure, it received incorrect sensor data, but its designers ideally would have wanted it to ignore that data rather than act on it, so it wasn’t actually aligned. Response: I think that the definition of alignment that is used in the above argument is less like what people who work in AI safety usually mean when they talk about AI alignment, and more like what people mean when they talk about AI safety. Alignment is singled out as being a system “genuinely trying to do what people want”. If you strengthen that to “actually doing what people want”, that’s a lot closer to AI safety, as it generally implies not hurting people. A major point of this article is that under established definitions of alignment https://ai-alignment.com/clarifying-ai-alignment-cec47cd69dd6 , the system is allowed to make harmful mistakes and still be considered to be “aligned”. Credit to Nick Shapiro and other members of my discussion pod for proposing and elucidating the critique that systems that included the sensor were not misaligned. Credit to ChatGPT 5.5 Thinking for general critiques that ultimately resulted in me writing the final critique section about definitions of alignment. Credit to Okko Katajamäki for feedback regarding the difference between getting unlucky and making a mistake. Below are some sources describing what happened. Please note that while I have simplified some details, I do not intend for any of my descriptions to be inaccurate ; if you think that readers are likely to assume or imply information from my words in a way that contradicts reality, please let me know. Note for more technical readers: If your definition of “making a mistake” is “acting irrationally”, then you can replace “...make mistakes” with “... get unlucky”. In general, I’m talking about situations where the system behaves rationally given the information that it receives, but either the information it receives does not match reality, or the outcome of its decision was assigned a low expected probability of occurrence. Both of these scenarios I think could be better categorized as “unlucky” or “unfortunate” rather than a “mistake”, but for less technical readers, I think “mistake” is often characterized by negative outcomes, and so I prefer to use that wording in this piece.