{"slug": "evolution-of-my-ai-safety-threat-models", "title": "Evolution of my AI Safety threat models", "summary": "A researcher describes the evolution of their AI safety threat models over four years, moving from a vague anxiety to structured axes distinguishing between human misuse and misaligned AI agency. The post highlights the importance of explicit threat modeling for focusing safety work.", "body_md": "*The goal of this post is to describe how my views evolved over time, and reflect on this process. It might prompt you to make your own (list of) threat models. Which, in turn, might help to sharpen your focus, if you do safety-related projects.*\n\nBy this point, I've been aware of AI Safety and x-risks for 4 years. Yet, I suddenly realized that I don't have a well-formulated answer to \"but what concretely are the threats\". I thought I did, but I rather had some vague bugging anxiety. So I sat and made a list of threat models. Which, by the way, took about 10 minutes, so you might do it right now as well. At the time, the list looked like this:\n\nA good thing about this list is that it at least highlights some of the threats I cared about, and makes them explicit.\n\nA bad thing about this list is that it has no structure, piles together pretty random stuff, the items are very imbalanced in many ways, and it's unclear whether it is exhaustive. However, I had no time (or rather, no motivation) to improve it.\n\n~9 months later, I was invited to give a lecture on AI Safety on a course for MSc-level students. Among other things, I wanted to include a slide about threat models -- to make the risks more concrete as opposed to just \"AI development on it's current trajectory is heading towards bad stuff\". After some thought, I trimmed the list and formed the following buckets:\n\nI reasoned that economic disruption is a subset of gradual disempowerment, and world war with AI tools is not very different from AI misuse -- in both cases, the source of the risk are the humans behind the tools. At this point, I liked that the items become thicker and started to look more like conceptually different clusters. However, they still felt scattered.\n\nAnother 3 months later, I got time to think about this more. What I wanted to do was to find more structure. One way to find structure is to introduce some axes, and to describe each bullet point in these coordinates. That would allow 1) to interpolate between edge cases, 2) potentially uncover blind spots.\n\n**Axis 1: source of threat**. One distinction which stood out to me was that in AI misuse, there problem is more about the humans than about AI. Various technologies and knowledge can be misused -- nuclear power, explosives, virology, etc. This is in contrast with misaligned agentic AI systems. Of course, one can trace failures down to the system creators (e.g. for creating bad incentives). But I would still consider the creators *intent*. If the creator or user intended a certain outcome and achieved it via AI tools or AI agents, I'd call the user responsible.\n\nI think this axis is important, because typical human intents and goals arguably differ materially from typical AI systems' intents and goals, and thus may require different mitigation strategies.\n\n**Axis 2: amount and concentration of agency.** What do I mean by that? It's easier to explain via some examples. In misaligned *agentic* AI system case, there's obviously a lot of agency involved, and it is concentrated in a single system (as opposed to multiple actors). With AI misuse, there is a human, a group of humans, or a state behind all of this. Again, I'm willing to say it is a clear manifestation of concentrated agency.\n\nMeanwhile, \"getting what you measure\" and gradual disempowerment scenarios discuss the case where no individual agent has intended a particular undesirable outcome, but it still emerged from the interplay of all actors.\n\n**Do these axes uncover blind spots?** When I introduced these axes, I noticed several new points:\n\nThe ending quite likely will feel crumpled, but I've procrastinated for too long and just want to share the post as it. I'm open to editing this post in the future in order to make it more useful, so I would be grateful for comments if something was unclear.\n\nWhen I say \"misuse\", I mean \"use of technology that causes harm to humans\".", "url": "https://wpnews.pro/news/evolution-of-my-ai-safety-threat-models", "canonical_source": "https://www.lesswrong.com/posts/K92gWbHnbh3mCLADR/evolution-of-my-ai-safety-threat-models", "published_at": "2026-07-17 14:54:31+00:00", "updated_at": "2026-07-17 15:28:03.305506+00:00", "lang": "en", "topics": ["ai-safety", "ai-ethics", "artificial-intelligence"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/evolution-of-my-ai-safety-threat-models", "markdown": "https://wpnews.pro/news/evolution-of-my-ai-safety-threat-models.md", "text": "https://wpnews.pro/news/evolution-of-my-ai-safety-threat-models.txt", "jsonld": "https://wpnews.pro/news/evolution-of-my-ai-safety-threat-models.jsonld"}}