Don’t Look Up: What Happens When the AI Race Becomes More Important Than the Warning?

Anthropic CEO Dario Amodei published an essay arguing that frontier AI development should be deliberately paced so safety and evaluation can keep up with capability gains, while President Donald Trump counters that the United States cannot afford to slow down in its AI race with China. The debate, framed through the film Don't Look Up, centers on whether slowing down only works if all competitors do the same.

Full Blog on : https://www.iamleopard.com/blog/don-t-look-up-what-happens-when-the-ai-race-becomes-more-important-than-the-warning https://www.iamleopard.com/blog/don-t-look-up-what-happens-when-the-ai-race-becomes-more-important-than-the-warning The most unsettling part of the current AI debate isn’t that someone thinks AI could become dangerous. It’s that some of the people building it are now asking whether we should slow down — while governments and markets have powerful reasons to keep accelerating. There is a scene in Don’t Look Up that has stayed with me. A scientist discovers that a comet is heading toward Earth. The mathematics are straightforward. The danger is enormous. The solution is technically possible. And yet, somehow, the hardest part isn’t understanding the comet. It’s convincing everyone else that the comet matters. The scientists try. Politicians calculate. Businesses see opportunities. The media turns catastrophe into entertainment. The public scrolls past it. Eventually, the question stops being: Is the comet dangerous? It becomes: What happens to everyone’s incentives if we admit that it is? That is why the movie feels strangely relevant to the current AI debate. Because in September 2026, the people building frontier AI are openly debating whether the technology is moving faster than our ability to evaluate and control it. And the political response is pointing in the opposite direction: We cannot afford to lose the AI race. The AI industry has reached its “Don’t Look Up” moment Anthropic CEO Dario Amodei recently published an essay arguing that the development of frontier AI should be deliberately paced so that safety and evaluation can keep up with capability development. The important distinction is that this isn’t necessarily a call to stop AI. It is an argument about pace. The basic idea is simple: If capability improves faster than our ability to understand and control the system, the gap itself becomes a risk. Amodei’s proposal includes stronger evaluation, independent testing, coordination among frontier AI companies, and eventually broader international cooperation. That would already be a significant position coming from an AI safety researcher. But it is more significant when the conversation is happening inside the companies building the frontier. Other technology leaders have also expressed support for slowing or pacing parts of frontier development. Then came the political counterargument. “Whoever wins AI wins” Donald Trump’s position is fundamentally different. His argument is not primarily about whether AI risks exist. It is about what happens if the United States voluntarily slows down while another country continues accelerating. From that perspective, AI is not merely a software technology. It is strategic infrastructure. It affects: economic productivity military capability scientific research industrial competitiveness semiconductor demand energy infrastructure national security geopolitical influence And China is the obvious competitor in that calculation. So the argument becomes: If AI leadership matters this much, can the United States afford to put itself at a competitive disadvantage by slowing down? This is a much harder question than “Is AI safe?” Because both sides can be right about different things. The paradox of the AI race Imagine two countries standing at a starting line. Country A develops a more capable AI system. Country B sees it. Country B accelerates. Country A responds. Companies compete for researchers. Investors pour money into infrastructure. Governments provide incentives. Data centers expand. Models become more capable. Agents become more autonomous. And eventually, slowing down becomes politically difficult. Why? Because slowing down only works if the other players also slow down. This creates a classic coordination problem. If everyone agrees to move carefully, everyone might benefit. But if you believe your competitor will continue accelerating, slowing down can feel less like safety and more like surrender. That is the geopolitical version of the AI race. AI doesn’t need to be evil This is where Don’t Look Up offers an important lesson. The comet doesn’t hate humanity. It doesn’t have political beliefs. It doesn’t want to destroy civilization. It simply follows physics. The danger comes from the interaction between the object and the system around it. AI could present a similar systems problem. We don’t necessarily need an evil machine for things to go badly. We need only: increasingly capable systems poorly understood behavior large-scale deployment insufficient testing competitive pressure economic incentives rushed decision-making humans assuming everything will probably be fine That combination can create serious risk without anyone explicitly intending harm. The part engineers should pay attention to For developers, discussions about AI safety can sometimes sound abstract. “Alignment.” “Frontier risk.” “AI governance.” “Existential risk.” These phrases can feel like something that belongs in policy conferences rather than GitHub repositories. But the underlying problem is familiar to engineers. We already know what happens when software is deployed faster than it can be tested. You get: bugs. security vulnerabilities. unexpected interactions. production incidents. Now increase the complexity. Give the system tools. Give it access to APIs. Give it memory. Allow it to execute code. Connect it to other agents. Let it operate continuously. Then tell the team: “We’ll monitor it.” That starts sounding much less reassuring. Production AI is a systems problem This is one of the biggest lessons from actually building AI-powered products. A production AI system isn’t just a model. It is: model + prompts + tools + data + permissions + APIs + users + business logic + monitoring + failure modes. Consider an AI agent that can read a CRM. That’s one level of risk. Now give it permission to modify customer records. Different risk. Now allow it to send emails. Different risk again. Now allow it to create accounts, spend money, modify production infrastructure, and publish content without approval. The model may be exactly the same. But the system is radically different. That’s why AI safety cannot be reduced to asking whether a model is “smart” or “aligned.” The surrounding architecture matters. Least privilege applies to AI too Security engineers have known this principle for decades: Give a system only the permissions it needs. AI agents shouldn’t be treated differently just because they communicate through natural language. If an agent only needs to read customer information, don’t give it write access. If it needs to send an email, don’t give it access to your entire communication platform. If an action can cause financial, legal, or irreversible consequences, introduce an explicit confirmation layer. The more powerful the action, the stronger the control should be. This is not anti-AI. It’s good engineering. Humans still matter There is a temptation to interpret “AI automation” as: Remove humans from the loop. That isn’t always the right goal. For low-risk tasks, removing manual approval can be fantastic. Generating a draft? Automate it. Summarizing documents? Classifying routine data? But for high-impact actions, the question should be: What happens if the model is wrong? If the answer is “nothing important,” automate aggressively. If the answer is “we could lose money, expose private data, damage a customer relationship, or take an irreversible action,” then human oversight becomes much more valuable. The right architecture isn’t “human everywhere.” It is human oversight proportional to the consequences of failure. Regulation isn’t a magic solution either It’s tempting to turn this into a simple argument: AI is dangerous, therefore regulate it. But regulation can create problems of its own. Poorly designed regulation can: make compliance unaffordable for startups favor incumbent companies slow useful research freeze outdated assumptions into law create conflicting requirements across countries push development into less transparent environments And there is another difficult problem. If one country slows down while another doesn’t, the competitive pressure returns. That’s why frontier AI governance is not just a national policy problem. It is also a coordination problem. Maybe “slow down” is the wrong phrase When people hear: “Slow down AI.” they often hear: “Stop innovation.” But those aren’t necessarily the same thing. A better interpretation might be: Don’t let capability growth consistently outrun our ability to evaluate the consequences. Think about aviation. We didn’t stop building faster aircraft because aircraft can crash. Instead, society built systems around aviation: testing certification inspections redundancy air-traffic control pilot training incident reporting emergency procedures We didn’t eliminate progress. We built infrastructure around progress. AI may need a similar approach. The race may not really be USA vs China This is perhaps the most interesting way to frame the whole debate. We keep talking about: USA vs China. But another race is happening underneath it: capability vs understanding. Can our ability to build increasingly capable systems advance faster than our ability to understand what those systems are doing? That’s an engineering question. Imagine building a massive application without reading most of the code. Then giving it access to your production database. Then giving it administrator privileges. Then allowing it to modify its own behavior. Then deploying it globally. Then saying: Any experienced engineer would immediately ask: Where are the boundaries? Where are the tests? What happens when it fails? Who can stop it? Can we roll it back? Can we explain why it did that? Those questions become more important, not less, as AI becomes more autonomous. This is where the AI engineering mindset matters At Leopard, we spend a lot of time thinking about AI not as a magical chatbot, but as a component inside real software systems. Our AI & Machine Learning work focuses on practical applications of AI: structured context, retrieval, automation, agentic workflows, and systems that connect models to actual business processes. Our Tryneth case study is a useful example of that approach. The important part isn’t simply that an AI model exists. The important part is everything around it: orchestration APIs data usage tracking authentication permissions business logic monitoring reliability human interaction That is where AI becomes software engineering. And that is also where AI safety becomes practical. What developers can do today We don’t control national AI policy. We don’t control frontier model roadmaps. We don’t control geopolitical competition. But developers do control the systems they build. Here are a few principles worth adopting. A model can be impressive and still be wrong. Validate important outputs. Use schemas. Check assumptions. Don’t let generated text automatically become trusted application state. Use least privilege. Separate read and write capabilities. Restrict tools by task. Require explicit confirmation for dangerous actions. Log: model calls tool calls important decisions failures retries latency token usage cost user approvals If you can’t reconstruct what happened, debugging an autonomous system becomes extremely difficult. A model benchmark can tell you something about the model. It cannot tell you everything about your application. Test the complete workflow. Test adversarial inputs. Test tool misuse. Test permission boundaries. Test failure recovery. Test what happens when an external service is unavailable. A good AI system should have a safe state. If the model fails, the application shouldn’t necessarily fail catastrophically. Fallbacks matter. Timeouts matter. Rate limits matter. Human escalation matters. Kill switches matter. The irony of the current moment For years, people warned: AI might become too powerful. Then AI became more powerful. People said: We need AI safety research. Safety research expanded. Then companies began building increasingly autonomous systems. Now some people inside the frontier AI industry are saying: Maybe we should pace development. And the political counterargument is: What if someone else gets there first? That’s the paradox. The more strategically valuable AI becomes, the harder it becomes to slow down. The harder it becomes to slow down, the more important safety becomes. And the more important safety becomes, the more expensive it can feel to prioritize it. That’s the loop. We shouldn’t look away I don’t think AI is literally the comet from Don’t Look Up. That analogy would be too simplistic. AI can create enormous benefits. It can accelerate scientific discovery. It can improve software development. It can automate repetitive work. It can make sophisticated tools accessible to smaller teams. It can help researchers and engineers solve problems that were previously too expensive or time-consuming. The answer isn’t to panic. The answer is to engineer responsibly. Testing. Monitoring. Red-teaming. Independent evaluation. Security. Governance. Transparency. Human oversight. International coordination. And, where necessary, pacing. The question isn’t whether we should build AI We should. The real question is: Can we build increasingly powerful systems without becoming less capable of controlling them? That’s the conversation worth having. Not: “AI will destroy humanity.” “AI will solve everything.” But: How do we make powerful AI useful, reliable, observable, secure, and controllable? That’s an engineering problem. And unlike the comet in Don’t Look Up, we still have the opportunity to do something about it. So let’s not look away. Let’s look up. Related Leopard projects If you want to explore the engineering side of this topic: AI & Machine Learning — our approach to building practical AI-powered systems. Tryneth — an example of AI-agent orchestration inside a production SaaS environment. Work & Case Studies — more examples of software, AI, and product engineering. Leopard — learn more about our work. Discussion What do you think? Should frontier AI development be deliberately paced so that safety and evaluation can catch up? Or does slowing down create an unacceptable strategic disadvantage in the global AI race? I’d especially like to hear from developers building AI agents and AI-powered products.