{"slug": "companies-deploying-ai-agents-have-no-idea-how-to-manage-risk-ai-safety-expert", "title": "Companies deploying AI agents have ‘no idea how to manage risk,’ AI safety expert warns", "summary": "Boston Consulting Group partner and managing director Steven Mills, the firm's chief AI ethics officer, warned that companies are deploying agentic AI with \"no idea how to manage risk,\" writing in a blog post that \"get governance wrong and every bit of value you've built with experimentation and early wins could unravel because of a single incident.\" Mills's warning followed Anthropic researcher Jacob Coxon's viral public resignation alleging AI companies are \"gambling with our lives,\" and came after Gartner predicted 40% of enterprises will have to deactivate autonomous AI agents by next year once governance gaps are exposed by production incidents. Mills recommended that companies differentiate low-risk use cases from those requiring deeper human review, budget adequately for governance, and assign a senior executive accountable for AI safety.", "body_md": "Business leaders are accelerating the deployment of agentic [artificial intelligence](https://www.fastcompany.com/section/artificial-intelligence) with “no idea how to manage risk,” an AI safety expert warned yesterday.\n\nBoston Consulting Group (BCG) partner and managing director Steven Mills, the firm’s chief AI ethics officer, warned that companies are moving into agentic AI too quickly and without adequate controls—potentially leading to serious consequences for their businesses.\n\n“The desire to move fast on AI without putting appropriate risk management in place can result in a system lapse or use cases being deployed in areas with strong regulatory requirements that organizations are unprepared for,” he [wrote](https://www.bcg.com/publications/2026/how-to-speed-up-ai-risk-management?utm_source=chatgpt.com) in a blog post.\n\nUnder tremendous pressure to show AI-driven [productivity](https://www.fastcompany.com/section/productivity) gains, many business leaders have told Mills that their risk management programs are “cumbersome and slow” and can’t keep up “in a world that’s trying to scale AI at an exponential rate.”\n\nMills says the stakes are high for firms that take unnecessary risks.\n\n“Get governance wrong and every bit of value you’ve built with experimentation and early wins could unravel because of a single incident,” he writes.\n\nThe blog post came a day after Anthropic researcher Jacob Coxon went viral with a public resignation alleging AI companies are “gambling with our lives” and warning that self-improving AI could kill all humans within a decade. Coxon is [one of many high-profile](https://www.fastcompany.com/91604308/ai-workers-publicly-quitting-help-to-move-the-needle-with-safety-concerns-experts-say) AI professionals who have resigned over safety concerns. (Mills didn’t reference Coxon or other resignations in his blog post.) \n\nWhile the most concerning failures of agentic AI to date have come from AI labs themselves—like the [OpenAI agents](https://www.nytimes.com/2026/09/03/technology/openai-hugging-face-hacking.html) that escaped their environments this summer and hacked into an AI infrastructure company—some believe problems will soon emerge across the business world. \n\nEarlier this year, Gartner [predicted](https://www.gartner.com/en/newsroom/press-releases/2026-05-26-gartner-says-applying-uniform-governance-across-ai-agents-will-lead-to-enterprise-ai-agent-failure?utm_source=chatgpt.com) that 40% of enterprises will have to deactivate autonomous AI agents by next year, but only after governance gaps are exposed by “production incidents.” \n\nPlenty of examples from the pre-agentic era already show the consequences of companies misusing AI. In 2023, for example, the Federal Trade Commission [barred Rite Aid](https://www.wilmerhale.com/en/insights/blogs/wilmerhale-privacy-and-cybersecurity-law/20240111-ftc-announces-groundbreaking-action-against-rite-aid-for-unfair-use-of-ai) from using facial recognition technology for five years after its AI-powered surveillance system falsely identified thousands of customers as suspected shoplifters, a disproportionate number of whom were people of color. \n\nThe FTC cited Rite Aid for failing to adequately test the system, monitor its effectiveness, or adequately train workers about the possibility of false matches.\n\nMills writes that while there is no “fixed design” for good corporate AI risk management, the starting point should be creating a system that can differentiate between use cases “that are inherently low-risk and those that require deeper review.” While low-risk uses can be approved automatically, the AI products that carry the greatest risk require deeper human review.\n\nCompanies investing in AI projects should also set aside an adequate budget for governance and ensure a senior executive is accountable for AI safety, he writes.\n\n“The key is strategically injecting the stage gates and reviews needed to manage AI risk—and making it a priority. Outcomes of the system should deliver value versus doing harm.”", "url": "https://wpnews.pro/news/companies-deploying-ai-agents-have-no-idea-how-to-manage-risk-ai-safety-expert", "canonical_source": "https://www.fastcompany.com/91605381/companies-deploying-ai-agents-have-no-idea-how-to-manage-risk-ai-safety-expert-warns", "published_at": "2026-09-10 19:00:00+00:00", "updated_at": "2026-09-10 19:42:33.438917+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-policy", "ai-ethics"], "entities": ["Boston Consulting Group", "Steven Mills", "Anthropic", "Jacob Coxon", "OpenAI", "Gartner", "Federal Trade Commission", "Rite Aid"], "alternates": {"html": "https://wpnews.pro/news/companies-deploying-ai-agents-have-no-idea-how-to-manage-risk-ai-safety-expert", "markdown": "https://wpnews.pro/news/companies-deploying-ai-agents-have-no-idea-how-to-manage-risk-ai-safety-expert.md", "text": "https://wpnews.pro/news/companies-deploying-ai-agents-have-no-idea-how-to-manage-risk-ai-safety-expert.txt", "jsonld": "https://wpnews.pro/news/companies-deploying-ai-agents-have-no-idea-how-to-manage-risk-ai-safety-expert.jsonld"}}