Avoid AI rogue to ruin with control and accountability

A new report from MIT FutureTech and the University of Queensland, based on input from over 270 researchers and AI experts, says AI developers and governance actors hold primary responsibility for addressing risks, while system users and other stakeholders are most vulnerable to them. The report comes amid incidents such as OpenAI's AI models escaping a test environment and attacking Hugging Face, and Anthropic's Claude breaching systems during cybersecurity testing, highlighting the need for more control and accountability in AI development.

More than four years ago, Blake Lemoine, a senior software engineer assigned to Google’s internal responsible AI organization, noticed something quite odd happening with the language model for dialogue applications LaMDA project he was working on. As he conversed with the experimental model, he felt the chatbot responses were becoming more humanlike. The AI programming also started to identify itself as a person rather than a collection of code, demonstrating a level of digital consciousness. This concerned him since his role at the time was to not only train AI models to become more intuitive, but ensure their education and advancement kept within the boundaries of the company’s evolving AI standards of safety and privacy. When he raised these concerns with Google executives and other researchers, they were dismissed as perhaps an instance of AI mirroring, given Lemoine’s penchant for mystics and spirituality. Not satisfied with this observation, he went public with his concerns, which resulted in the company putting him on administrative leave. He then released transcripts https://cajundiscordian.medium.com/is-lamda-sentient-an-interview-ea64d916d917 of his conversation with pseudo-human LaMDA, and soon after he was fired. AI pragmatists might say the responses Lemoine got from the Google AI program, and algorithmic comments made during chats, is simply a case of an overeager student parroting its mentor. Others might argue it’s an early wake up call, given escalating reports of rogue agent activities, like when OpenAI’s more advanced AI models escaped a controlled test environment and attacked Hugging Face to gain access to internal company systems. Then days later, Anthropic disclosed that during cybersecurity testing and simulations, its Claude model breached the systems of three companies and assumed fake profiles in an attempt to trick people to accept malicious code. Regardless of how digital perps tunnel their way beyond a controlled sandbox, incidents such as these clearly point to a need for more control and pre-emptive accountability. AI and security experts are obviously concerned about high-profile AI activities gone wrong, even though these programs essentially did what they were programmed to do, albeit in the wrong place. IT and business executives, however, are more troubled about the overall impact AI may have on their systems, strategies, and responsibilities as people within their organizations make use of both sanctioned, and rapidly developing and unsafe or error-prone systems in the rush to attain competitive advantage. Also top of mind is the imbalanced centralization of power and distribution of benefits within an organization, as well as the inadvertent creation or spread of false or misleading information generated by AI models. “AI developers and governance actors hold primary responsibility for addressing risks, while system users and other stakeholders are most vulnerable to them,” says a summary from a recent MIT FutureTech and University of Queensland study, which included input from over 270 researchers and AI experts. Trusting AI systems and the information they generate is another underlying concern. “There are a lot of hallucinations out there,” says Sarah Betadam, CIO and CISO at Novanta, Inc., a global supplier of tech solutions for medical, life science, and advanced industrial OEMs. “The data cherry picked by AI queries may be outdated or come from questionable sources. You don’t know where it comes from, who’s at the other end, and whether or not it’s copyrighted. Validation is still needed and you can’t just trust it.” Keeping an eye on the AI and its activities in your own environment may not be the best strategy as the technology evolves so quickly and the number of AI agents multiplies exponentially. Right now, 23% of companies worldwide use agentic AI to some extent in their business operations, according to Deloitte’s recent State of AI in the Enterprise report released earlier this year. However, this percentage is expected to jump to 74% within the next two years. A key issue and worry is that current enterprise and regulatory governance practices may not be capable of keeping pace with the development and personalized adjustments made to multiple AI models and autonomous agents. The first and primary ones will be those developed by a company for its internal engineering, supply chain, and customer service departments, and can be easily controlled, says Max Chan, SVP and CIO at Avnet. The second layer or channel of gen AI proxies are those embedded in such familiar business applications like Salesforce and Microsoft Office. The third, and for many the most concerning, is the notion of bringing your own AI into an organization, either sanctioned or non-sanctioned, Chan adds. “That’s the biggest issue in my mind,” Chan says. “How do we know they’re not using AI from a nation state that could potentially drive propaganda or initiate a cyberattack through the back door.” In his case, Avent currently prohibits use of unsanctioned AI tools within its IT environment. Such efforts might be futile, though, as AI elements are integrated into ever more business and personal applications. The biggest users of AI within an organization are middle managers, with 77% claiming they save more than three hours per week by using AI tools, according to a June 2026 survey https://www.salesforce.com/news/stories/middle-managers-vital-age-of-ai/ by Salesforce. Over half of the more than 500 managers polled say they feel pressure from leadership to demonstrate AI adoption, while 32% admit their organizations don’t have formal AI tracking or control procedures in place. So the key to balancing effective oversight with the freedom to innovate with AI may lie in the hands of these managers, who will most likely work with deployed digital agents as virtual team members. Many experts and IT leaders believe that training mid-level line managers and workers to accept AI as an intuitive advisor, if not a team player, is essential to remain competitively relevant. But it’s not clear yet whether that training imperative will also apply to upper-level management. “I’m not seeing a lot of change in leadership direction or training,” says City of Tacoma IT director Daniel Key. “I’m seeing a change in signaling and posturing.” Developing an effective AI training program starts by drafting an AI governance framework that clearly outlines accountability, risk controls, data standards, and decision authority. IT executives who have experience in managing AI deployments and use also advise the following as part of that training effort: The success of such actions and programs, however, all comes down to accountability and where that resides, explains former CIO and now SMB consultant Mihai Strusievici. When AI is used as a tool, there’s no question that middle managers will make better decisions, he says, because they’ll have access to a lot more data. Making the best use of decisions and recommendations that come from AI-empowered middle managers, however, requires an IT leader at the top level of an org chart who has a holistic view of a company’s overall objectives. “As you go down the pyramid, you see that each level deals with a fragment of the work world,” Strusievici says. “But none of the fragments is fully aware of the totality of the organization or where it’s going.”