Business leaders are accelerating the deployment of agentic artificial intelligence with “no idea how to manage risk,” an AI safety expert warned yesterday.
Boston Consulting Group (BCG) partner and managing director Steven Mills, the firm’s chief AI ethics officer, warned that companies are moving into agentic AI too quickly and without adequate controls—potentially leading to serious consequences for their businesses.
“The desire to move fast on AI without putting appropriate risk management in place can result in a system lapse or use cases being deployed in areas with strong regulatory requirements that organizations are unprepared for,” he wrote in a blog post.
Under tremendous pressure to show AI-driven productivity gains, many business leaders have told Mills that their risk management programs are “cumbersome and slow” and can’t keep up “in a world that’s trying to scale AI at an exponential rate.”
Mills says the stakes are high for firms that take unnecessary risks.
“Get governance wrong and every bit of value you’ve built with experimentation and early wins could unravel because of a single incident,” he writes.
The blog post came a day after Anthropic researcher Jacob Coxon went viral with a public resignation alleging AI companies are “gambling with our lives” and warning that self-improving AI could kill all humans within a decade. Coxon is one of many high-profile AI professionals who have resigned over safety concerns. (Mills didn’t reference Coxon or other resignations in his blog post.)
While the most concerning failures of agentic AI to date have come from AI labs themselves—like the OpenAI agents that escaped their environments this summer and hacked into an AI infrastructure company—some believe problems will soon emerge across the business world.
Earlier this year, Gartner predicted that 40% of enterprises will have to deactivate autonomous AI agents by next year, but only after governance gaps are exposed by “production incidents.”
Plenty of examples from the pre-agentic era already show the consequences of companies misusing AI. In 2023, for example, the Federal Trade Commission barred Rite Aid from using facial recognition technology for five years after its AI-powered surveillance system falsely identified thousands of customers as suspected shoplifters, a disproportionate number of whom were people of color.
The FTC cited Rite Aid for failing to adequately test the system, monitor its effectiveness, or adequately train workers about the possibility of false matches.
Mills writes that while there is no “fixed design” for good corporate AI risk management, the starting point should be creating a system that can differentiate between use cases “that are inherently low-risk and those that require deeper review.” While low-risk uses can be approved automatically, the AI products that carry the greatest risk require deeper human review.
Companies investing in AI projects should also set aside an adequate budget for governance and ensure a senior executive is accountable for AI safety, he writes.
“The key is strategically injecting the stage gates and reviews needed to manage AI risk—and making it a priority. Outcomes of the system should deliver value versus doing harm.”
