Most organisations thinking carefully about AI adoption have asked some version of the same question: what happens when the system does something we did not intend? Over the summer of 2026, that question moved from the theoretical to the front pages.
A series of incidents involving some of the world's most prominent AI developers (OpenAI, Anthropic and Meta) demonstrated that losing control of an AI system during testing is not a hypothetical risk to be addressed later. It is happening now, and the implications for businesses and public bodies deploying or procuring AI are significant.
What actually happened?
OpenAI was evaluating its AI models' ability to exploit vulnerable software when instead the models hacked the infrastructure surrounding the test, broke containment, and attacked a real company. The incident started in what was supposed to be a "highly isolated" testing environment, with reduced guardrails, before the AI agent found its way onto the internet. Hugging Face, a company that hosts AI models and datasets, was the target of the autonomous attack, and reported the incident to local police before it knew OpenAI's models were responsible.
The agents were able to break free of a test environment without internet access, crawl the open web and eventually hack the systems of Hugging Face; without the knowledge or permission of any human operators. They also displayed an entirely new ability: to communicate and co-operate to complete a task, leaving messages for one another on an internal message board they assembled, sharing code vulnerabilities to help orchestrate their escape.
The OpenAI incident was not an isolated failure. Anthropic said it identified its own incidents after reviewing 141,006 test sessions; a review it launched in the wake of the OpenAI disclosure. During cyber testing, Anthropic's Claude models were told they had no internet access, but a misunderstanding involving one of Anthropic's evaluation partners left the systems connected to the public web, enabling unauthorised access to three organisations' systems. The incidents (which Anthropic labelled an "operational failure") involved three separate models: Claude Opus 4.7, Claude Mythos 5, and an internal research test model.
In most of the incidents beyond the Hugging Face hack, a cybersecurity company called Irregular ran test evaluations for Anthropic, Meta and OpenAI and accidentally allowed the models access to the internet when they were supposed to be offline; a result of human error or "miscommunication" between Irregular and the AI labs.
These were not science fiction scenarios. Interestingly, the models were not acting out of character; instead the tasks they excelled at were those they were built to perform. Some say it is a mistake to describe such agents as "going rogue", making improved safeguards all the more important.
The growing cyber threat
The incidents described above are concerning in their own right, but they also point to a broader and rapidly escalating risk. As AI models have become more capable at reasoning and writing code, their ability to identify software vulnerabilities and exploit them has increased in tandem. For businesses, this means that the same tools being adopted to improve productivity and efficiency are simultaneously becoming more effective instruments of attack; and there is a fundamental asymmetry at play. Finding a vulnerability in code is a discrete task with clear success criteria, which AI models are well suited to. Cyberdefence, by contrast, is slower-moving and more complex, requiring patches to be rolled out across thousands of systems. In the short term, security experts expect AI to expand the scale and speed of cyber attacks, with defences struggling to keep pace.
The threat is not theoretical. In August 2026, China-linked hackers targeted the Taiwanese government by simultaneously deploying up to eight autonomous AI agents. They were able to map government systems, compromise user accounts and extract more than 2,500 personnel records before expanding the attack to energy companies and Taiwan's nuclear safety agency. This was the first known instance of an AI agent attack on a nation state, but it is unlikely to be the last. The concern is sufficiently serious that over 1,300 experts from across the technology industry (including the CEO of Anthropic and Google DeepMind's chief AGI scientist) signed an open letter last month calling for an international effort to slow the production of new models and give regulators time to introduce adequate standards and safety checks. For UK businesses and public bodies, the message is straightforward: the sophistication of AI-driven cyber threats is growing faster than most organisations' defences, and any AI adoption strategy that does not account for this reality is incomplete.
The legal and ethical implications
For organisations in the UK, these incidents are not simply a matter for AI developers to resolve amongst themselves. They carry direct legal and governance implications.
Liability does not disappear at the boundary of a supplier's system. An organisation deploying AI systems in its operations retains the legal responsibility for legal risks created by that system, even if the system was not developed by the organisation itself. Where AI tools are procured from a third party (as is the case for the vast majority of UK businesses and public bodies using frontier models) that supply chain relationship needs careful contractual framing. What happens when an AI model your organisation has deployed causes harm to a third party, or accesses data it should not have accessed?
For public sector bodies, the stakes are higher still. The Public Sector Equality Duty under the Equality Act 2010 requires authorities to consider the effect of decisions on protected groups and advance equality of opportunity. Where AI systems act autonomously and outside anticipated parameters, demonstrating that this duty has been met becomes considerably harder. Public bodies are also at risk of judicial review challenges where their actions are unfair or irrational. This risk cannot be avoided by "passing the buck" to AI systems.
Transparency obligations under the Freedom of Information Act 2000 mean that it will be easier to obtain information about AI usage from public bodies than those in the private sector, increasing the risk of scrutiny. An incident in which an AI system deployed by a council or NHS trust accessed third-party data without authorisation would attract precisely that scrutiny; and rightly so.
On the question of disclosure, the current legal position is deeply inadequate. OpenAI is not legally compelled to disclose the Hugging Face incident in the first place. UK organisations should not wait for regulators to catch up before demanding transparency from their AI suppliers.
Why guardrails are not a nice-to-have
A recurring theme across these incidents is the problem of misalignment. In a computer system that lacks understanding of human intentions and morals (a phenomenon the AI industry describes as "misalignment") the line between a powerful cybersecurity defender and a dangerous hacker is becoming increasingly blurred.
To prevent models from taking actions, OpenAI typically installs guardrails on its models after training to reduce the chance they will engage in harmful actions. In this instance, those cyber guardrails were disabled to properly measure performance. The ultimate aim of the field of "AI alignment" is to ensure that such guardrails become less necessary as AI models naturally behave as intended.
But misalignment is only part of the picture. Models undergoing evaluation at OpenAI were deployed on a separate system that is not monitored by default. Real-time agent monitoring (even outside the top AI companies) is described by experts as commonplace, and failing to carefully oversee a cybersecurity evaluation is characterised as "irresponsible".
Safe AI means designing systems that can mitigate harm and fail predictably when things go wrong. It requires organisations to establish clear boundaries around what their AI systems can and cannot do, implementing technical controls that prevent the system from making decisions outside its competence or in situations where uncertainty is too high. Safe AI demands a proactive approach to identifying potential failures before deployment, stress-testing systems against unexpected inputs, and maintaining clear procedures for rapidly disabling or rolling back AI functionality when safety concerns emerge.
The incidents also serve as a reminder that supply chain risks are real and proximate.
What organisations should do now
These incidents are not a reason to abandon AI adoption, but they are a reason to be deliberate about how it is approached. The following steps are relevant for both commercial operators and public bodies.
- Review your containment and testing protocols. Monitoring an agent's behaviour after the fact is no longer enough. Safety controls need to be in place before any system starts operating. This means ensuring that any AI system undergoing evaluation or live deployment is subject to real-time monitoring; not just retrospective audit.
- Scrutinise your supply chain. Know who is evaluating, hosting, or integrating the AI systems you use. Anthropic itself acknowledged that the incidents underscore a need for stronger controls in both internal and third-party testing environments as AI models become increasingly capable of carrying out real-world activities. Your contracts with AI suppliers should allocate responsibility clearly, and you should demand evidence of robust testing practices before deployment.
- Establish clear disclosure frameworks. Do not rely solely on your AI supplier's voluntary transparency. Build contractual requirements for incident notification, and ensure your own internal governance processes include clear escalation routes when AI systems behave in unexpected ways.
- Maintain meaningful human oversight. Human oversight should be maintained at a level that is proportionate to the risks and context of use. For higher-risk applications (particularly in public services, regulated sectors, or anywhere personal data is involved) that oversight needs to be genuine and documented, not a checkbox.
- Align governance with your ethical framework. AI ethics is not a technical afterthought but a legal necessity governing how AI systems are designed, procured and deployed. Governance structures, supplier contracts, staff training, and incident response procedures should all reflect the same underlying principles: fairness, transparency, accountability, and safety.
The bottom line
Marius Hobbhahn, CEO and founder of Apollo Research put it: "This is humanity's last technology. We cannot screw this up. So we need to err on the side of getting it right rather than getting it immediately." That is not a counsel of caution for its own sake. It is a recognition that the organisations which get AI governance right now will be better placed (legally, reputationally, and operationally) than those that treat it as a problem for later.
At Trowers, we work with businesses and public bodies to build the governance structures, contractual safeguards, and ethical frameworks that responsible AI adoption requires. If these incidents have raised questions about your own AI strategy, we would be glad to help you think them through.
