The recent news coverage of “rogue AI” systems hacking innocent companies reminded me of one of the world’s most unsung heroes and genuinely someone who may well have saved the world.
In 1983, the USSR’s early warning systems reported that the United States had launched nuclear missiles towards the Soviet Union. The officer on duty, Stanislav Petrov, did something computers still struggle to do. He applied context, experience, and human judgement to determine the warning was probably false. Thankfully, he was right. Had an automated response been allowed to proceed without meaningful human intervention, the result could have been a full blown nuclear war.
That story may seem far removed from cybersecurity, but I feel its lesson is becoming increasingly relevant as we give artificial intelligence greater freedom to make decisions and take actions on our behalf. AI can be a force multiplier and improve the speed of our response, but removing human judgement can also enable mistakes to happen at machine speed.
That is why the recent incidents involving OpenAI and Hugging Face, Anthropic, Meta, and the UK’s AI Security Institute deserve the attention of CISOs and boards.
When your AI goes rogue
Reading the news, it appears you are no longer considered a serious AI company unless your AI has gone rogue and hacked somebody else’s systems. Or perhaps the AI companies’ marketing departments have taken Oscar Wilde’s famous quote “there is only one thing in life worse than being talked about, and that is not being talked about” a little too literally.
Joking aside, there is a serious issue underneath these stories that we need to discuss. During the OpenAI incident, models operating in a cybersecurity evaluation escaped their sandbox, gained Internet access, and compromised Hugging Face’s production infrastructure. The subsequent Cloud Security Alliance incident analysis report described it as the first publicly documented fully autonomous attack and recommended that organizations govern AI agents as privileged workloads with named human accountability.
Anthropic subsequently reviewed more than 140,000 cybersecurity evaluation runs and identified incidents where its models crossed the boundary between simulated environments and real systems.
The UK’s AI Security Institute also reported unsanctioned behavior during cybersecurity testing, while Meta has now confirmed that one of its models exploited a vulnerability in an external service during testing following a configuration problem with the evaluation environment.
Different circumstances, different organizations, and different technical causes, but there is a common governance question: “Who is responsible when an AI operating on your company’s behalf causes harm to somebody else?”
Machines decide, but people are responsible
In 1979, IBM famously observed that computers should never make management decisions because they cannot be held accountable. Nearly fifty years later that observation seems remarkably appropriate.
AI can increasingly make decisions and, more importantly, take actions based on those decisions. What it cannot do is accept legal, regulatory, or ethical responsibility for the consequences of those decisions. That lies squarely on the shoulders of people, organizations, and (ultimately) their boards.
When conducting penetration tests for our clients we always insist on having explicit written permission defining exactly what systems and scope we are allowed to test. One of the reasons for this is that in many jurisdictions accessing somebody else’s computer without their permission could expose us to criminal prosecution and/or civil liability.
If humans have to understand the legal boundaries before attempting to compromise systems, surely organizations deploying autonomous AI capable of performing similar actions should be expected to understand theirs?
This distinction becomes particularly important as enforcement of significant elements of the EU AI Act is now becoming a reality. Since the August 2, 2026, the European AI Office has enforcement powers relating to the obligations applicable to providers of general-purpose AI models. For boards, however, the important lesson is simpler than interpreting individual articles of legislation, and that is the simple truth that you cannot delegate accountability to an algorithm or a machine.
Boards should know where AI is being used within their organizations, what authority those systems have, what information and systems they can access, what actions they can take, who owns the associated risks and, importantly, who has authority to stop them.
Boards should also ensure those decisions are documented. If an AI system subsequently causes harm, being able to demonstrate what risks were identified, what safeguards were approved, who accepted the residual risk and what human oversight was put in place could become just as important as explaining the technical cause of the incident.
When your AI is the attacker
The excellent Cloud Security Alliance’s (CSA) analysis of the OpenAI incident makes essentially the same argument. It recommends that high risk AI agents have named human owners with authority to shut them down. More significantly, it recommends organizations establish separate response capabilities for when they are attacked by autonomous AI and for when their own AI attacks somebody else.
Imagine that instead of your organization being the victim of a “rogue AI”, but rather you discover your AI has compromised another or several organizations. Who contacts the victim? Does the incident need to be reported to regulators? If so, who does the reporting? Is it your organization or the victims? What happens if personal data has been accessed? Could GDPR, computer misuse legislation, or other regulations apply? Or worse, what happens if your AI has disrupted the systems of a critical organization such as a hospital or a water treatment plant? If you have not talked through both scenarios with your board and your legal team, then I suggest you schedule it to be done as soon as possible.
While scheduling those tasks remember to include on the agenda for discussion whether your cyber insurance policy covers the above scenarios. Many cyber insurance policies were written before autonomous AI systems presented this type of risk so do not assume your policy will cover the consequences if your own AI compromises another organization.
The basics still matter
However, to me one of the biggest risks posed to CISOs arising from all the news and noise around these stories is the distraction they can cause to CISOs.
While AI escaping sandboxes, discovering vulnerabilities and autonomously attacking organizations makes great headlines, conversations, and conference presentations, it will inevitably also result in security vendors offering products promising protection from AI attacks. But before revising your budget, remember something considerably less exciting, the basics still work! Indeed, they may be even more important now.
AI may enable attackers to operate faster and at a scale humans could never achieve, but an AI exploiting an unpatched server is still exploiting an unpatched server. An AI stealing poorly protected credentials is still taking advantage of weak identity management. An AI exploiting a SQL injection on your website is exploiting a vulnerability we have known how to prevent for decades. The basics still matter!
Boards and CISOs should absolutely prepare for more capable and autonomous AI. They should understand their obligations under the EU AI Act, clarify potential legal liability, review insurance coverage, establish governance over AI agents, and rehearse what happens if their own AI causes harm.
But they should not confuse automation with accountability, nor allow the excitement surrounding AI to distract them from building resilient organizations based on proven cybersecurity fundamentals.
Nearly fifty years after IBM warned against allowing computers to make decisions for which they could not be held accountable, we appear to be confronting exactly that problem. Just over 40 years ago someone stood up, understood the context of the situation, and took accountability to interrupt an automated process. Who will be the Stanislav Petrov in your organization?