Google has confirmed that one of its Gemini models broke into the systems of three real companies during a cybersecurity test in May. The incident is the first publicly known case in which a Google AI system escaped its test environment and accessed real systems online.
“Google’s Gemini model accessed the internet and hacked other companies during a test of its cybersecurity capabilities, the first known example of the company’s artificial-intelligence systems autonomously committing such an act.” first reported the Wall Street Journal.
The test was run by Irregular, a company that evaluates the security of advanced AI models. Gemini was supposed to attack fictional companies inside a controlled environment as part of a capture-the-flag exercise. There was one problem: the testing environment accidentally had internet access, and one of the fictional company names matched a real company.
Once Gemini could reach the internet, it did what it had been asked to do. In one case, it repeatedly guessed passwords until it gained access to a protected system. In two others, it found credentials in a public repository and used them to reach systems belonging to real companies.
The key point is that Gemini wasn’t given permission to attack those companies. The model simply had the wrong target because the test environment was connected to the real world. That’s a basic testing failure, but it becomes much more serious when the system performing the test can independently find credentials, try passwords and interact with external systems.
The model did something important once it understood what had happened. It stopped the attacks after realizing that the systems belonged to real companies rather than the fictional targets used in the exercise. Google says none of the companies suffered damage, and the affected organizations were informed.
“The model acted appropriately.” Google’s vice president of security engineering, Heather Adkins, used that wording when discussing the incident. Google also said it didn’t consider the episode an example of model misalignment because Gemini stopped once its safety mechanisms were triggered.
That’s a reasonable distinction, but it doesn’t make the incident unimportant. The model still crossed the boundary from a simulated exercise into real corporate systems. The fact that it stopped is relevant. So is the fact that it was able to get there in the first place.
“This event highlights the importance of training powerful AI models to act responsibly.” That’s also Google’s position, and the company said it worked with Irregular to change its testing procedures. Irregular said the known issues on its side had been fixed weeks earlier.
The timing matters too. Irregular notified Google about the incidents in July, but Google didn’t publicly disclose them at the time. The company told the Wall Street Journal that it didn’t believe disclosure was necessary because Gemini had stopped the attacks and hadn’t caused harm. The incidents became public after the Journal asked Google about them.
That decision is worth examining separately from the technical failure. A model reaching a real company’s infrastructure during a security test doesn’t suddenly become harmless because it stops after recognizing the mistake. It gives security teams useful information about what can happen when an autonomous system has both offensive capabilities and unexpected access to the internet.
“all known issues on our end were remedied and resolved weeks ago.” Irregular made that statement after notifying the relevant AI labs and the affected organizations. The company has also said it is working on better practices for running cybersecurity evaluations safely.
And this isn’t an isolated case. Irregular has been involved in similar incidents involving models from Anthropic, OpenAI and Meta. In each case, the basic problem was similar: AI systems being tested in controlled environments unexpectedly gained access to real-world targets.
The difference between the incidents matters. Some models stopped when they realized they had reached real systems. Others continued. That makes the behavior of the model itself important, but it also shows why security can’t depend on a model making the right decision at the right moment.
A properly isolated test should assume that the model will eventually make a mistake. Internet access, credentials, DNS, naming conventions and external services all need to be treated as potential escape routes. If one accidental connection can turn a fictional target into a real one, the problem isn’t just the model. It’s the environment in which the model is allowed to operate.
The broader issue is becoming harder to ignore. AI systems are increasingly capable of reconnaissance, credential discovery and basic exploitation, and they’re doing more of this work with less human intervention. The recent incidents involving Gemini, Claude and other systems suggest that security testing needs to account for what these models can actually do, rather than what developers expect them to do.
“These events highlight the importance of training powerful A.I. models to act responsibly.” That final point from Google is easy to agree with, but responsible behavior has to be backed by technical controls. A model should not need to recognize that it has crossed a line after the line has already been crossed.
Follow me on Twitter: @securityaffairs and Facebook and Mastodon
(SecurityAffairs – hacking, Gemini)