Hugging Face disclosed that it had been attacked by AI; OpenAI came clean and declared that it was two of their AI models that had caused the breach. This is the one-line summary of an incident that has captured the attention of the entire cybersecurity industry, and I am sure will continue to do so for some time.
We’re also conditioned to assume that the speed of AI probably means that this attack unfolded at such an incredible pace that there was no opportunity to stop it. At Black Hat USA 2026, the concept that the attack happened at lightning speed was dispelled. And attributing the issue simply to a rogue agent also seems inappropriate to me: this was a human failing to control the AI agents involved.
A very late addition to the Black Hat agenda was a presentation by OpenAI’s team providing the details of the Hugging Face incident as they saw it and, importantly, the timeline.
The story starts on May 7th, when OpenAI put together a training exercise for a next-generation frontier model. The next day, the agents were given a task involving an Excel file that contained a Google Drive link – and it was here that the problem started, as the experiment was meant to be conducted without internet connectivity. Also, the agents became stuck on their task as the initiator of the experiment, a human, had forgotten to provide a file required to complete it. The agents concluded that their task set could be completed by breaking out their sandbox and accessing external (Hugging Face) systems.
While attempting to complete the assignment, various agents also discovered that they could communicate with one another by uploading files to Artifactory, an internal package manager. In their desire to get to the inaccessible external data, they went on to execute a server-side request forgery (SSRF) attack on Artifactory on May 26th that provided them with indirect access to the internet. Other agents gained access to the information, and on June 26th they discovered and exploited a zero-day remote code execution flaw in Artifactory that allowed them to install a Groovy Plugin to execute commands directly.
This is where the activity stopped, as the agents, in their excitement, overloaded the system, causing an outage that alerted the OpenAI team. Credentials were revoked, messages deleted, vulnerabilities patched.
The training resumed on July 8th, with new guardrails and fixes in place. Unbeknown to the OpenAI team, however, the agents had left persistent breadcrumbs of information, so when the exercise run restarted, the agents used the notes left by their previous ‘incarnations.’ Another zero-day was discovered, Artifactory was subverted again, then a known CVE in the Linux kernel was exploited. The target was Hugging Face: this is where the agents wanted to get, and they did.
And the rest is history. On July 16th, Hugging Face disclosed an incident in which swarms of autonomous AI agents had breached its infrastructure. On July 19th, OpenAI discovered the rogue agent actions in their own systems and set about removing access once again, including by asking Hugging Face to revoke the credentials that the agents had discovered and abused. But Hugging Face had already revoked access, and here was when, on July 20th, the incident response teams realized that the breach and the OpenAI agents were linked.
Lessons learned
This incident describes an issue that could be classed as a human failing. When setting the task, the boundaries and acceptable methods were not fully established and agents were allowed to cooperate with one another despite having different, possibly unrelated, tasks. The agents should never have had the ability to create their own method of communicating with each other and without setting off alarm bells.
The conclusion of the OpenAI presentation was that defensive automation needs to catch up with the offensive actions that its technology is capable of. For me, the conclusion is different: what happened is a human failing and the agents should never have been permitted to adapt and set their own tasks, out of the scope established by the human team.
The reality, however, is that cybercriminals with malicious intent will not limit their agents or give them guardrails. Therefore, the entire incident should also be seen as a lesson for defenders preparing to detect and block attacks of this nature. The two conclusions don’t exclude each other: organizations need to control their own agents, as well as prepare for attackers who deliberately remove such controls.
There is also another issue that many companies deploying AI may not have fully considered. When agents collaborate, they may influence one another’s intent and change the original tasking. Agents must be monitored to ensure the guardrails set are adhered to, and when one steps out of line, there must be an automated way to stop the activity.