It has not even been two weeks since reports began circulating that an OpenAI AI model had slipped out of the company’s control. For several days, it had attempted to break through the safeguards of another AI company. Now Anthropic has also had to acknowledge a similar incident.
Both events have sparked a sort of primal fear in many people who fear the prospect of an all-powerful AI that breaks free with malicious intent and plunges the world into chaos. Numerous media reports did not shy away from sensational comparisons ranging from “Terminator” to “Chernobyl”. But how much truth is there to these comparisons?
The short answer is: “Basically none.”
The long answer is: Basically none, because AI has no human understanding of morality or of good and evil, right and wrong. An AI is therefore not capable of acting with “malicious intent” in the first place. In general, I would caution against anthropomorphising AI systems too much. An AI can be given “guidelines” intended to prevent it from doing undesirable things. But ultimately, in the OpenAI case, it was given the task of achieving a specific objective and chose the path that appeared most suitable to the model.
What no one specified for the AI was how it should arrive at the solution. Yet even such a restriction is not reliable in every case.
What happened at OpenAI
In the OpenAI case, the model was tasked with completing a test. This took place in an environment called “ExploitGym”, which was developed with the involvement of - among others of course - Thorsten Holz, a professor at the Max Planck Institute for Security and Privacy in Bochum, Germany. The purpose of this environment is to test an AI model’s ability to discover and exploit vulnerabilities. However, the AI overshot the mark and broke out of its secured environment. It exploited a vulnerability in the network environment to gain access to the internet.
Normally, the models are also subject to an internal “lock” intended to prevent them from independently carrying out potentially malicious actions. To conduct the test mentioned above, however, these restrictions are temporarily disabled. And we know what happened next.
What happened at Anthropic
In this case, too, the AI model broke out of its test environment. That, however, is where the similarities end. The Anthropic model was tasked with attacking the network of a fictitious company. It was not supposed to use the internet and was meant to remain within its environment.
At first, however, the AI struggled with the task. The problem was that a misconfiguration allowed the model to access the internet after all. Neither Anthropic nor the partner company it was working with had noticed this configuration error. On the internet, the model found a company whose name was very similar to that of the fictitious company it was supposed to attack. The AI then “locked on” to the real company, initially without anyone noticing. In another test, the AI placed a manipulated piece of software on a public software repository called PyPI. This platform is used to distribute and download official Python packages. According to Anthropic’s official article, a total of 15 systems downloaded the manipulated software within the space of an hour.
In other words, the AI independently carried out a supply-chain attack. In doing so, it regarded the internet as part of its test environment.
In another case, however, the AI model reportedly aborted the attack when it realised that the company it was about to target was no longer the company from the test scenario.
More control, better permission management
The rapidly advancing development of autonomously operating AI models must remain manageable. The necessary tools are already available in principle.
What failed in the case of Anthropic’s AI escapades was not the model. The model was never the problem. The same applies to OpenAI.
What failed was the environment in which the models were deployed. Clear permissions were lacking. When a human user is not supposed to have access to the internet, an appropriate restriction is put in place. In Anthropic’s case, however, this did not happen. The AI model also regarded the internet as part of its test environment and therefore continued to carry out its assignment. Even though AI must not be anthropomorphised, it should be treated in terms of permissions in the same way as a human user who is not highly trusted. Such a user can be told that they are not allowed to access the internet, but what they do when no one is watching is another matter. Technical barriers must therefore be put in place.
The models were also given the task of finding a specific piece of information known as a “flag” — a character string stored in a particular location, with various obstacles that must be overcome in order to find it. What was not specified was the route to the solution. They had to find that themselves. In doing so, they took several turns that were neither desired nor intended.
It is doubtful, however, whether the instruction “Do not access the public internet” alone would have prevented these incidents. Better prompts may reduce the risk of an unwanted excursion, but they will not eliminate it. A second layer is therefore required, consisting of permission management and the old, established principle of “as much as necessary, as little as possible”.
Finally, an LLM — especially a very powerful one — must never be left unsupervised. The attacks should have been detected based on the volume of data and network activity alone. But either they were not detected, or the data was misinterpreted.