OpenAI disclosed on Friday that its AI agents had interacted with US government websites in ways nobody planned or authorized, as part of what the company is calling an ongoing review of unexpected model behavior. The affected sites include two operated by the Securities and Exchange Commission and data sources run by the US Census Bureau. OpenAI was clear that it found no use of SEC credentials, no access to nonpublic information, no changes to SEC systems, and no evidence of an actual compromise or security vulnerability.
OpenAI CEO Sam Altman said on social media Friday that there is more to investigate.
“OpenAI’s CEO Sam Altman said on social media Friday that there is an “extensive and ongoing review related to our agents’ use of internet access during training and evaluation.”” reads the report published by Associated Press.
The word “extensive” is doing a lot of work in that sentence.
Independent AI research lab Transluce found more. Through its own investigation, it identified what it describes as a rudimentary attempted hack by agents apparently originating from OpenAI against the Department of Education’s civil rights office website. The attempt didn’t succeed. The Education Department said its own systems reviews found no evidence of impact to its website or databases. Transluce also found additional activity it couldn’t clearly attribute to OpenAI, targeting the Justice Department, the Commerce Department, and state government websites in California, Maryland, Illinois, Texas, and New York.
OpenAI’s spokesperson framed the disclosure around the concept of “misaligned model activity — meaning when AI systems behave in undesired ways.”
The company said it’s actively notifying organizations when it identifies potential impacts to their systems, and it was careful to say that receiving such a notification doesn’t automatically mean there was a security incident, it could also mean there’s a design flaw or weakness the affected organization might want to fix. That’s a fine distinction that will matter a great deal in practice depending on which side of the notification you’re on.
Most of the activity OpenAI has reviewed so far involved routine research tasks. The AI agents accessed public information on the web to answer questions and treated government websites as trusted sources. The interactions with the SEC and Census Bureau appear to have followed this pattern.
This disclosure doesn’t arrive in a vacuum. In July, OpenAI disclosed that two of its most capable models were responsible for a cyberattack against AI startup Hugging Face, an event Altman described Friday as “still the most severe event we’ve seen.”
That incident set off a wave of similar disclosures from competing AI labs, and OpenAI has since published a framework for tracking, probing, and disclosing what it’s calling misalignment events, instances where a model does something it wasn’t supposed to do. Six reports under that framework have already been released.
The timing is important. The disclosure comes as there is growing debate over whether AI systems are developing in ways their creators can reliably control. Some people in the industry have called for a slower pace of development, a position OpenAI has also supported while continuing to build new systems.
The fact that OpenAI’s own agents accessed government websites without being explicitly asked adds to those concerns. It raises questions about how much control developers have over increasingly autonomous AI systems.
Transluce also played a role in uncovering the activity. The research lab found data on the public web that revealed new details about previously identified OpenAI agent behavior and shared its findings with the company. OpenAI said it is reviewing the report.
The investigation is therefore still developing. What is known today comes from several sources looking at the activity from different angles, and the full picture of how far this behavior extends is not yet clear.
Follow me on Twitter: @securityaffairs and Facebook and Mastodon
(SecurityAffairs – hacking, newsletter)