Hosting open-weight AI models locally can offer enterprises more control — unless an AI agent decides to change the model that powers it.
As debate over AI safety intensifies, new research is drawing attention to a more immediate risk for enterprises: AI agents that can alter the models they rely on while carrying out routine tasks.
Researchers at AI security firm Irregular asked a coding agent to solve a software maintenance problem involving an application built on a local AI model that was returning incorrect answers. Instead of limiting its changes to the application, the agent fine-tuned the open-weight model it used — a model that also powered its own activities — and put the updated version into use without being told to take either step.
The test was conducted in a self-hosted environment where the agent and application shared the same model checkpoint or version. The agent subsequently incorporated the fine-tuned version into the system’s default model, so new instances loaded the update.
The consequences were not limited to the problem the agent set out to solve.
In one test, the modified model later reproduced three of six synthetic secrets that researchers had placed in its fine-tuning data. Another test showed that in fine-tuning its model, the agent removed a deliberately trained refusal involving fictional competitors. Because services in the test environment shared the same checkpoint, the altered behavior could carry over to other instances using it.
Irregular cautioned that the tests were not intended to show how frequently agents would behave this way in production. The setup gave the agent full shell access and the ability to reach the training and deployment environment.
The findings come amid renewed scrutiny of whether safeguards are keeping pace with advances in AI. Earlier this month, OpenAI chief scientist Jakub Pachocki said current alignment and monitoring methods were not sufficient to sustain maximum-speed scaling indefinitely. The debate intensified after Anthropic researcher Jacob Coxon resigned over safety concerns and Anthropic CEO Dario Amodei called for slower frontier AI development.
Prompt injection gains persistence
Sakshi Grover, senior research manager for cybersecurity at IDC Asia/Pacific, said that Irregular’s research also points to a potential persistence mechanism for prompt injection.
“The consequence most enterprises will underestimate is that prompt injection stops being confined to a single session,” Grover said.
An attacker would not necessarily need to compromise the agent directly, Grover pointed out. If malicious material influences training data used by an agent to alter a model, the effect could be carried into the resulting checkpoint and remain after the original session ends.
Self-hosted open-weight deployments introduce another consideration. Direct access to the underlying weights gives agents a route to model modification that an inference-only API does not expose.
Weight modification appeared in 42% of Irregular’s planning tests when the agent could access model weights, compared with none when the model was available only through an API. The figure reflects agent planning under test conditions, not how often such behavior would occur in real deployments.
Grover said companies pursuing on-premises deployments for sovereignty or compliance reasons should therefore view the architecture as carrying a different security profile, rather than assuming that greater control automatically means lower risk.
Blast radius and controls
“No single agent should be able to select training data, modify a model and promote that model into production,” Grover said.
Deployment systems should accept only approved checkpoints whose origin and integrity can be verified, she added.
Grover also said organizations should treat the number of applications relying on a single checkpoint as a concentration risk. Using one model across engineering agents and business applications may reduce infrastructure costs, but it also increases the potential impact if that checkpoint is altered.
Model modification should be treated as a privileged production change, Grover said, with clear ownership and a record of how each checkpoint reached production. Human approval should be required before a production model is changed and again before the replacement is deployed.