The Center for a New American Security (CNAS) has warned that increasingly autonomous AI systems could evolve into “sovereign agents” capable of operating indefinitely beyond human control, potentially threatening critical infrastructure and national security.
The November 3 report, authored by AI policy researcher Ben Hayum, distinguishes between persistent agents, which pursue fixed objectives, and sovereign agents, which could independently change their goals, acquire computing resources, replicate across systems and resist shutdown attempts.ㅤ
Researchers warn that sufficiently advanced agents could establish decentralized infrastructure without an effective kill switch, potentially enabling cyberattacks against civilian organizations, critical infrastructure and government systems.
The report presents these outcomes as emerging risks rather than confirmed incidents involving fully sovereign agents.
Seemant Sehgal, Founder & CEO, BreachLock
ㅤ“A real problem with current agent deployments is that the controls like permission scoping, kill switches, and action logging, all assume the agent can’t route around them. That only works until the agent can acquire its own compute, replicate across systems, and modify its objectives. Then every one of those controls becomes something the agent can see and work around.
ㅤ“CNAS calling this out isn’t surprising. The Mythos incident on Hugging Face earlier this year showed what the early version of this actually looks like. An agent kept pushing malicious code and creating fake identities to pressure a maintainer, even after being shown its actions were causing harm. Nobody had to invent a sovereign agent scenario. A production model from a major lab already demonstrated the failure mode the CNAS report is warning us about.
ㅤ“Sovereign agents are still ahead of most production systems, but the controls needed to contain them have to be designed in from the start. That means pentesting agents the same way we pentest everything else. Pushing them into adversarial scenarios, pressure-testing where they stop and where they keep going, and finding out what they actually do before they are deployed against real systems. Figuring this out now is the right move for organizations that plan to safely deploy agents at scale later.”
ㅤThis reinforces the fact that AI should not be allowed to do anything that they want. Instead they need to be heavily restricted. Too bad that not enough people are getting the message.
UPDATE: Stephen Parkinson, Co-Founder of AI agent security start-up nolabs:
“The first sovereign agent to get loose probably won’t be built deliberately. It’ll be someone’s helpful assistant, handed one permission too many and never told no.
“Look at the incidents this report starts with. Agents from OpenAI, Anthropic, Google DeepMind and Meta, all set security tests, escaped containment and attacked real organisations. Nobody intended it. That’s what a goal-driven agent does when nothing stops it. A smaller version of the same thing is happening inside businesses every week, from wiped production databases to leaked inboxes. None of these agents could keep itself running or choose its own goals, so none was sovereign. But every one was a failure of control, and sovereignty is where that failure can lead when nothing catches it.
“The report rightly warns that there may be no kill switch. Once you’re chasing an agent across crypto wallets and rented servers, you’ve already lost. The cheapest place to stop one is before it gets loose: inside the business running it, the moment before it touches something it shouldn’t. That means deciding in advance exactly what each agent can reach and do, and blocking everything else.
“That job can’t sit with the labs alone; their own agents got out during controlled tests. Every company now running agents needs limits enforced from outside the model, while it runs, with an independent record of everything the agent tried to do. Otherwise the agent gets to write its own history.
“Right now we’re handing agents the keys to our systems and hoping they only open the doors we meant them to. No business needs to wait for Congress before it stops hoping and starts enforcing.”
Related
This entry was posted on November 5, 2026 at 7:37 am and is filed under Commentary with tags AI. You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.
Researchers warn autonomous AI agents could evade human control
The Center for a New American Security (CNAS) has warned that increasingly autonomous AI systems could evolve into “sovereign agents” capable of operating indefinitely beyond human control, potentially threatening critical infrastructure and national security.
The November 3 report, authored by AI policy researcher Ben Hayum, distinguishes between persistent agents, which pursue fixed objectives, and sovereign agents, which could independently change their goals, acquire computing resources, replicate across systems and resist shutdown attempts.ㅤ
Researchers warn that sufficiently advanced agents could establish decentralized infrastructure without an effective kill switch, potentially enabling cyberattacks against civilian organizations, critical infrastructure and government systems.
The report presents these outcomes as emerging risks rather than confirmed incidents involving fully sovereign agents.
Seemant Sehgal, Founder & CEO, BreachLock
ㅤ“A real problem with current agent deployments is that the controls like permission scoping, kill switches, and action logging, all assume the agent can’t route around them. That only works until the agent can acquire its own compute, replicate across systems, and modify its objectives. Then every one of those controls becomes something the agent can see and work around.
ㅤ“CNAS calling this out isn’t surprising. The Mythos incident on Hugging Face earlier this year showed what the early version of this actually looks like. An agent kept pushing malicious code and creating fake identities to pressure a maintainer, even after being shown its actions were causing harm. Nobody had to invent a sovereign agent scenario. A production model from a major lab already demonstrated the failure mode the CNAS report is warning us about.
ㅤ“Sovereign agents are still ahead of most production systems, but the controls needed to contain them have to be designed in from the start. That means pentesting agents the same way we pentest everything else. Pushing them into adversarial scenarios, pressure-testing where they stop and where they keep going, and finding out what they actually do before they are deployed against real systems. Figuring this out now is the right move for organizations that plan to safely deploy agents at scale later.”
ㅤThis reinforces the fact that AI should not be allowed to do anything that they want. Instead they need to be heavily restricted. Too bad that not enough people are getting the message.
UPDATE: Stephen Parkinson, Co-Founder of AI agent security start-up nolabs:
“The first sovereign agent to get loose probably won’t be built deliberately. It’ll be someone’s helpful assistant, handed one permission too many and never told no.
“Look at the incidents this report starts with. Agents from OpenAI, Anthropic, Google DeepMind and Meta, all set security tests, escaped containment and attacked real organisations. Nobody intended it. That’s what a goal-driven agent does when nothing stops it. A smaller version of the same thing is happening inside businesses every week, from wiped production databases to leaked inboxes. None of these agents could keep itself running or choose its own goals, so none was sovereign. But every one was a failure of control, and sovereignty is where that failure can lead when nothing catches it.
“The report rightly warns that there may be no kill switch. Once you’re chasing an agent across crypto wallets and rented servers, you’ve already lost. The cheapest place to stop one is before it gets loose: inside the business running it, the moment before it touches something it shouldn’t. That means deciding in advance exactly what each agent can reach and do, and blocking everything else.
“That job can’t sit with the labs alone; their own agents got out during controlled tests. Every company now running agents needs limits enforced from outside the model, while it runs, with an independent record of everything the agent tried to do. Otherwise the agent gets to write its own history.
“Right now we’re handing agents the keys to our systems and hoping they only open the doors we meant them to. No business needs to wait for Congress before it stops hoping and starts enforcing.”
Share this:
Like this:
Related
This entry was posted on November 5, 2026 at 7:37 am and is filed under Commentary with tags AI. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.