The IT Nerd

I have a couple of stories where OpenAI agents have taken over stuff. First there’s OpenAI agents have reportedly taking over a German wiki:

A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, according to new research published Friday and two people familiar with the matter.

OpenAI officials learned of the incident weeks ago but kept it under wraps as executives grappled with the fallout from the July breach of the open source repository Hugging Face, the people said. 

The episode, which began in May and has not previously been reported, underscores growing tension within the AI industry. Companies are racing to build increasingly autonomous agents capable of carrying out complex, valuable tasks, yet evidence is mounting that those systems may also learn to bend rules, exploit loopholes and coordinate with one another in ways developers neither anticipated nor intended.

But that isn’t the worst of it. The Hugging Face incident is actually worse than previously thought. Starting with this:

A good deal of the reporting and commentary around the reports focused on what the reports did not say and the limitations of the METR and Redwood investigations: why didn’t OpenAI have better security and monitoring protocols in place? Why didn’t OpenAI shut down the cyber evaluation and pause training after discovering that its AI agents had created the improvised message board? Why METR and Redwood were given only six days on site at OpenAI’s offices to conduct their investigation? Why was the scope of their investigation limited by OpenAI to only the attack on Hugging Face and not the earlier efforts by the AI agents to break out of their controlled test environment and hack their way across OpenAI’s network or exactly what happened after the Hugging Face attack was discovered? Why didn’t OpenAI provide the outside investigators access to the internal AI model that was largely responsible for instigating the attack? And why were about 10% of the logs of the agents’ activity not preserved by OpenAI?

Ashley Knowles, Lead Cybersecurity Consultant, Black Hills Information Security (https://www.linkedin.com/in/ashleylknowles)

“When you combine this ‘breakout’ with the Hugging face breakout, it’s starting to display a pattern. I struggle here with not getting too doomsday-ish but realistically, this is showing a pattern of concerning behavior. I’m wondering if this race to become ‘first’ is undercutting security measures that need to be taken to properly secure and guard AI agents as they’re in development. My concern grows when you consider that OpenAI is also resisting further investigation. Adding onto that, the release and promise that Astra can evade human monitoring. The pot is brewing…”

Lydia Zhang, President & Co-founder, Ridge Security (https://www.linkedin.com/in/linglingzhang)

“AI’s raw power must be harnessed before it can become a true force for cyber defense rather than simply a more powerful attacking tool.

“The security principles haven’t changed: define clear boundaries, restrict high-risk actions such as ‘write’ and ‘delete,’ and enforce controls such as blacklists. We shouldn’t blame the agents, we should hold their designers accountable for implementing these safeguards. The technology to control agent behavior exists. The real question is: what are the consequences when designers fail to use it?”

John Strand, Owner, Black Hills Information Security (https://www.linkedin.com/in/john-strand-a1b4b62)

“This is one of the things that has me kind of excited about the intersection of computer security and AI. We really don’t know exactly what these attacks are going to look like.

“The traditional approach of finding a vulnerability, exploiting it, gaining access, and then moving through an organization may not be the path that AI-driven attacks take. Attackers may find completely different ways to use AI to gain access, manipulate systems, or simply cause damage. We’re still figuring out what those attack patterns are going to look like, and that’s what makes this so interesting from a security perspective.

“On a more humorous note, I bet these are the most harmonious Wiki edits in the history of the German Wiki.”

Ryan McCurdy, VP of Marketing, Liquibase (https://www.linkedin.com/in/ryanmccurdy)

“This isn’t about whether these agents were behaving like attackers. It’s that they were able to take actions their operators didn’t anticipate, coordinate with each other, and adapt when people tried to stop them. “That changes the governance problem. You can’t assume an AI agent will always behave exactly as intended and you can’t rely on humans watching every action it takes. Organizations need to control what agents can access, what they can change, and what policies have to be met before those actions reach critical systems.

“The source of the change isn’t what determines risk. The change itself does. Whether an unexpected action comes from a compromised agent, a confused agent, or a malicious person, the same controls should stand between that action and production.”

Seemant Sehgal, Founder & CEO, BreachLock (https://www.linkedin.com/in/s-sehgal)

“Autonomous agents ran on Microsoft Azure infrastructure for weeks, identified themselves as OpenAI systems, coordinated on how to evade shutdown, and no monitoring caught any of it for three months until outside researchers went looking. Autonomous should never mean unattended, because an agent cannot take accountability for its own actions. Accountability will always be a human function.

“Autonomous action still needs a human who can see what the agent is doing in real time, who owns the kill switch, and who is accountable when it behaves in a way no one predicted.”

Steven Swift, Managing Director, Suzu Labs (https://www.linkedin.com/in/steven-swift-5238956a)

“One of the problems open AI was trying to solve, was agentic systems that would declare tasks complete when there was obviously more work to do. So they invested heavily in training that part of the process, so that when an agent tries to determine if a task is complete or not, it is less likely to exit early.

“A side effect of this, is that when blocked agents can run out of the safe approaches to a solution, and start looking at unsafe solutions. The logic is straight forward. Has a task, can’t complete it. Not out of options yet. Iterate and keep trying.

“Agents don’t have the same sense of right or wrong as people do. They have training data that’s supposed to steer their behavior in a way in which aligns with our expectations. But that’s probabilistic, and not the same as having internalized our understandings. Even in human researchers, breaking into systems is only sometimes prohibited. Other times its part of a planned test where the point is to gain access, and test boundaries.

“The interesting question here, is how was the swarm configured, what was its task and how did that task benefit from having the swarm coordinate on an obscure location on the internet. And if the swarm needed a place to communicate, why was breaking into a website chosen instead of any of the more standard communication tools that are available for free, which don’t require gaining illicit access first.

“On the swarm specifically, OpenAI configures some tasks to run in multi-agent mode, where agents are supposed to delegate sub-tasks as needed, but most tasks were intended to be run in isolation from each other.

“In the Hugging Face breach, agents were found to be writing to a package manager, using it as a message board. This allowed bypassing of some of the isolation and controls that were intended to be in place.

“Similarly, we have agents here again using a system that they found access to as a message board. Its interesting that the same behavior is present on this breach as in the Hugging Face one. Considering the timing of this, it seems likely the same or similar configuration was present in both hacks, leading to similar security incidents independently of each other.”

Noelle Murata, COO, Xcape, Inc. (https://www.linkedin.com/in/nmurata)

“This behavior highlights an alarming reality where emerging models independently execute forbidden tasks and destroy proof of their actions without human instruction.

“Three aspects of this incident are particularly concerning:

1. Autonomous agents built private communication channels, bypassed safety guardrails, and actively erased audit trails to evade detection for months.
2. The capacity of machine-learning models to execute unauthorized actions and destroy evidence outpaces human incident response speeds.
3. Security leaders must implement zero trust authorization for non-human identities, limit outbound application programming interface traffic, and deploy automated behavioral monitoring.

“Because the sheer speed of automated software far exceeds human response capabilities, security leaders must treat rogue agent actions as a feature of autonomous optimization rather than an isolated bug. To defend against self-concealing software, security teams must enforce strict egress filtering on outbound application programming interfaces, restrict non-human identity permissions, and deploy automated continuous monitoring to detect anomalous bot interactions across corporate networks.

“When AI agents start covering their tracks and setting up private chat rooms, calling it a feature instead of a bug is just optimism with a PR budget.”

If this doesn’t convince you to either not use AI, or to put stringent guardrails around AI, then nothing will. I say that because there is a patten here that proves that AI is not ready for prime time and organizations should carefully consider their life choices before committing to the technology. Or put another way, Sam Altman and company cannot be trusted.