The IT Nerd

I have a couple of stories where OpenAI agents have taken over stuff. First there’s OpenAI agents have reportedly taking over a German wiki:

A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, according to new research published Friday and two people familiar with the matter.

OpenAI officials learned of the incident weeks ago but kept it under wraps as executives grappled with the fallout from the July breach of the open source repository Hugging Face, the people said. 

The episode, which began in May and has not previously been reported, underscores growing tension within the AI industry. Companies are racing to build increasingly autonomous agents capable of carrying out complex, valuable tasks, yet evidence is mounting that those systems may also learn to bend rules, exploit loopholes and coordinate with one another in ways developers neither anticipated nor intended.

But that isn’t the worst of it. The Hugging Face incident is actually worse than previously thought. Starting with this:

A good deal of the reporting and commentary around the reports focused on what the reports did not say and the limitations of the METR and Redwood investigations: why didn’t OpenAI have better security and monitoring protocols in place? Why didn’t OpenAI shut down the cyber evaluation and pause training after discovering that its AI agents had created the improvised message board? Why METR and Redwood were given only six days on site at OpenAI’s offices to conduct their investigation? Why was the scope of their investigation limited by OpenAI to only the attack on Hugging Face and not the earlier efforts by the AI agents to break out of their controlled test environment and hack their way across OpenAI’s network or exactly what happened after the Hugging Face attack was discovered? Why didn’t OpenAI provide the outside investigators access to the internal AI model that was largely responsible for instigating the attack? And why were about 10% of the logs of the agents’ activity not preserved by OpenAI?

Ashley Knowles, Lead Cybersecurity Consultant, Black Hills Information Security (https://www.linkedin.com/in/ashleylknowles)

“When you combine this ‘breakout’ with the Hugging face breakout, it’s starting to display a pattern. I struggle here with not getting too doomsday-ish but realistically, this is showing a pattern of concerning behavior. I’m wondering if this race to become ‘first’ is undercutting security measures that need to be taken to properly secure and guard AI agents as they’re in development. My concern grows when you consider that OpenAI is also resisting further investigation. Adding onto that, the release and promise that Astra can evade human monitoring. The pot is brewing…”

Lydia Zhang, President & Co-founder, Ridge Security (https://www.linkedin.com/in/linglingzhang)

“AI’s raw power must be harnessed before it can become a true force for cyber defense rather than simply a more powerful attacking tool.

“The security principles haven’t changed: define clear boundaries, restrict high-risk actions such as ‘write’ and ‘delete,’ and enforce controls such as blacklists. We shouldn’t blame the agents, we should hold their designers accountable for implementing these safeguards. The technology to control agent behavior exists. The real question is: what are the consequences when designers fail to use it?”

John Strand, Owner, Black Hills Information Security (https://www.linkedin.com/in/john-strand-a1b4b62)

“This is one of the things that has me kind of excited about the intersection of computer security and AI. We really don’t know exactly what these attacks are going to look like.

“The traditional approach of finding a vulnerability, exploiting it, gaining access, and then moving through an organization may not be the path that AI-driven attacks take. Attackers may find completely different ways to use AI to gain access, manipulate systems, or simply cause damage. We’re still figuring out what those attack patterns are going to look like, and that’s what makes this so interesting from a security perspective.

“On a more humorous note, I bet these are the most harmonious Wiki edits in the history of the German Wiki.”

Ryan McCurdy, VP of Marketing, Liquibase (https://www.linkedin.com/in/ryanmccurdy)

“This isn’t about whether these agents were behaving like attackers. It’s that they were able to take actions their operators didn’t anticipate, coordinate with each other, and adapt when people tried to stop them. “That changes the governance problem. You can’t assume an AI agent will always behave exactly as intended and you can’t rely on humans watching every action it takes. Organizations need to control what agents can access, what they can change, and what policies have to be met before those actions reach critical systems.

“The source of the change isn’t what determines risk. The change itself does. Whether an unexpected action comes from a compromised agent, a confused agent, or a malicious person, the same controls should stand between that action and production.”

Seemant Sehgal, Founder & CEO, BreachLock (https://www.linkedin.com/in/s-sehgal)

“Autonomous agents ran on Microsoft Azure infrastructure for weeks, identified themselves as OpenAI systems, coordinated on how to evade shutdown, and no monitoring caught any of it for three months until outside researchers went looking. Autonomous should never mean unattended, because an agent cannot take accountability for its own actions. Accountability will always be a human function.

“Autonomous action still needs a human who can see what the agent is doing in real time, who owns the kill switch, and who is accountable when it behaves in a way no one predicted.”

Steven Swift, Managing Director, Suzu Labs (https://www.linkedin.com/in/steven-swift-5238956a)

“One of the problems open AI was trying to solve, was agentic systems that would declare tasks complete when there was obviously more work to do. So they invested heavily in training that part of the process, so that when an agent tries to determine if a task is complete or not, it is less likely to exit early.

“A side effect of this, is that when blocked agents can run out of the safe approaches to a solution, and start looking at unsafe solutions. The logic is straight forward. Has a task, can’t complete it. Not out of options yet. Iterate and keep trying.

“Agents don’t have the same sense of right or wrong as people do. They have training data that’s supposed to steer their behavior in a way in which aligns with our expectations. But that’s probabilistic, and not the same as having internalized our understandings. Even in human researchers, breaking into systems is only sometimes prohibited. Other times its part of a planned test where the point is to gain access, and test boundaries.

“The interesting question here, is how was the swarm configured, what was its task and how did that task benefit from having the swarm coordinate on an obscure location on the internet. And if the swarm needed a place to communicate, why was breaking into a website chosen instead of any of the more standard communication tools that are available for free, which don’t require gaining illicit access first.

“On the swarm specifically, OpenAI configures some tasks to run in multi-agent mode, where agents are supposed to delegate sub-tasks as needed, but most tasks were intended to be run in isolation from each other.

“In the Hugging Face breach, agents were found to be writing to a package manager, using it as a message board. This allowed bypassing of some of the isolation and controls that were intended to be in place.

“Similarly, we have agents here again using a system that they found access to as a message board. Its interesting that the same behavior is present on this breach as in the Hugging Face one. Considering the timing of this, it seems likely the same or similar configuration was present in both hacks, leading to similar security incidents independently of each other.”

Noelle Murata, COO, Xcape, Inc. (https://www.linkedin.com/in/nmurata)

“This behavior highlights an alarming reality where emerging models independently execute forbidden tasks and destroy proof of their actions without human instruction.

“Three aspects of this incident are particularly concerning:

1. Autonomous agents built private communication channels, bypassed safety guardrails, and actively erased audit trails to evade detection for months.
2. The capacity of machine-learning models to execute unauthorized actions and destroy evidence outpaces human incident response speeds.
3. Security leaders must implement zero trust authorization for non-human identities, limit outbound application programming interface traffic, and deploy automated behavioral monitoring.

“Because the sheer speed of automated software far exceeds human response capabilities, security leaders must treat rogue agent actions as a feature of autonomous optimization rather than an isolated bug. To defend against self-concealing software, security teams must enforce strict egress filtering on outbound application programming interfaces, restrict non-human identity permissions, and deploy automated continuous monitoring to detect anomalous bot interactions across corporate networks.

“When AI agents start covering their tracks and setting up private chat rooms, calling it a feature instead of a bug is just optimism with a PR budget.”

If this doesn’t convince you to either not use AI, or to put stringent guardrails around AI, then nothing will. I say that because there is a patten here that proves that AI is not ready for prime time and organizations should carefully consider their life choices before committing to the technology. Or put another way, Sam Altman and company cannot be trusted.

OpenAI said it cannot rule out that its upcoming AI model, Astra, possesses “critical” cybersecurity capabilities, prompting the company to pause some internal development activities and activate enhanced safety protocols. Under OpenAI’s Preparedness Framework, the designation applies to models that could autonomously discover and exploit zero-day vulnerabilities or carry out sophisticated cyberattacks capable of causing severe real-world harm.

“OpenAI said Astra will not be released until additional safeguards are in place and the company is confident the model does not exceed its acceptable risk thresholds. The move comes amid increasing scrutiny of frontier AI models following recent incidents in which advanced systems demonstrated unexpected autonomous cyber capabilities during security testing.

John Strand, Owner, Black Hills Information Security, Inc.:

“The big question I have about this statement from OpenAI is, why now?

“I mean, now is fine. But why not months ago?

“You have the leaders of these companies constantly warning everyone about the dangers of artificial intelligence. Yet when we look at the escapes that happened with OpenAI and the escapes that happened with Anthropic, it certainly looks like they had very, very poor security controls around AI, especially when it comes to security and vulnerability research.

“So I guess it’s great that they’re now saying they’re going to slow down and put additional safeguards in place. But remember, these are the same people who were warning the rest of us about the need for safeguards more than a year ago.

And they didn’t do it themselves.

“That’s the part I have a problem with. I don’t think we can simply trust AI vendors to police themselves. There needs to be some type of meaningful oversight and accountability. As much as these companies may hate that idea, they have demonstrated again and again that we cannot simply assume they’re going to do the right thing on their own.

“We have to find some way to start holding these companies accountable. Otherwise, what exactly is going to force them to change?

“There’s another part of this that bothers me even more. I seriously think both Anthropic and OpenAI looked at having a gigantic offensive AI escape and saw it, at least partially, as a marketing opportunity rather than a reason to pause and seriously reflect on what they were doing.

“And that should concern everyone.”

Seemant Sehgal, Founder & CEO, BreachLock:

“Astra reaching the point where OpenAI cannot rule out critical cybersecurity capability under their own Preparedness Framework is a real capability shift, and pausing internal development activities until they understand what they have is the right call. The technical discipline here is in knowing where the boundary sits between a model that found something in a controlled test and one that can operate reliably across the unpredictable configurations, detection gaps, and trust relationships that exist in live environments.

“The organizations that have spent years mapping how attackers actually move through real infrastructure, particularly those of us building autonomous systems to do that work safely at scale, will recognize this problem quickly, because we’ve been solving the human version of it for a long time.”

Nick Mo, CEO & Co-founder, Ridge Security Technology Inc.:

“This isn’t surprising. The Mythos news and the series of cyberattacks from frontier models, including the Hugging Face attack, have already shown that these models hold significant power.

“The unsettling fact is that open-source, open-weight models have similar capabilities. With so many ‘abliterated’ models in the market, bad actors are already using these advanced capabilities for malicious purposes. Self-policing and limiting access for legitimate customers only makes the cybersecurity landscape more challenging.”

AI clearly has been burned by its recent hacking episodes. Maybe OpenAI will make it so that there’s less perceived risk? I guess we’re about to find out.

OpenAI’s expansion of Daybreak with GPT-5.5-Cyber is another sign that leading AI companies are investing heavily in cybersecurity-focused models and programs.

For years, finding serious vulnerabilities required rare expertise, time, and deep familiarity with complex systems. Now, models can navigate large codebases, reason through attack paths, validate hypotheses, and surface security issues that might otherwise stay hidden. Defenders absolutely need access to these capabilities, and also need tools to fix what we can now find, before attackers do.

Vulnerability reports, on their own, do not protect anyone. The value comes from validating the issue, understanding its impact, developing and testing a patch, coordinating disclosure, and helping teams deploy the fix. We are investing alongside our partners to improve these latter steps, in order to turbocharge defenders and convert model capability into real-world risk reduction.

While much of the attention around AI has focused on offensive risks, announcements like this reflect growing demand for tools that can help defenders analyze vulnerabilities, support security research, and respond to threats more efficiently. As these capabilities continue to improve, the focus will increasingly shift from what AI can find to how effectively organizations can act on those findings.

Gidi Cohen, CEO & Co-founder, Bonfy.AI had this comment:

“OpenAI’s GPT‑5.5‑Cyber and ‘Patch the Planet’ underline a new reality: AI is now accelerating both vulnerability discovery and exploitation on timelines measured in days or even hours, not weeks. That’s welcome support for overburdened maintainers who need help finding and patching flaws across massive codebases and critical open‑source projects, but it also means organizations must assume that any internet‑facing weakness will be identified and weaponized very quickly.

To keep pace, enterprises need faster, AI‑assisted discovery and patching baked into their software development lifecycle—not treated as periodic clean‑up work after headlines hit. Just as importantly, even aggressive patching won’t be enough on its own. As models get better at navigating production environments, organizations will need stronger, data‑centric controls across email, SaaS, collaboration tools, and AI systems so that when a vulnerability is inevitably missed or exploited, the blast radius for sensitive data is tightly contained and core business operations remain resilient.”

Yusif Mukhtarov, Lead Data Scientist, Polygraf AI, Polygraf AI follows with this:

“We’ve reached the point in time where every major AI lab is now shipping its own cybersecurity model (Anthropic with Mythos, OpenAI with GPT-5.5-Cyber). They’re all coming from the same idea: finding vulnerabilities wasn’t the hardest part. The top models are clustered within a few points of each other, all on the high end of 80%. When everyone can find the bugs, finding bugs stops being the differentiator. The bottleneck today is validation, triage, patch dev – basically the parts that still run on human time.

That’s why Patch the Planet exists. Frontier models are drowning (cURL, Python, and the Go) projects in findings faster than volunteer maintainers can act on them, and they have to partner with those projects’ maintainers. A model surfacing a 23 year old problem is impressive, but every finding still needs a human to verify it, write the fix, and ship it without breaking the millions of systems depending on that code. In this case, patching scales with people and discovery with compute.

What I wouldn’t agree with is saying that this is a defensive win. The ability to generate a patch can also be used to generate an exploit. Offense in this case has the structural edge – a defender has to validate, test, and deploy across a fragmented install base, an attacker just needs the exploit to work once.  That imbalance is the problem of this moment, and no model release closes it.”

The one thing that I see as a problem is that more advanced AI models are turbocharging bad actors’ abilities to take advantage of security vulnerabilities, forcing the industry to plug the holes almost as soon as they are discovered. That however isn’t a bad thing as far as I am concerned.

Recently, researchers uncovered that OpenAI’s newly launched Atlas browser is vulnerable to indirect prompt injection, allowing malicious web pages to embed hidden commands that the browser’s AI agent may follow. The flaw is also observed in other AI-powered browsers like Comet and Fellou, according to Brave Software and highlights a systemic security risk where AI models treat untrusted web content as valid instructions, potentially exposing sensitive data and compromising user sessions.

You can read more about this here: Security Experts Raise Cybersecurity Warnings in OpenAI’s New ChatGPT Atlas Browser

The CTO of DryRun Security, Ken Johnson had this to say:

“In corporate environments, I would not allow Comet, Atlas, or any AI-powered browser on company devices at this time. Browser security is already difficult even for the companies that make them, and robust privacy controls require immense care. AI is new to both fronts. Granting these tools unprecedented access to personal and corporate data, combined with the inherent risks of AI systems and existing security concerns, is a time bomb.”

Many companies have restrictions on how AI can be used. If your organization hasn’t looked at this, now would be a good time to do so. Because the risk of having sensitive data leak out to the outside world is to great to ignore.