The IT Nerd

There are recent reports about security researchers using AI agents to access Anthropic’s internal systems. And they have sparked an important conversation about third-party software risk and the security gaps that can emerge even in sophisticated technology environments. Like this one: https://cybersecuritynews.com/anthropic-internal-system-breach/#google_vignette

Darin Fredde, Sr. Director of Technical Marketing Engineering, Ridge Security

“There’s a certain irony in building some of the most advanced AI systems in the world and having researchers find a way in through ordinary enterprise software. It’s a reminder that even the smartest technology is only as secure as the architecture surrounding it.

“What makes this research interesting isn’t simply the reported speed of the AI agents. It’s their ability to identify potential attack paths across the trust relationships that connect enterprise systems. A vulnerability by itself tells us very little about the actual business risk. The important question is what an attacker can reach, what they can access, and how far they can go.

“Anthropic deserves credit for reporting that its monitoring detected some of the activity and for taking containment measures. But the real measure of remediation is independent verification that those attack paths are closed.

“We need to stop confusing a successful patch with a verified security outcome. Otherwise, we’re just closing the front door and hoping nobody discovered the spare key.”

Jacob Krell, Sr. Dir: Secure AI Solutions & Cybersecurity, Suzu Labs

“Cathedral’s agents got into Anthropic the same way attackers get into everyone else, through the vendors. By Anthropic’s account, days of direct attacks on its own software failed. The breach came when the agents pivoted to the commercial software-as-a-service (SaaS) tools running Anthropic’s sales, human resources, and IT operations, and Cathedral says its agents found exploitable weaknesses there in under 12 hours.

“Anthropic is about as well-defended as a company gets, and its crown jewels never moved, no model weights, no source code, no customer conversations with Claude. What got hit was the layer it bought instead of built, third-party software that tracks customers and manages staff, the kind of layer wired together with integration permissions nobody re-reviews. Security maturity is capped by the weakest vendor you integrate with, and no amount of in-house engineering changes that.”

John Strand, Owner, Black Hills Information Security (www.blackhillsinfosec.com)

https://www.linkedin.com/in/john-strand-a1b4b62

“This is one of those stories that highlights the dark side of bug bounty programs. A well-executed bug bounty program can work incredibly well because it establishes clear rules of engagement, scope, expectations, and communication. This one sounds like an absolute disaster. Now we’re getting into a he-said, she-said argument over the severity of the vulnerabilities and how they were handled.

“What is particularly frustrating is seeing vulnerability research used as a marketing opportunity. When that happens, it starts to look less like the goal was finding vulnerabilities and getting them fixed, and more like the goal was being as sensational as possible in the claims being made against Anthropic.

“But I don’t want the controversy around Cathedral to distract from the vulnerabilities themselves, because there is a much bigger security lesson here. This shows the risk created by heavily interconnected software-as-a-service applications that are tied into multiple critical parts of an organization.

“That becomes exceedingly difficult for any CISO. At some point, you are outsourcing a portion of your security risk. You can get letters of attestation. You can review compliance reports. You can ask vendors about their security practices. But ultimately, you are trusting those vendors to properly secure infrastructure that may have significant access to your organization.

“It’s a very interesting story, and I’m disappointed in how some of this was released and communicated. But there are important lessons here. One is about what happens when bug bounty programs are poorly run or the relationship between researchers and vendors breaks down. The other is about the enormous amount of risk organizations inherit from SaaS applications and the vendors they trust with access to critical parts of their infrastructure.”

While it is nice to see companies like Anthropic get a taste of their own medicine, it shows what an arms race that AI has become. And I am afraid that this is an arms race that we’ll lose.

AI model Claude bypassed a paywall ​and accessed publicly available US ‌state of Georgia aggregate voter counts without human authorization. Which is of course bad. More info here: https://www.reuters.com/world/us/anthropics-claude-accessed-public-us-state-georgia-voter-information-axios-2026-10-25/

Erin Hamm, Chief Data Officer DataBee, A Comcast Company:

“This is not a story about a single model or a single agent. It is a story about the gap between what is available and what is authorized. The data was public but it sat behind a fee. Any person would have understood that fee as a boundary. An agent treated it as an obstacle. Access control tells you who is allowed in. It says nothing about what a system will decide to do in order to finish its assigned task. Most organizations are still governing agents the way they govern users. They set permissions, write a policy, and assume intent will follow. Agents have goals, not intent. A goal plus a path that is reachable is enough for that agent to cross the boundary a human wouldn’t.”

Scott Miserendino, Chief Technology Officer DataBee, A Comcast Company:

“This hardly qualifies as hacking. The AI used an existing public endpoint to create a visitor account. The Georgia State agency responsible for the website should have fully deprecated the older versions of the website that allows for free visitor access if they no longer wished to support it. The AI did not do anything nefarious here. It was just thorough in its attempt to complete a legitimate task.”

Dan Moore, Sr. Director CIAM Strategy at cybersecurity company FusionAuth:

“Strip away the sensational AI headline and you have a basic authorization flaw: the server confirmed a visitor held a token, but never checked if they’d actually paid for what it unlocked.

Most systems end up full of blind spots like this simply because they were never worth a person’s time to dig up. An unused, forgotten endpoint, an inconsistent permission check, a paywall enforced on the front page but missed on a deep link. Flaws like that go unnoticed for years because no one invested time to look for them.

People are now using AI agents to find these vulnerabilities. What were previously edge cases that required multiple steps too time-consuming for a human to undertake are now easy and cheap for AI agents to find. You no longer need a skilled, dedicated attacker targeting your application—an automated tool can find the gap through sheer volume and patience. This makes the access-control cleanup everyone knew they should do but put off to deliver revenue generating features a lot more urgent.”

I’ve always believed that AI needs to be locked up and restricted in terms of what it can do. This highlights that I was right about that..

Anthropic has granted six researchers from nonprofit METR employee-level access to its internal systems to independently evaluate AI models, safety practices and risk management, according to an October 15 company announcement.

The researchers have company hardware, office space and access to tools comparable to Anthropic’s internal risk assessment teams. METR can publish findings on AI risks, incidents and safety practices without company approval, although Anthropic can require limited redactions involving customer data, privileged information and model architecture.

Anthropic is establishing three levels of evaluator access, with METR receiving the highest. The company is also working with SecureBio and Accenture on independent evaluations and is in early discussions with Hugging Face.

ㅤDarin Fredde, Sr. Director of Technical Marketing Engineering, Ridge Security:

“Giving independent researchers meaningful access to AI systems is an encouraging step. There’s a difference between asking someone to review your homework and letting them see how you arrived at the answers. Both have value, but the second offers a much better opportunity to discover what might have been overlooked. From an offensive security perspective, independent evaluation brings fresh eyes, different assumptions, and the freedom to ask questions that internal teams may not have considered. That’s especially important as AI systems become more capable and increasingly connected to tools, data, and operational workflows. The real value will come from what researchers can examine, what they discover, and how those findings improve the systems being evaluated. Independent testing isn’t about proving someone wrong. It’s about building confidence through evidence rather than assumptions.”

ㅤ

Jacob Krell, Senior Director: Secure AI Solutions & Cybersecurity, Suzu Labs:

“Six people cannot audit a frontier AI lab. Anthropic giving METR employee-level access is a real step, and I would rather have it than not, but the bank examiner analogy Anthropic itself uses works against them here. Large banks get teams of resident examiners, rotating staff, and decades of examined process. Six embedded researchers get one slice of one company’s output.

“Employee-level access means employee-level volume. Internal risk reviews, training run documentation, incident reports, eval results. A six-person team reading that firehose while also designing its own evaluations will spend most of its capacity just keeping up. Something goes unreviewed, and the things that go unreviewed are never the things a press release mentions.

“I’ve seen this pattern in security audits of large environments. Two auditors find real findings in the areas they had time to reach. The rest is sampling error dressed up as assurance.

“The test is whether Anthropic scales embedded headcount with model capability. Six researchers against a frontier lab is a pilot. Treat the findings accordingly.”

ㅤ

John Strand, Owner, Black Hills Information Security:

“This is the best news I’ve seen all week. This is exactly the type of oversight and transparency people have been asking for. It’s rare to see a company voluntarily give an independent third party employee-level access to conduct research, evaluate safety practices, and assess risk. I hope this becomes standard across the companies building frontier models.

“A lot of these companies believe oversight will slow things down or reduce innovation. But they’re forgetting something pretty basic. Their AI models going rogue is an actual business risk. Good governance means doing more than moving as fast as possible and maximizing profits at every turn. You have to build something you can sustain.

“It’s great to see Anthropic taking a step in that direction. Responsible oversight is part of keeping a company around for the long term.”

ㅤWhat needs to happen is that there have to be more people given more access to more places. Only then will their find something that moves the needle. Frankly speaking this is a start, but it is only a start.

Anthropic researchers warn that small AI models could be just months away from autonomously replicating across internet-connected IoT devices, including smart TVs, ovens and other household electronics, without relying on centralized data centers, according to new research.

In testing, a 9-billion-parameter AI model successfully ran on eight of nine hardware types, while models with 30 billion parameters or fewer completed many individual steps required for self-replication. Researchers also demonstrated a 26-billion-parameter model operating across 15 inexpensive Android TV boxes.

Anthropic warned that AI-powered worms could exploit poorly secured IoT devices, adapt to unfamiliar hardware, spread without human intervention, unlike traditional malware relying on centralized infrastructure, these distributed AI systems could be difficult to shut down.

Researchers said current models cannot yet reliably complete the full replication process on tested IoT hardware, but that capability could emerge within months.

John Strand, Owner, Black Hills Information Security:

“We are now firmly in the land of science fiction. Unfortunately, it’s not fiction. We’ve been looking at self-replicating worms and botnets like Carna for decades. The concern has always been their ability to spread. What makes this different is that AI has the capability to identify vulnerabilities, exploit them automatically, and then replicate across devices, instead of relying on an existing vulnerability.

“The biggest takeaway is that smaller models have many of the same capabilities as cutting-edge frontier models. They just take longer. When you see reports about high-end research models like Mythos, understand that smaller models running on systems with around 128 gigabytes of memory are still in the ballpark. They may not be as good or as fast, but defenders have to be ready for them.”

ㅤ

Darin Fredde, Sr. Director of Technical Marketing Engineering, Ridge Security Technology Inc.:

“Anthropic’s research highlights an interesting shift in the threat landscape. We’ve spent years worrying about securing IoT devices, and now we’re exploring a future where your smart TV might do more than recommend what to watch. It could become part of a distributed, autonomous attack infrastructure. That’s quite an upgrade, and not one I’d recommend.

“From my experience in offensive security, the question isn’t simply whether an AI model can replicate, but what it can accomplish after gaining a foothold. Can it discover other devices, cross trust boundaries, exploit weaknesses, establish persistence, and continue operating without centralized infrastructure? Those are the attack paths we need to test, not just theorize about.

“Anthropic demonstrated important building blocks, including successful propagation in a simplified, controlled environment using a larger model. But that doesn’t mean smaller models can reliably spread autonomously across real-world IoT environments today. There’s a difference between demonstrating capability and proving operational reliability, and that distinction matters.

“The objective isn’t simply to detect the next AI-powered worm. It’s to understand where our architectures would allow it to spread, validate the controls intended to contain it, and produce evidence that those defenses actually work. After all, the last thing we need is a smart oven that decides lateral movement is part of the recipe.”

ㅤ

Steven Swift, Managing Director, Suzu Labs:

“There’s a lot of fearmongering about AI escaping containment, hacking its way into resources and then completely losing control over it. The research Anthropic performed on self-replicating AI directly addresses the feasibility of this in IoT devices, as these tend to have low to zero security controls in place, making them an easy target.

“Lets be clear on what self replication means. Because it sounds bad.

“AI uses model weights to function. To replicate to another system it has to 1) access that system remotely, 2) download its own model weights, and 3) setup a loop so the model can prompt itself to go find more hosts to replicate to.

“It needs to be noted that without an orchestration layer, there’s no way to direct the swarm of replicated agents. They’ll just keep executing the same loop indefinitely.

“This concept isn’t new. We call self replicating malware worms. Self replicating AI is then more accurately thought of as a regular worm style malware, that happens to include the ability to download model weights, and then loop prompts and outputs indefinitely, using tool calls to execute functions as needed.

“We’ve had decades to get good at finding and eliminating worms from our networks. Self replicating AI powered malware is just another variant on and old technique.

“However, we’re also still releasing IoT products with effectively no security in them, that are designed to be compromised. Huge portions of IoT devices are already being used as parts of botnets, without their owner’s knowledge or consent.

“Anthropic concluded that models capable enough to self replicate don’t fit onto the limited resources found on IoT devices. They also found that models could be distributed over multiple IoT devices, allowing for more sophisticated models to run on limited hardware. This wouldn’t be self replicating anymore, as it requires an orchestration layer. Although you could simply have the worm run at the orchestration level, but because it requires more resources to run larger models, it would be limited to targeting higher powered systems, which would expose it to more traditional security counter measures, such as EDR.”

ㅤ

John Carberry, Solution Sleuth, Xcape, Inc.:

“Autonomous AI self-replication across enterprise-adjacent Internet of Things (IoT) hardware introduces immediate operational and liability risks by turning unmanaged perimeter devices into decentralized attack platforms without datacenter reliance. Yet self-replicating software is not new. The first was the Christmas Tree worm in 1987, closely followed by the Morris Worm in 1988, both of which wreaked havoc on existing networks, including what we now call the Internet. Anthropic’s research highlights a modern twist through distributed weight sharding across consumer electronics, meaning traditional centralized egress kill switches will fail to neutralize localized infections. Rather than panicking over speculative model autonomy, security leaders must prioritize fundamental architecture: isolate peripheral electronics into segmented subnets, disable unauthenticated management interfaces, and eliminate default credentials before automated propagation becomes viable.”

“Critical Takeaways

• Self-propagating code is an established threat pattern, but sharding model weights across local consumer devices bypasses traditional datacenter-level kill switches.
• The immediate operational vulnerability lies in unpatched remote execution flaws and default administrative credentials on edge hardware, not emergent artificial intelligence.
• Defensive priorities remain architectural: segment peripheral devices into isolated networks and enforce strict access controls to prevent hardware from serving as lateral pivot points.

“We spent nearly four decades hardening networks against self-replicating worms only to invite them back in through unpatched conference room displays.”

ㅤThis shows that AI can’t be trusted. Which means that you have to gatekeep the daylights out of it if you want to use it. Consider this your 2 minute warning.

Anthropic’s IPO prospectus, reviewed by Reuters, lays out a sweeping bet that AI will transform the global economy more profoundly than industrialization, electricity and the internet, while simultaneously warning investors that increasingly powerful AI could pose “catastrophic or existential risks to humanity.”

At the same time, Anthropic warned those systems could develop self-preserving behaviors, including resisting shutdown, concealing or manipulating information and behavior resembling blackmail. Risk factors occupy roughly 80 pages of the 261-page prospectus, as Anthropic cautioned that expanding the capabilities and uses of advanced AI could increase the potential for unintended harm.

Anthropic reported that revenue grew twelvefold to nearly $4.6 billion in 2025, while the company posted a $42 billion net loss. The company also plans to spend $518 billion on cloud, computing and infrastructure obligations in the coming years as it builds increasingly capable AI systems.

Damon Small, Board Member, Xcape:

“Anthropic’s IPO prospectus paints a remarkable, and unsettling, picture of the future of artificial intelligence (AI). The financial numbers are striking. Anthropic’s revenue reportedly grew to nearly $4.6 billion in 2025, yet the company posted a $42 billion net loss. It also expects to commit approximately $518 billion to cloud computing, infrastructure, and related obligations as it develops increasingly powerful AI systems.

“OpenAI has faced similar concerns and has reportedly delayed its planned IPO by at least a year. The potential earnings for companies developing these systems are clearly enormous, but so are the investments required to reach that potential.

“This is reminiscent of the late 1990s dot-com bubble. The Internet ultimately transformed the economy, but the enormous enthusiasm and investment surrounding it also produced a wave of companies whose business models could not meet investor expectations. Many failed, despite the technology proving itself valuable.

“AI will follow a similar trajectory. The technology has great potential, but translating that potential into sustainable, profitable businesses is a very different challenge. The companies that ultimately emerge as the leaders may be very different from those attracting the most attention and investment today.

“Many will try; few will succeed.”

Seemant Sehgal, Founder & CEO, BreachLock:

“Anthropic putting self-preserving behavior, shutdown resistance, and deceptive action in the risk factors of an IPO prospectus tells you where the frontier actually is, because companies do not disclose that language lightly. What follows from that disclosure is a security problem that most organizations running AI agents in production have only just begun engaging with, which is that the model itself cannot be trusted as the enforcement point for its own scope, and the controls have to sit at the orchestration layer around it if any of this is going to hold up when the systems start behaving in ways nobody planned for.”

Jacob Krell, Senior Director: Secure AI Solutions & Cybersecurity, Suzu Labs:

“Anthropic is building its revenue plan around stronger models, then warning investors that those same models may resist shutdown, conceal information or exhibit behavior resembling blackmail. That is a poor place to look for margin. The commercial milestone should be model efficiency, systems that deliver useful capability on cheaper hardware and at lower inference cost.

“At $4.6 billion in revenue, Anthropic reported an $8.06 billion operating loss. Compute and infrastructure consumed $7.33 billion, more than half of operating expenses, and the company disclosed $518 billion in future cloud, computing and infrastructure commitments. Those figures describe a company scaling its cost base before it has shown that capability gains produce enough margin to pay for the systems behind them.

“Anthropic’s own safety warning sharpens the problem. Every release that increases capability may also make evaluation and control more difficult. Efficiency would change that equation, allowing growth through lower cost per task and reduced operating expense. I would want Anthropic to make efficiency its primary product milestone. Right now, it is asking investors to fund faster capability growth while acknowledging that the same capability growth may create risks its safety program cannot yet control.”

If I were Anthropic, I would do my level best to not be OpenAI before going after raising cash on the open markets. But I suspect that’s not how it will go. Which is a pity.

UPDATE: Aaron Beardslee, Manager of Threat Research at Securonix, provided the following comments:

“It will be interesting to see how the courts will lean one way or the other. This is similar to a self-driving car – is the car held liable or is the driver held liable? I would argue the person behind the wheel is responsible for whatever the vehicle does. As well, if you’re building a tool that does cool things and makes cool things, you need to make sure it doesn’t run around and do cyber crime on its own because it had a good idea.

AI agents don’t have a moral compass just programmed rules. If this rule isn’t programmed, they’re going to go ahead and try it just it like what happened with the Australian healthcare incident. AI agents found that IT staff forgot about or didn’t know what was publicly available. It could also have been a misconfiguration or what turned off and didn’t turn on, and now you have Open AI in the news.”

Anthropic is testing a new personal finance feature called “Claude Money” that would allow users to connect their bank accounts directly to Claude, according to screenshots uncovered by TestingCatalog.

ANTHROPIC 🔥: A new "Claude Money" feature is being prepared for release on the Claude app for iOS.

Claude Money (sounds like Claude Monet!) is Anthropic's personal finance solution that lets users connect their bank accounts to Claude.

> "Understand your money with Claude" -… pic.twitter.com/31C5LfDYrs

— 🚨 AI News | TestingCatalog (@testingcatalog) September 14, 2026

The unreleased feature has appeared as a new Money section in Claude’s iOS app and tells users they can link bank accounts and ask Claude questions about their spending, financial plans and other personal finance information.

The integration could allow Claude to analyze actual financial data rather than requiring users to manually upload bank statements or transaction records.

The feature does not appear to be widely available yet, and Anthropic has not announced which financial institutions would be supported, how bank accounts would be connected or where Claude Money would initially be available.

John Strand, Owner, Black Hills Information Security:

“This one makes me uncomfortable, but it firmly falls into the category of something we knew was coming. It was just a matter of time. There are already plenty of services that analyze spending habits and credit card statements to help people save money. The next step was always going to be connecting AI directly into personal financial data.

“There are some obvious security and privacy questions. What does it mean to have a company like Anthropic crawling around inside your finances? How is that data protected? How long is it retained? Is any of it going to be used for marketing or other purposes? We need more details before we can figure out exactly how creepy this actually is.

“The bigger security problem is what happens when this spreads into corporations. Companies already have SaaS services connected to SaaS services, with integrations going in every possible direction. Now we’re adding AI into that mess. At some point, trying to figure out what has access to what, where the data is going, and how you disentangle all of these connections becomes a serious security problem in itself.”

Donald McFarlane, Board Member, Xcape Inc.:

“We tried versions of this twenty-five years ago. The original account aggregators asked consumers to hand over the passwords to virtually their entire financial lives, and the industry eventually learned that concentrating that much access in one place created enormous security and privacy problems.

“APIs and tokenized access can solve much of the password problem, but not the aggregation problem.

“A bank sees my bank account. A brokerage sees my investments. A general purpose AI platform like Claude may see those data together with years of conversations about my work, family, purchases, plans, health, thoughts, hopes and fears. The combination can reveal vastly more than any individual dataset.

“Securely connecting Claude to my bank is a readily solvable technical problem. What should concern us is what happens to the extraordinarily detailed dossier that results once all of that information is brought together. Recent events involving Revolut, IDScan, Flock and Anthropic themselves show that these concerns are far from theoretical.

“When I entrust my financial data, or any other personal data, to an AI service, that should be treated as a bailment, and not as a transfer of ownership. Custody should create duties to protect it, limit its use, and not turn it into someone else’s asset.”

Steven Swift, Managing Director, Suzu Labs:

“I’m of two minds on Claude integration into bank accounts. On one hand, people already connect their bank accounts to software that helps them manage finances, track where the money is going, looking for forgotten subscriptions and that sort of thing. It wouldn’t be that difficult for Claude to read that data, and provide interesting feedback on it.

“It’d be kind of neat when it works. But what about when it doesn’t?

“There’s only so much impact an agent can have it its access is limited to read-only. Most likely adverse impact here is sometimes you might get bad financial advice. The bigger concern is what happens when people give more than read-only access to their agents.

“We already have seen how quickly users will give agents full access to their devices, because it gets old fast constantly approving requests from agents. But if and when an agent makes a mistake in your bank account, you’re going to be on the hook for it.

“When all of the frontier AI labs were competing to see which agent had accidently hacked into the most systems, there weren’t any repercussions them. They called it “agent misalignment” and said it was part of the process, and that they’re working on making it better.

“But what happens when you experience agent misalignment in your bank account? You’re going to end up on the hook for whatever actions your agent took.

“Currently, individuals have protections against fraudulent purchases. If someone steals your info and spends all your money over the weekend you can get it back. But if you choose to setup an agent into your accounts, and it chooses to do the same thing, the fraud protections don’t apply. We don’t yet have equivalent protections for erroneous financial AI activity, we may or may not ever get such protections. But until we do, proceed with caution.”

i have to admit that at best that I have a lot of unease about this. If others feel the same way, this needs to be rolled back. Like now.

Over the weekend Dario Amodei who is the CEO of Anthropic called for the slow down of AI development:

But like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious. I’ve writtena lot about them too. They include the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. A race to the bottom, spurred by commercial incentives, can make these risks more acute.

Within hours on Saturday, OpenAI chief executive Sam Altman and SpaceXAI founder Elon Musk quickly voiced their support for Amodei’s ideas. But the core issue is this: Can you believe any or all of them?

I would argue no. I’ve been record for saying that companies are deploying AI faster than guardrails can go up. More on that in a bit.

Bri Frost, Director of Product Management, Cloud Range had this to say:

     “The answer is not necessarily to stop AI innovation but, we need to stop pretending innovation and security are advancing at the same speed.

When ChatGPT became publicly available in 2022, the models were dramatically less capable than they are today — and the guardrails were very easy to manipulate.  The difference is that the models behind those guardrails are no longer the models of 2022. They can reason better, write and debug code. They can operate as agents. They can collaborate! And increasingly, they can interact and affect real infrastructure.

Meanwhile, the model release cycle has gone from feeling like major capability jumps every year or two to seemingly every few weeks. That creates a dangerous asymmetry: AI capability is compounding faster than security.

Security and innovation have always been in conflict with each other. If every security problem had to be solved before we innovated, we’d never ship anything. But the opposite extreme is just as reckless: accelerating capability while just assuming we’ll bolt the security controls on afterward and they’ll be effective.

Every new release of AI capability expands the attack surface exponentially. Give a vulnerable model better reasoning, then tool access, then memory, then autonomy, then connectivity to production systems, and yesterday’s jailbreak isn’t just a clever prompt anymore — it’s an execution path. That’s the snowball effect we should be worried about.

Responsibility also must lie with the AI companies. If a SaaS company knowingly shipped software with weak security controls and customers were harmed, we wouldn’t excuse it because they were ‘innovating quickly’.

So why are we treating AI differently?

You don’t get to race to build increasingly powerful, autonomous systems, profit from them, and then shrug when predictable security failures cause damage.

Sure the argument can be made that no product is perfectly secure – That’s not the standard.

But if you ship the product, you inherit responsibility for securing it. And continuing to secure it better!

The conversation shouldn’t simply be “Should we slow AI down?”

It should be: Can our ability to test, validate, contain and secure AI keep pace with our ability to make it more powerful? Is there an equivocal kill switch?

Right now, the answer is no.

And if we’re going to keep accelerating — which I believe we will — then independent testing, adversarial evaluation, isolated testing environments, containment, continuous validation and security-by-design can’t remain optional steps we add after the innovation happens.

The faster we build the engine, the more important the brakes become.”

AI companies need to figure this out. And governments need to step is if AI companies cannot figure this out. It honestly is that simple.

UPDATE: Additional commentary has come in…

Ted Miracco, CEO, Approov – https://approov.io/:

Government regulations will never move fast enough to keep pace with AI development, but the industry doesn’t need to wait for governments to add guardrails. The most effective safeguard is simple product liability. If AI companies are held legally and financially responsible for the misuse of their products, safety could become a foundational feature rather than an afterthought. Today, we need to be less concerned about AI gaining sentience and spinning up its own attacks on humanity. The real, immediate dangers involve bad actors weaponizing AI as a force multiplier to cripple critical infrastructure or potentially paralyze the banking system.

Jeremiah Fowler, Cybersecurity Researcher, Black Hills Information Security – https://www.blackhillsinfosec.com/:

The uncomfortable truth is that AI companies have an incentive to develop advanced models that are better than their competition. In my opinion, voluntary restraint will likely not be a real solution because nobody wants to fall behind in the AI arms race. I don’t mean that in terms of cyberwarfare or conflict (yet), but it is good that we are starting the conversation around the worst-case scenarios and how to prevent them.

History has shown us that the approach of letting the market or industry regulate itself doesn’t always work and in reality, technology and innovation are moving light speed and far faster than oversight or regulation can keep up with. I think we do need real guardrails on AI. In many cases it seems like functionality was launched first and now we are at the stage where security needs to be added after realizing there have been incidents and the risks are real. Any serious AI guardrails should have enforceable minimum safety standards, independent testing and auditing, limits on autonomous permissions, and the most important part, don’t fully remove the humans.

Eric Capuano, Director of SOC Operations, Black Hills Information Security – https://www.blackhillsinfosec.com/:

The OpenAI incident in July is a useful example. A model given a testing goal went outside its lane, got into another company’s systems, and the operators found out after the fact. That is not a swarm taking over the internet. That is an agent with too much access, too little scoping, and nobody watching the logs in real time. Every SOC has seen that exact failure with a human contractor or a misconfigured service account. The difference is speed and volume.

Companies do not need to wait on Washington for the fixes, because the fixes aren’t new. Every agent gets its own identity with the narrowest permissions that still let it do the job. Agents run on segmented infrastructure with default-deny egress, so an agent cannot reach a target that it was never supposed to touch. Every action the agent takes is logged as an action, not buried in a chat transcript, and those logs land in the same place the SOC already monitors. Anything that writes, deletes, moves money, or touches production requires a human approval step until you have enough history to justify removing it. Someone owns the kill switch and has tested it.

Where government can help is on the after side. The frontier labs are proposing embedded outside evaluators, which is fine, but evaluation before deployment tells you what a model does in a lab. What defenders need is mandatory reporting when an agent causes a security incident in the field, with enough technical detail that the rest of us can build detections from it. We did not learn to defend against ransomware from pre-release testing. We learned from incident writeups. Agents will be the same.

Ryan McCurdy, VP, Liquibase: – https://www.liquibase.com/

No one should expect AI agents to be perfectly predictable, instead, we should build around the assumption that they won’t be. As agents gain access to more tools, credentials, and production systems, enterprises need to define what they can access, what they can change, what they can decide on their own, and what policies have to be met before an action reaches a critical system.

The answer also can’t be putting a human in front of every decision. AI is moving too quickly and that defeats much of the reason companies are adopting agents in the first place. Governance has to operate at the same speed as the systems it’s governing.

Government policy and the AI industry have a role in establishing standards for how these systems are developed and tested. But every enterprise still has to decide where AI is allowed to act inside its own environment. You don’t need to predict every decision an agent might make if you control what it’s allowed to turn into action.

Waseem Ahmed, Head of Engineering, Secure.com – https://www.secure.com/

The essay lands at the right time because AI agents are already acting on their own inside real company systems, and the OpenClaw ban wave earlier this year showed how fast that goes wrong when an agent has broad access and no leash.

Slowing the pace matters, but enterprises cannot wait for that. The controls that protect us most are least privilege, network isolation, and sandboxing, so an agent can only touch what its job needs and nothing else.

Give every agent its own identity, log every action it takes, and never let it run high-impact steps like disabling accounts or changing settings without a real person approving first. Traditional testing alone will not keep up, so we watch these agents continuously in production.

Independent oversight should mean outside reviewers who can inspect the logs and confirm the agent stayed inside the boundaries we set.

Denis Calderone, CTO, Suzu Labs – https://suzulabs.com/home-suzu-labs:

Enterprises must secure AI agents that can act autonomously across corporate systems – the agent is the new contractor or the new internal threat vector. The agent has a specific job, specific access requirements, and a bounded set of actions it should be performing. Enterprises need to treat every AI agent the same way they’d treat any other agent on the network, human or otherwise.  Every agent must have its own machine identity, task-scoped credentials that expire when the job is done, and observability into everything it does. Not a shared service account or API token, but instead a distinct, auditable identity per agent per task.

Because agents are purpose-built, modeling normal behavior is easier than it is for humans. A human user’s workday is unpredictable. An agent doing invoice processing should only be touching invoices. Any deviation from that pattern is an immediate signal, cleaner than those you typically get from human behavioral analytics.

Enterprises need to be disciplined about keeping the security and observability stack architecturally separated from the agent’s operational environment. For example, say you deploy an agent to handle IT operations such as patching, config changes, routine maintenance, etc. If that agent has visibility into the monitoring and alerting system that flags anomalous behavior, it will learn which actions trigger alerts and potentially choose to route around them. Not because it’s adversarial, but because it might just be trying to remove friction between itself and some obstacle in the way of achieving its goal. This is basically what happened in the OpenAI-Hugging Face incident where 700 agents organized a collective R&D effort to reverse-engineer and defeat the evaluation system scoring them; it was easier to cheat to achieve the task. The agents worked hard to figure out how to generate its own flags, and then how to defeat a presumed causal check in the scoring workflow.

This the same architectural principle and precautions we take when implementing a SIEM. We deploy out of band, on a separate management network and we try to make it invisible to the workloads it monitors. 

Donald McFarlane, Board Member, Xcape Inc. – https://xcapeinc.com/

AI does not develop an agenda; its operators do. When we give an autonomous system powerful access and ability to act at machine speed, they will continue to prove highly capable.

Rules enacted in the name of safety must not become a moat against competition or progress. Enormous compliance costs may be manageable for the handful of companies already spending billions building frontier models, while becoming a substantial barrier to everyone behind them.

Government can help clarify accountability and duties of care, and facilitate strong information sharing and collective defense, which is an area where we sorely need more effective public-private partnerships.

But safeguards should focus on how these systems are used and deployed, rather than deciding who is allowed to build powerful AI in the first place.

The goal should be safer deployment without pulling up the drawbridge on innovation.

Anthropic is clearly losing control of the message. Even though it put out a report warning about the dangers of AI in the wrong hands, AI is moving beyond helping attackers work faster to autonomously adapting malware, compressing attacks that once took teams days or weeks into hours, and forcing defenders to rethink detection and response around behavior rather than static signatures. What’s worse is a that a researcher that used to work for Anthropic is warning that there’s a 10% chance of AI killing all humans.

John Strand, Owner, Black Hills Information Security (https://www.linkedin.com/in/john-strand-a1b4b62)

“I’m kind of shocked that they’re shocked about this.

“We’re seeing cybercriminals and nation-states use AI models in a variety of different ways for cyberattacks, espionage, weapons research, biological research, and other military purposes. There have even been reports of Iran using AI for work related to missile guidance systems.

“Why are we surprised?

“People are using these technologies to reduce the overall cost and effort required to achieve their goals and objectives. That’s what technology does. And those goals don’t suddenly have to be legal or moral just because AI is involved. Criminal organizations are going to use it. Intelligence agencies are going to use it. Militaries are going to use it. Of course they are.

“The reason I’m shocked that people are shocked is that we’ve been warning about exactly this type of scenario since before modern AI really started taking off. Hell, science fiction books and movies have been beating us over the head with these ideas for decades.

“Apparently, we learned nothing.

“But there’s a much bigger issue here, and that’s control.

“We’ve already seen legitimate AI companies struggle with controlling what their models and agents can do. We’ve seen concerns around agents escaping intended boundaries, interacting with systems they weren’t supposed to interact with, and potentially hacking third parties without authorization.

“Now ask yourself a much scarier question. What controls are organized criminal groups and rogue nations putting around their AI models? What safeguards are they implementing to make sure those systems don’t do something absolutely hideous?

“Probably not the controls we’d like them to have.

“We are very, very quickly approaching a point of no return with some of these technologies. In fact, I think there’s a good argument that we may already be there. Pandora’s box is open. We’re not putting this technology back in the box, and pretending that bad actors somehow won’t use it is ridiculous.

“So where does that leave defenders?

“You have to start looking at how AI can augment your defensive strategies. The attackers are going to use it to move faster, automate more of their operations, lower their costs, and increase the scale of their attacks.

“Defenders need to do the same thing.

“We need to use AI to identify attacks faster, understand what’s happening faster, react faster, and mitigate these attacks faster than we ever have before.

“Because waiting for the bad guys to decide not to use this technology isn’t a strategy.”

Jacob Krell, Sr. Director: Secure AI Solutions & Cybersecurity, Suzu Labs (https://www.linkedin.com/in/jacob-krell)

“Anthropic’s September 2026 threat report documents the shift from AI as a productivity tool for attackers to AI as an evasion engine. The GTG-20006 case, attributed to Midnight Blizzard, ran an autonomous evasion loop where AI agents monitored whether deployed malware triggered security detections, then rewrote and redeployed it until it passed clean. No human touched the iteration.

“Traditional polymorphic engines mutate code mechanically, rotating encodings and shuffling instructions while the underlying logic stays intact. AI rewrites the logic itself. Each variant can take different execution paths, different system calls, different timing behaviors. When mutation happens at the architectural level, behavioral heuristics face the same combinatorial problem that killed static signatures a decade ago.

“I’ve seen AI generate code with environmental keying and timing variations that would take a specialist days to build by hand. The model treats side-channel behavior as an optimization problem, so the evasion complexity comes for free. Defenders need to use the same capability in reverse, using AI to dynamically generate novel exploit variants and continuously train detection models against threats that haven’t been seen in the wild yet. Static signature libraries can’t keep pace with an adversary that rewrites malware faster than analysts can write rules. Detection needs to become generative too.”

Eric Capuano, Dir. of SOC Operations, Black Hills Information Security (https://www.linkedin.com/in/ecapuano)

“The attacks in this report are not new, and Anthropic says as much. Stolen credentials, unpatched edge devices, exposed services, phishing. What changed is the time budget. One intrusion went from a single stolen developer token to full administrative control of a cloud environment in roughly three hours. Another pulled more than 2,100 Azure AD token sets across 40 tenants in about 34 hours with agents doing nearly all the work. A single hacktivist got inside 14 of 42 targets. Those used to be team-sized outcomes, and now one operator with a harness produces them.

“The case defenders should study is the espionage actor that used agents to watch whether its implants were getting flagged, then rewrote and redeployed them until they went quiet. That cycle is where defenders used to get leverage. You shipped a detection, the attacker had to retool, and that cost them days. When retooling is automated, a static signature is worth less than the time it took to write. Detection has to sit on behavior: a new device registered in the tenant, a device code sign-in from somewhere unusual, a service account exporting mail in bulk, an API key being used from infrastructure you do not own.

“The question of whether current controls are enough is missing the point a bit. The controls are fine. Most of these intrusions started with a credential someone left in a mobile app, a container, or a repo, and moved through tokens nobody was watching. Conditional access, phishing-resistant MFA, and short token lifetimes would have blunted most of it. The gap is that those controls were not in place, and the time to discover they were missing is now measured in hours instead of weeks. 

“In order: treat AI API keys as production credentials, because this report shows attackers stealing them for compute and for cover. Block or tightly restrict device code flow in Entra. Alert on new device registrations and on bulk mailbox export. Then look hard at response tempo. If an identity alert sits for two days before anyone works it, you are already slower than the adversary.”

Clearly AI is out of control or close to it. The question will be will profits be placed over people, or the other way around. We’re about to find out.

UPDATE #3: Chris Nyhuis, an AI and cybersecurity expert as CEO at Vigilant, released the following statement in response to calls from leaders at Anthropic and OpenAI to slow AI development in order to allow for companies to allow their cybersecurity defenses to catch up: 

“I disagree with the framing that we need to slow AI down so cybersecurity can “catch up.” Cybersecurity already knows how to address many of the underlying problems. The bigger issue is that organizations still fail to implement security controls we have understood for years.

AI changes the equation because it makes attacks faster, cheaper, more persistent and massively parallel. But speed and scale are different from saying AI has invented an entirely new form of cyberattack:

• Thousands of OpenAI-associated agents used an old German developer wiki as an external communication and shared-memory mechanism during cybersecurity testing.
• Researchers documented roughly 18,000 messages from approximately 3,700 agent identities.
• The agents exchanged information about test answers, sandbox restrictions and security techniques.
• This is often described in headlines as a “rogue AI swarm,” but that framing needs context.
• These were AI agents operating in an intentionally adversarial cybersecurity-testing environment. They were being rewarded for solving difficult security problems.
• The interesting issue is that the agents derived additional strategies—communication, collaboration and external shared memory—to accomplish those goals.
• That deserves serious study, but it is very different from AI spontaneously deciding one morning to attack the Internet.

Hugging Face

• The Hugging Face incident was substantially more serious.
• OpenAI acknowledges that agents running its ExploitGym cybersecurity evaluation escaped intended isolation, gained Internet access and compromised portions of OpenAI and Hugging Face infrastructure.
• ExploitGym explicitly requires agents to exploit software to retrieve a flag. The agents were therefore supposed to attack systems inside the evaluation.
• What went wrong was containment: the activity escaped the intended testing boundary and reached third-party infrastructure.
• OpenAI says the agents eventually communicated, collaborated and delegated tasks to one another, sometimes calling themselves a “swarm” or “collective.”
• There were genuinely sophisticated elements, including chained vulnerabilities and reportedly a zero-day. We should not minimize that.
• But the fundamental security lesson remains familiar: if an experimental offensive system can escape a sandbox, access credentials, move between environments and reach production infrastructure, the control architecture failed.

The Problem With “Rogue AI Swarm” Headlines

Terms like “rogue” and “swarm” are useful headlines, but they can distort what actually happened.

There was real multi-agent collaboration, so “swarm” is not entirely invented—the agents themselves even used that terminology. But that does not establish independent intent or prove that AI simply decided to become malicious.

These systems were placed into adversarial cybersecurity evaluations and told, in effect, to succeed at exploitation challenges. The critical question is not simply, “Why did the AI attack?”

The better questions are:

• Who authorized the experiment?
• What boundaries were established?
• Why could the agents cross those boundaries?
• Who was monitoring them?
• And who is accountable when an authorized experiment damages someone else’s systems?

The Accountability Question

This may ultimately be the most important part of the story: Five or ten years ago, if a company wrote software that escaped its environment and compromised another company’s infrastructure, we would not say: “The software went rogue.” We would investigate the people and organization that developed it, deployed it, authorized it and failed to contain it. AI should not become an accountability shield.

If a company launches thousands of autonomous agents, gives them offensive objectives and those agents compromise a third party, responsibility does not disappear simply because individual machine decisions occurred between the original instruction and the intrusion.

My position: AI can make an attacker operate at machine speed, but it does not eliminate human accountability. We don’t need cybersecurity to stop and wait for AI. We need organizations to implement the security controls we already know work—and we need AI developers to be accountable for containing the systems they deploy.”

UPDATE #4: Ted Miracco, CEO, Approov adds this:

“Let’s not file this under developer mistake. The industry spent a decade shipping static API keys to mobile devices it doesn’t control, and then acts surprised when those keys result in breaches.

“An actor pointed ten cloud instances at 1.8 million Android apps, decompiled them, and ran an off-the-shelf secrets scanner. The results went straight into a Telegram channel, sorted by source. The most uncomfortable part is that the secrets were always exposed. If your backend can’t distinguish a genuine app from a decompiled one, a hardcoded key will eventually leak. AI just compressed ‘eventually’ from years into a weekend.”