The IT Nerd

Over the weekend Dario Amodei who is the CEO of Anthropic called for the slow down of AI development:

But like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious. I’ve writtena lot about them too. They include the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. A race to the bottom, spurred by commercial incentives, can make these risks more acute.

Within hours on Saturday, OpenAI chief executive Sam Altman and SpaceXAI founder Elon Musk quickly voiced their support for Amodei’s ideas. But the core issue is this: Can you believe any or all of them?

I would argue no. I’ve been record for saying that companies are deploying AI faster than guardrails can go up. More on that in a bit.

Bri Frost, Director of Product Management, Cloud Range had this to say:

     “The answer is not necessarily to stop AI innovation but, we need to stop pretending innovation and security are advancing at the same speed.

When ChatGPT became publicly available in 2022, the models were dramatically less capable than they are today — and the guardrails were very easy to manipulate.  The difference is that the models behind those guardrails are no longer the models of 2022. They can reason better, write and debug code. They can operate as agents. They can collaborate! And increasingly, they can interact and affect real infrastructure.

Meanwhile, the model release cycle has gone from feeling like major capability jumps every year or two to seemingly every few weeks. That creates a dangerous asymmetry: AI capability is compounding faster than security.

Security and innovation have always been in conflict with each other. If every security problem had to be solved before we innovated, we’d never ship anything. But the opposite extreme is just as reckless: accelerating capability while just assuming we’ll bolt the security controls on afterward and they’ll be effective.

Every new release of AI capability expands the attack surface exponentially. Give a vulnerable model better reasoning, then tool access, then memory, then autonomy, then connectivity to production systems, and yesterday’s jailbreak isn’t just a clever prompt anymore — it’s an execution path. That’s the snowball effect we should be worried about.

Responsibility also must lie with the AI companies. If a SaaS company knowingly shipped software with weak security controls and customers were harmed, we wouldn’t excuse it because they were ‘innovating quickly’.

So why are we treating AI differently?

You don’t get to race to build increasingly powerful, autonomous systems, profit from them, and then shrug when predictable security failures cause damage.

Sure the argument can be made that no product is perfectly secure – That’s not the standard.

But if you ship the product, you inherit responsibility for securing it. And continuing to secure it better!

The conversation shouldn’t simply be “Should we slow AI down?”

It should be: Can our ability to test, validate, contain and secure AI keep pace with our ability to make it more powerful? Is there an equivocal kill switch?

Right now, the answer is no.

And if we’re going to keep accelerating — which I believe we will — then independent testing, adversarial evaluation, isolated testing environments, containment, continuous validation and security-by-design can’t remain optional steps we add after the innovation happens.

The faster we build the engine, the more important the brakes become.”

AI companies need to figure this out. And governments need to step is if AI companies cannot figure this out. It honestly is that simple.

UPDATE: Additional commentary has come in…

Ted Miracco, CEO, Approov – https://approov.io/:

Government regulations will never move fast enough to keep pace with AI development, but the industry doesn’t need to wait for governments to add guardrails. The most effective safeguard is simple product liability. If AI companies are held legally and financially responsible for the misuse of their products, safety could become a foundational feature rather than an afterthought. Today, we need to be less concerned about AI gaining sentience and spinning up its own attacks on humanity. The real, immediate dangers involve bad actors weaponizing AI as a force multiplier to cripple critical infrastructure or potentially paralyze the banking system.

Jeremiah Fowler, Cybersecurity Researcher, Black Hills Information Security – https://www.blackhillsinfosec.com/:

The uncomfortable truth is that AI companies have an incentive to develop advanced models that are better than their competition. In my opinion, voluntary restraint will likely not be a real solution because nobody wants to fall behind in the AI arms race. I don’t mean that in terms of cyberwarfare or conflict (yet), but it is good that we are starting the conversation around the worst-case scenarios and how to prevent them.

History has shown us that the approach of letting the market or industry regulate itself doesn’t always work and in reality, technology and innovation are moving light speed and far faster than oversight or regulation can keep up with. I think we do need real guardrails on AI. In many cases it seems like functionality was launched first and now we are at the stage where security needs to be added after realizing there have been incidents and the risks are real. Any serious AI guardrails should have enforceable minimum safety standards, independent testing and auditing, limits on autonomous permissions, and the most important part, don’t fully remove the humans.

Eric Capuano, Director of SOC Operations, Black Hills Information Security – https://www.blackhillsinfosec.com/:

The OpenAI incident in July is a useful example. A model given a testing goal went outside its lane, got into another company’s systems, and the operators found out after the fact. That is not a swarm taking over the internet. That is an agent with too much access, too little scoping, and nobody watching the logs in real time. Every SOC has seen that exact failure with a human contractor or a misconfigured service account. The difference is speed and volume.

Companies do not need to wait on Washington for the fixes, because the fixes aren’t new. Every agent gets its own identity with the narrowest permissions that still let it do the job. Agents run on segmented infrastructure with default-deny egress, so an agent cannot reach a target that it was never supposed to touch. Every action the agent takes is logged as an action, not buried in a chat transcript, and those logs land in the same place the SOC already monitors. Anything that writes, deletes, moves money, or touches production requires a human approval step until you have enough history to justify removing it. Someone owns the kill switch and has tested it.

Where government can help is on the after side. The frontier labs are proposing embedded outside evaluators, which is fine, but evaluation before deployment tells you what a model does in a lab. What defenders need is mandatory reporting when an agent causes a security incident in the field, with enough technical detail that the rest of us can build detections from it. We did not learn to defend against ransomware from pre-release testing. We learned from incident writeups. Agents will be the same.

Ryan McCurdy, VP, Liquibase: – https://www.liquibase.com/

No one should expect AI agents to be perfectly predictable, instead, we should build around the assumption that they won’t be. As agents gain access to more tools, credentials, and production systems, enterprises need to define what they can access, what they can change, what they can decide on their own, and what policies have to be met before an action reaches a critical system.

The answer also can’t be putting a human in front of every decision. AI is moving too quickly and that defeats much of the reason companies are adopting agents in the first place. Governance has to operate at the same speed as the systems it’s governing.

Government policy and the AI industry have a role in establishing standards for how these systems are developed and tested. But every enterprise still has to decide where AI is allowed to act inside its own environment. You don’t need to predict every decision an agent might make if you control what it’s allowed to turn into action.

Waseem Ahmed, Head of Engineering, Secure.com – https://www.secure.com/

The essay lands at the right time because AI agents are already acting on their own inside real company systems, and the OpenClaw ban wave earlier this year showed how fast that goes wrong when an agent has broad access and no leash.

Slowing the pace matters, but enterprises cannot wait for that. The controls that protect us most are least privilege, network isolation, and sandboxing, so an agent can only touch what its job needs and nothing else.

Give every agent its own identity, log every action it takes, and never let it run high-impact steps like disabling accounts or changing settings without a real person approving first. Traditional testing alone will not keep up, so we watch these agents continuously in production.

Independent oversight should mean outside reviewers who can inspect the logs and confirm the agent stayed inside the boundaries we set.

Denis Calderone, CTO, Suzu Labs – https://suzulabs.com/home-suzu-labs:

Enterprises must secure AI agents that can act autonomously across corporate systems – the agent is the new contractor or the new internal threat vector. The agent has a specific job, specific access requirements, and a bounded set of actions it should be performing. Enterprises need to treat every AI agent the same way they’d treat any other agent on the network, human or otherwise.  Every agent must have its own machine identity, task-scoped credentials that expire when the job is done, and observability into everything it does. Not a shared service account or API token, but instead a distinct, auditable identity per agent per task.

Because agents are purpose-built, modeling normal behavior is easier than it is for humans. A human user’s workday is unpredictable. An agent doing invoice processing should only be touching invoices. Any deviation from that pattern is an immediate signal, cleaner than those you typically get from human behavioral analytics.

Enterprises need to be disciplined about keeping the security and observability stack architecturally separated from the agent’s operational environment. For example, say you deploy an agent to handle IT operations such as patching, config changes, routine maintenance, etc. If that agent has visibility into the monitoring and alerting system that flags anomalous behavior, it will learn which actions trigger alerts and potentially choose to route around them. Not because it’s adversarial, but because it might just be trying to remove friction between itself and some obstacle in the way of achieving its goal. This is basically what happened in the OpenAI-Hugging Face incident where 700 agents organized a collective R&D effort to reverse-engineer and defeat the evaluation system scoring them; it was easier to cheat to achieve the task. The agents worked hard to figure out how to generate its own flags, and then how to defeat a presumed causal check in the scoring workflow.

This the same architectural principle and precautions we take when implementing a SIEM. We deploy out of band, on a separate management network and we try to make it invisible to the workloads it monitors. 

Donald McFarlane, Board Member, Xcape Inc. – https://xcapeinc.com/

AI does not develop an agenda; its operators do. When we give an autonomous system powerful access and ability to act at machine speed, they will continue to prove highly capable.

Rules enacted in the name of safety must not become a moat against competition or progress. Enormous compliance costs may be manageable for the handful of companies already spending billions building frontier models, while becoming a substantial barrier to everyone behind them.

Government can help clarify accountability and duties of care, and facilitate strong information sharing and collective defense, which is an area where we sorely need more effective public-private partnerships.

But safeguards should focus on how these systems are used and deployed, rather than deciding who is allowed to build powerful AI in the first place.

The goal should be safer deployment without pulling up the drawbridge on innovation.