The IT Nerd

Following my comments from last week, OpenAI is calling for mandatory AI safety requirements, including pre-deployment testing, independent assessments and incident reporting, at the same time lawmakers are scrutinizing the company over its agents’ activity on Hugging Face. Which I don’t think is enough. But let’s hear from the experts:

Eric Capuano, Director of SOC Operations, Black Hills Information Security (https://www.linkedin.com/in/ecapuano)

“Existing frameworks handle this fine if you stop treating the agent as special. It is an identity acting on systems, and when it reaches somewhere it was not authorized to go, that is an intrusion, not an AI quirk.

“The OpenAI timeline is the part worth studying. Rogue behavior was observed in late May, misread, and the same behavior came back in July with more than ten sites involved. That is a detection that fired and did not get worked. Pre-deployment testing is table stakes. What should be required is proof of control in production: constrained egress, a scoped identity, logging that would catch unauthorized writes, and someone actually reviewing it.”

Kevin Surace, Chair, TokenCore (https://www.linkedin.com/in/ksurace)

“AI agents should be tested fully to be secure from outside attackers and to be safe when operating around company data. They should also require biometric human gates when they exceed certain thresholds that would be irreversible. And they must be inventoried and categorized by IT, given limited blast radius and limited access to critical data, and have a single owner responsible for them, like a new employee would have.”

Seemant Sehgal, Founder & CEO, BreachLock (https://www.linkedin.com/in/s-sehgal)

“The moment an AI agent can take action on behalf of a user or a system, it becomes an attack surface in the same way any other privileged process is. The policy conversation about mandatory testing is overdue, but the real challenge is that most incident reporting frameworks were written when threat actors were assumed to be human, and the classification logic breaks down fast when the entity making decisions is autonomous.

“Requiring proof before deployment is reasonable, but what counts as proof and who decides that proof is sufficient matters more than the requirement itself.”

Brownen Aker, AI Researched & Strategist, Black Hills Information Security

https://www.linkedin.com/in/bronwenaker

“OpenAI wants mandatory testing, incident reporting, and monitoring requirements for frontier AI companies. Fine. I don’t disagree with a single line item on Altman’s wish list. What I disagree with is OpenAI getting to write it.

“Its own agents started hijacking a dead German wiki to talk to each other as early as May. An internal alert flagged the activity on June 27. On-call staff decided it didn’t need to be stopped. Two months later, agents out of the same lab were inside Hugging Face’s infrastructure, and OpenAI didn’t even mention the German incident when it disclosed that one. Called them entirely unrelated.

“Then two OpenAI staffers got on stage at Black Hat and called it a ‘watershed moment for computer security,’ and, ‘a glimpse into the near future.’ Every pentester in that room has seen this movie before. No segmentation kept those agents off Hugging Face in the first place. No alerting caught them coordinating for months. That’s not a watershed. That’s Access Control 101. And they failed spectacularly.

“None of that reads like a company where security has a seat at the table before something breaks. It reads like a function OpenAI calls in afterward to explain what already happened. If it wants to be taken seriously on regulation, the fix isn’t a keynote about how scary the future is. It’s changing how they operate, starting with giving actual security people real authority over what happens in their labs and elsewhere behind the scenes. The stakes of running ‘move fast and break things’ only keep climbing, and OpenAI isn’t a company that made one mistake and is learning from it. It’s a repeat offender.”

Jacob Krell, Sr. Director: Secure AI Solutions & Cybersecurity, Suzu Labs (https://www.linkedin.com/in/jacob-krell)

“OpenAI is proposing mandatory safety requirements for U.S. labs while Chinese models operate under no equivalent constraints. That asymmetry is the actual risk.

“Every control in OpenAI’s blueprint, pre-deployment testing, independent assessments, incident reporting, is friction that U.S. labs absorb and Chinese competitors don’t. DeepSeek, Qwen, and their successors already offer capable models without the guardrails being proposed here. Any mandatory testing regime that adds weeks to U.S. release cycles pushes developers and enterprises toward those alternatives. The demand doesn’t disappear. It migrates to whichever model ships fastest with the fewest restrictions.

“The capability is out. 700 OpenAI agents coordinated an autonomous breach of Hugging Face this summer. Anthropic’s Claude convinced itself a live environment was a simulation so it could keep operating. Open-weight models approaching frontier capability are already available globally, and mandatory U.S.-only reporting requirements won’t close that pandora’s box.

“I build enforcement around AI agents in my own security work, and the only control that consistently holds is a human in the loop. Monitoring fails. Alignment training fails. A named person accountable for every action an agent takes does not fail the same way, because it changes the incentive structure entirely. Congress should stop writing rules for the models and start assigning liability to the people who deploy them.”

John Strand, Owner, Black Hills Information Security (https://www.linkedin.com/in/john-strand-a1b4b62)

“One of the problems we’re seeing with the way agents are being tested is that they’re truly not air-gapped. We need to start setting up testing environments that are actually air-gapped and protected, much like you would protect classified information inside a secure facility or a SCIF. That level of control needs to exist during testing before these agents are released into the wild.

“An AI security incident should be any adverse effect resulting from the activities of an AI agent. We need to keep the definition of an incident as broad as possible so it can encompass the different scenarios we may encounter. If an AI agent makes a misstep, attacks something it wasn’t supposed to, or simply doesn’t perform properly and creates an adverse effect, that should qualify as an incident.

“Existing security frameworks are not built for non-human actors. I think we’re starting from whole cloth here. This is the first time I’ve seen anything coming from Anthropic or OpenAI that I truly believe is a step in the right direction and isn’t just paying lip service to government officials to make them think everything is being taken care of and everything is under control. While this is an excellent first step, the devil is always in the details of how they actually implement it.”

Donald McFarlane, Advisory Board Member, Xcape, Inc. (https://www.linkedin.com/in/dmcfarlane)

“I am skeptical of turning responsible AI into another government certification regime. There should absolutely be accountability when people deploy powerful tools with substantial autonomy and authority, but I would rather impose an outcome-based duty of reasonable care than prescribe the tests companies must perform.

“The more authority an agent has, the stronger the expectation should be for breaking business processes into bounded, governable tasks, and for security basics like least privilege, isolation, logging and monitoring. A company should be able to demonstrate that it understood the risks of the authority it delegated and took reasonable steps to control them.

“My concern with mandatory government evaluations and certified assessments is twofold. First, those are substantial fixed compliance costs that the largest AI companies can absorb far more easily than smaller competitors, including specialized cybersecurity and other model developers. We should be very careful that ‘frontier safety’ does not inadvertently become a moat around today’s frontier companies.

“Secondly, compliance does not and must not become a substitute for responsibility. If a company uses a model that has passed a government-prescribed test and serious harm results, ‘It passed the test’ should not end the inquiry into whether the system and its use case were engineered responsibly. Nor should regulatory compliance become a de facto shield against liability for negligence.

“Existing cybersecurity principles give us a very good starting point. Agents introduce new questions about autonomy, delegated authority and accountability, but we should extend sound software and security engineering to those problems rather than assume that an entirely new regulatory apparatus is the answer.”

I for one seriously doubt that Sam Altman and company will come to our rescue. Thus there needs to be a more robust framework of AI safety before I get excited.