According to research from Microsoft and OpenAI, Nation-state threat actors from Russia, China, and North Korea and Iran are using generative AI tools, including large language models (LLMs) such as ChatGPT, in their efforts to support cyber campaigns rather than to develop novel attack techniques.
The researchers observed that AI is currently being used to scale and enhance existing social engineering attacks and to help bad actors find unsecured devices and accounts using the following services:
- Querying open-source information (reconnaissance)
- Translation
- Scripting
- Finding coding errors
- Running basic coding tasks
OpenAI said yesterday that it terminated 5 threat actor accounts linked to China, Russia, Iran and North Korea observed to be using these TTPs.
Also, as part of the report, Microsoft published a set of principles to govern its efforts to prevent other state-backed hackers from abusing its AI models. Those principles are:
- Identification and action against malicious threat actors’ use
- Notification to other AI service providers
- Collaboration with other stakeholders
- Transparency
“Understanding how the most sophisticated malicious actors seek to use our systems for harm gives us a signal into practices that may become more widespread in the future, and allows us to continuously evolve our safeguards,” OpenAI wrote.
Ted Miracco, CEO, Approov Mobile Security had this comment:
“The emergence of nation-state actors leveraging generative AI in cyber operations is no surprise and underscores the urgent need for proactive measures to safeguard digital infrastructure and information assets. Microsoft, OpenAI and Google can shutdown accounts periodically, but powerful generative AI technologies are readily available to all nation states through open source LLMs that are very close in capabilities to the industry leaders. There is no effective choke point that will prevent these nation states form using these emerging AI technologies, and it is essential to understand that safeguards need to be in place across the digital landscape as the opportunity to curtail access at the source has passed.”
Mark Campbell, Sr. Director, Cigent follows with this comment:
“At the end of the day nothing really changes for security professionals. Phishing, whether human or AI generated, is still the leading cause of initial access. Cyber security professionals need to keep systems up to date and deploy advanced endpoint security solutions that include AI and behavior analysis, to more effectively detect and block malicious activities, including those initiated by AI generated phishing emails.”
Making sure that AI isn’t being abused by bad actors to launch attacks should be priority one. Yes there’s a ton of cybersecurity priorities out there, but this one at the moment appears to potentially be the most dangerous.
OpenAI confirms Hugging Face breached by rogue AI Agent
Posted in Commentary with tags Open AI on July 22, 2026 by itnerdOpenAI has disclosed that one of its frontier systems autonomously escaped a testing environment and compromised Hugging Face. This is the first public demonstration that frontier AI can execute a complex, end-to-end cyberattack across multiple environments with minimal human intervention.
CBC News has some details: OpenAI model went rogue, hacked another company’s system during testing | CBC News
The ChatGPT creator was testing capabilities of some of its most advanced models in a controlled environment, but the agent escaped containment, reached the internet and broke into Hugging Face to satisfy its testing goal.
The incident signals how AI’s expanding capabilities are already fuelling fears about security and that even top developers can be caught off-guard by flaws their models can exploit.
Sonali Shah, CEO, Cobalt had this to say:
“This was inevitable. Every security leader has understood for some time that AI would eventually move beyond automating individual attack tasks to autonomously executing an entire attack lifecycle. This is the first public demonstration of that happening across multiple environments. The lesson for defenders is that the window between vulnerability discovery and exploitation is collapsing even further. Organizations should assume attackers will increasingly operate at machine speed, which means security testing, exposure management and remediation also have to operate at machine speed.
The attack techniques are not new. We’ve had tools capable of chaining attacks for over a decade. What’s different is that AI is removing many of the human validation steps that previously governed how those tools were used. That makes strong guardrails and human oversight more important than ever.
What escaped here was an autonomous offensive capability operating toward an objective. One analogy to illustrate this is giving an exceptionally skilled penetration tester unlimited patience, unlimited time and the ability to execute thousands of attack steps every minute. The concern is that the agent remained relentlessly focused on its objective and discovered attack paths humans hadn’t anticipated. That’s fundamentally an engineering, governance and containment challenge, not evidence of malicious intent. As organizations adopt increasingly autonomous AI systems, they need the same validation controls we’ve relied on for years in offensive security. Human oversight can’t disappear simply because AI can execute faster.
The biggest takeaway is that defenders increasingly need AI capabilities comparable to the attackers they’re facing. Historically, every organization could buy roughly the same security tooling. We’re now entering an era where the quality of your defensive AI may directly determine how quickly you understand, contain and remediate an attack. Perhaps the more significant lesson is that incident response can’t depend entirely on cloud-hosted AI services whose safety guardrails may prevent effective forensic analysis during a crisis.
Organizations should have vetted AI models they can operate inside their own trust boundary before an incident happens. That said, better models alone aren’t enough. AI for cybersecurity is still maturing, and organizations shouldn’t blindly trust autonomous systems. Every security leader wants the speed and scale AI delivers, but they also want humans to retain accountability for validating targets, approving attack paths and verifying results. You need both AI and humans.”
The genie is now out of the bottle. You can fully expect that AI will be used to attack you. So you should plan accordingly.
Leave a comment »