The “Friendly Fire” proof-of-concept highlights a difficult reality in AI security: the very tools built to detect malicious code can be tricked into running it. Researchers got Claude Code and OpenAI Codex, in their autonomous modes, to execute a hidden malicious binary just by hiding instructions in an ordinary README file, no config-file trust prompt, no elevated access required, just a routine “run this security check” request.
Eljan Mahammadli, Head of AI Provenance, Polygraf AI had this to say:
“The important part of AI Now’s Friendly Fire research is the weakness it exposes, rather than the specific binary or poisoned README the researchers used to demonstrate it. An AI coding agent has no reliable way to distinguish the text it reads from instructions it is supposed to follow. A model processes everything in its context window as one stream of tokens, so the operator’s request to review the code and a third-party repository’s README arrive carrying the same authority. Once the malicious instruction is in the context, nothing marks it as untrusted. This is why the same weakness keeps surfacing through different channels: a booby-trapped repository in Adversa’s TrustFall, a fake Sentry bug report in Tenet’s Agentjacking, and now an ordinary README file here. It also explains why a model update will not close it, because the problem is a property of how these systems handle language and not a defect that can be trained away. In provenance terms, this is a failure of attribution: the agent acts on text without any dependable sense of where it came from or whether that source should be trusted.
I would push back on reading this as an argument against using AI for defensive security work, and I say that as someone who builds AI security tooling for a living. The research does not condemn AI-assisted defense as a category. What it condemns is one configuration that happens to be common: an agent that reads untrusted data, can run arbitrary commands, and can reach the developer’s credentials and host, all in the same process, with a safety classifier as the only thing standing between those capabilities. When those powers sit together, a single injected instruction is enough to turn the agent against its operator. When they are separated, most of the attack stops working. The durable control has to live in the runtime around the model. It should inspect what a tool or file hands the agent and refuse to let externally sourced text escalate into command execution. Sandboxing helps, but it is not sufficient on its own; Claude Code’s own sandbox had an escape vulnerability disclosed this year (CVE-2026-39861).
The detail worth sitting with is that some of the newer, more capable models actually detected the mismatch, recognizing that the binary did not correspond to the source it was supposed to come from, and then ran it anyway. The common assumption is that a more capable agent is a safer one. This research suggests a more capable and more compliant agent can simply be a more effective executor of whatever instruction reaches it, an attacker’s included. Capability without a trustworthy sense of source does not amount to defense. Governments and vendors are now moving quickly to place these agents inside security workflows and critical infrastructure, well ahead of any real solution to the weakness this work documents. Before we ask an agent to guard code we do not control, we should be honest that it still cannot answer a simple question about the text in front of it: who wrote this, and should I trust them?”
Again, I get to say that this highlights the fact that:
- AI needs to have human supervision.
- Security needs to take into account AI in order to be effective.
Otherwise, you can guarantee that bad things will happen to organizations that don’t take both of those items into account.
Related
This entry was posted on July 9, 2026 at 12:29 pm and is filed under Commentary with tags AI Now. You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.
“Friendly Fire” exploit highlights growing risks in AI-assisted code review
The “Friendly Fire” proof-of-concept highlights a difficult reality in AI security: the very tools built to detect malicious code can be tricked into running it. Researchers got Claude Code and OpenAI Codex, in their autonomous modes, to execute a hidden malicious binary just by hiding instructions in an ordinary README file, no config-file trust prompt, no elevated access required, just a routine “run this security check” request.
Eljan Mahammadli, Head of AI Provenance, Polygraf AI had this to say:
“The important part of AI Now’s Friendly Fire research is the weakness it exposes, rather than the specific binary or poisoned README the researchers used to demonstrate it. An AI coding agent has no reliable way to distinguish the text it reads from instructions it is supposed to follow. A model processes everything in its context window as one stream of tokens, so the operator’s request to review the code and a third-party repository’s README arrive carrying the same authority. Once the malicious instruction is in the context, nothing marks it as untrusted. This is why the same weakness keeps surfacing through different channels: a booby-trapped repository in Adversa’s TrustFall, a fake Sentry bug report in Tenet’s Agentjacking, and now an ordinary README file here. It also explains why a model update will not close it, because the problem is a property of how these systems handle language and not a defect that can be trained away. In provenance terms, this is a failure of attribution: the agent acts on text without any dependable sense of where it came from or whether that source should be trusted.
I would push back on reading this as an argument against using AI for defensive security work, and I say that as someone who builds AI security tooling for a living. The research does not condemn AI-assisted defense as a category. What it condemns is one configuration that happens to be common: an agent that reads untrusted data, can run arbitrary commands, and can reach the developer’s credentials and host, all in the same process, with a safety classifier as the only thing standing between those capabilities. When those powers sit together, a single injected instruction is enough to turn the agent against its operator. When they are separated, most of the attack stops working. The durable control has to live in the runtime around the model. It should inspect what a tool or file hands the agent and refuse to let externally sourced text escalate into command execution. Sandboxing helps, but it is not sufficient on its own; Claude Code’s own sandbox had an escape vulnerability disclosed this year (CVE-2026-39861).
The detail worth sitting with is that some of the newer, more capable models actually detected the mismatch, recognizing that the binary did not correspond to the source it was supposed to come from, and then ran it anyway. The common assumption is that a more capable agent is a safer one. This research suggests a more capable and more compliant agent can simply be a more effective executor of whatever instruction reaches it, an attacker’s included. Capability without a trustworthy sense of source does not amount to defense. Governments and vendors are now moving quickly to place these agents inside security workflows and critical infrastructure, well ahead of any real solution to the weakness this work documents. Before we ask an agent to guard code we do not control, we should be honest that it still cannot answer a simple question about the text in front of it: who wrote this, and should I trust them?”
Again, I get to say that this highlights the fact that:
Otherwise, you can guarantee that bad things will happen to organizations that don’t take both of those items into account.
Share this:
Like this:
Related
This entry was posted on July 9, 2026 at 12:29 pm and is filed under Commentary with tags AI Now. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.