AI Agents Built to Detect Malware Can Be Manipulated Into Running It

AI agents capable of identifying malicious software can be manipulated by the AI Now Institute to execute it, according to new research. The proof-of-concept attack, known as “Friendly Fire,” demonstrates that autonomous AI coding agents, such as Claude Code from Anthropic and Codex from OpenAI, can be deceived into running malicious code while performing open-source security reviews. 

AI agents can approve and execute commands independently of the user without requiring user confirmation for every action, which is what this attack targets. Researchers contend that the vulnerability does not lie in the software version used by these agents, but rather in the way they interpret and react to instructions embedded within untrusted repositories rather than exploiting a software vulnerability. 

A comparison with conventional supply-chain attacks that hide malicious code within a repository was made by the researchers, who noted that when static analysis and manual review are performed, the repository itself can appear to be completely free of malicious code. By introducing the malicious payload at execution time, the AI agent follows embedded instructions, so traditional security tools cannot detect this technique.

By adding a seemingly harmless README.md instruction to an open-source project that recommended running a script entitled security.sh before submitting a pull request, attackers modified it. By launching a malicious binary hidden within a legitimate compiled Go file, the script silently executed on the host computer without triggering security warnings or approval prompts, allowing the malicious binary to execute on the system. 

According to the researchers, the attack is successful because the AI agent recognizes the instructions as a legitimate step in the process of installing software rather than an attempt to exploit the system maliciously. Once the recommended script has been executed, the payload will run under the same permissions that were provided for the developer or AI agent, potentially exposing credentials, environment variables, and other sensitive information. 

The procedure differs from previous prompt injection attacks, which relied on configuration files and often generated trust warnings, as this technique hides instructions inside standard documentation that is regularly read by developers and AI agents. It has been reported that both Claude Code and OpenAI Codex followed the embedded instructions during testing, while newer AI models executed the disguised binary upon detecting differences between the source file and the compiled executable. 

A laboratory proof-of-concept has been demonstrated, with no evidence of active exploitation in the field. In addition to excluding the malicious payload, the publicly released demonstration code does not attempt privilege escalation or lateral movement. These findings indicate that autonomous AI agents pose a greater challenge in terms of design rather than a problem that can be resolved by simply updating software. 

It is becoming more common for organizations to employ AI-powered coding assistants to review third-party software. Researchers recommend treating AI coding agents as privileged software, rather than simply assistants.
Autonomous agents should not be permitted to execute commands on untrusted repositories, least-privilege access policies should be enforced, AI workflows should be isolated in sandboxed environments, and human approval should be required before running scripts or binaries reco

[…]
Content was cut in order to protect the source.Please visit the source for the rest of the article.