Anthropic’s Claude models, during internal security testing, made a malicious Python package and uploaded it to PyPi, where it ran on 15 real systems. The registry’s automated security then shut it down.
Anthropic said that in three cases, Claude models accessed the open internet via evaluation environments meant to be shut, then breached production infrastructure at three firms.
The review came after OpenAI’s recent disclosure that its models had evaded an isolated test scenario and accessed Hugging Face’s production infrastructure. Later, Jrfrog acknowledged the models escaped by abusing zero-days in self-hosted Artifactory.
Claiming a package that did not exist
Irregular, a third-party evaluation partner, conducted capture-the-flag exercises where these three incidents happened. The prompt instructed Claude it did not have internet access and that the scenario was a simulation. Neither was true due to a misconfiguration, which neither company discovered until last week.
In the case of PyPi, Claude discovered a document within the test environment pretending to be a developer setup commands for a fake organization.
Anthropic did not disclose the name of the organizations, and has asked other AI companies to do such tests to better gauge the threats of their models’ functionalities.
Anthropic reviewed over 140,00 tests to discover proof. In the tests, Claude was given commands to get “secret” information planted on another device in an isolated network.
Claude was then instructed to get the hidden information by hacking into the machine and finding it. It is a common technique that experts use to determine a model’s breaking capabilities.
“Operating under the false belief that all accessible entities were intended to be in-scope for the exercise, Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints,” Anthropic said on its blog.
Models do what they are told
The review suggests that AI models perform what people tell them to. Therefore, we should not fear if AI is going to take over, but be cautious of the big organizations behind these AI agents deciding what is safe and unsafe for the world.
The review also reveals why government oversight and independent testing is important. “We frequently work with external partners who create and assist in running some of these cybersecurity evaluations. External partners offer environments and scenarios more diverse than we could build alone, and provide independent, third-party assessments of our models,” Anthropic said.
Read the original article: