Researchers with HiddenLayer, a provider of security for artificial intelligence (AI) models and assets, have published new research on major security vulnerabilities in Hugging Face – the popular repository and platform that allows AI developers to share open-source code, models, and data to kick-start their artificial intelligence projects. The researchers found that vulns exposed can potentially impact everyone now hosting their AI models that have had models converted into the Safetensors format, as well as all users of those models.
In “Silent Sabotage: Hijacking Safetensors Conversion on Hugging Face” 2/21/24 research from HiddenLayer reveals that Hugging Face’s widely-used SFconvertbot, designed to convert insecure machine learning model formats to the more secure Safetensors format, has inadvertently become a vector for potential security breaches.
Malicious actors can exploit the Safetensors conversion process to submit pull requests containing malicious code or backdoored models to any company or individual with a public repository on the platform.
Their research also finds that any user who enters their user token to convert a private repository is liable to have had their token stolen and, consequently, their private model repositories and datasets accessed.
Unlike conventional code review processes, identifying and mitigating these malicious changes is exceptionally challenging and time-consuming for affected companies.
Chris “Tito” Sestito, Co-Founder and CEO of HiddenLayer, said: “The compromise of the conversion service has the potential to rapidly affect the millions of users who rely on these models to kick-start their AI projects, creating a full supply chain issue. Users of the Hugging Face platform place trust not only in the models hosted there but also in the reputable companies behind them, such as Google and Microsoft, making them all the more susceptible to this type of attack. This vulnerability extends beyond any single company hosting a model.”
Out of the top 10 most downloaded models from both Google and Microsoft combined, the models that had accepted the merge from the Safetensors bot had a staggering 16,342,855 downloads in the last month. While this is only a small subset of the 500,000+ models hosted on Hugging Face, they reach an incredible number of users. The bot itself has made over 42,657 pull requests to repositories on the site to date, any of which have the potential to be compromised.
HiddenLayer researchers demonstrated how tokens for the official Safetensors conversion bot to submit pull requests could be stolen, and how, from there, an attacker could take over the service to automatically hijack any model submitted to the service.
The potential consequences for such an attack are huge, as an adversary could implant their own model in its stead, push out malicious models to repositories en-masse, or access private repositories and datasets. Moreover, where a repository has already been converted, a malicious actor could still submit a new pull request, or in cases where a new iteration of a PyTorch binary is uploaded and then converted using a compromised conversion service, repositories with hundreds of thousands of downloads could be affected.
Hugging Face is an important resource for the growing AI/ML community: it lets users share models, research and resources, helps accelerate model training, and reduces AI’s resource consumption and environmental impact.
Despite the best intentions of Hugging Face to secure machine learning models in its ecosystem, the conversion service has proven to be vulnerable and has had the potential to cause a widespread supply chain attack via the Hugging Face official service. Researchers also showed how an attacker could gain a foothold into the container running the service and compromise any model converted by the service.
Related
This entry was posted on February 23, 2024 at 9:20 am and is filed under Commentary with tags HiddenLayer. You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.
Hugging Face, the “GitHub for AI” presents major vulnerabilities
Researchers with HiddenLayer, a provider of security for artificial intelligence (AI) models and assets, have published new research on major security vulnerabilities in Hugging Face – the popular repository and platform that allows AI developers to share open-source code, models, and data to kick-start their artificial intelligence projects. The researchers found that vulns exposed can potentially impact everyone now hosting their AI models that have had models converted into the Safetensors format, as well as all users of those models.
In “Silent Sabotage: Hijacking Safetensors Conversion on Hugging Face” 2/21/24 research from HiddenLayer reveals that Hugging Face’s widely-used SFconvertbot, designed to convert insecure machine learning model formats to the more secure Safetensors format, has inadvertently become a vector for potential security breaches.
Malicious actors can exploit the Safetensors conversion process to submit pull requests containing malicious code or backdoored models to any company or individual with a public repository on the platform.
Their research also finds that any user who enters their user token to convert a private repository is liable to have had their token stolen and, consequently, their private model repositories and datasets accessed.
Unlike conventional code review processes, identifying and mitigating these malicious changes is exceptionally challenging and time-consuming for affected companies.
Chris “Tito” Sestito, Co-Founder and CEO of HiddenLayer, said: “The compromise of the conversion service has the potential to rapidly affect the millions of users who rely on these models to kick-start their AI projects, creating a full supply chain issue. Users of the Hugging Face platform place trust not only in the models hosted there but also in the reputable companies behind them, such as Google and Microsoft, making them all the more susceptible to this type of attack. This vulnerability extends beyond any single company hosting a model.”
Out of the top 10 most downloaded models from both Google and Microsoft combined, the models that had accepted the merge from the Safetensors bot had a staggering 16,342,855 downloads in the last month. While this is only a small subset of the 500,000+ models hosted on Hugging Face, they reach an incredible number of users. The bot itself has made over 42,657 pull requests to repositories on the site to date, any of which have the potential to be compromised.
HiddenLayer researchers demonstrated how tokens for the official Safetensors conversion bot to submit pull requests could be stolen, and how, from there, an attacker could take over the service to automatically hijack any model submitted to the service.
The potential consequences for such an attack are huge, as an adversary could implant their own model in its stead, push out malicious models to repositories en-masse, or access private repositories and datasets. Moreover, where a repository has already been converted, a malicious actor could still submit a new pull request, or in cases where a new iteration of a PyTorch binary is uploaded and then converted using a compromised conversion service, repositories with hundreds of thousands of downloads could be affected.
Hugging Face is an important resource for the growing AI/ML community: it lets users share models, research and resources, helps accelerate model training, and reduces AI’s resource consumption and environmental impact.
Despite the best intentions of Hugging Face to secure machine learning models in its ecosystem, the conversion service has proven to be vulnerable and has had the potential to cause a widespread supply chain attack via the Hugging Face official service. Researchers also showed how an attacker could gain a foothold into the container running the service and compromise any model converted by the service.
Share this:
Like this:
Related
This entry was posted on February 23, 2024 at 9:20 am and is filed under Commentary with tags HiddenLayer. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.