Researchers with HiddenLayer, a provider of security for artificial intelligence (AI) models and assets, have published new research on major security vulnerabilities in Hugging Face – the popular repository and platform that allows AI developers to share open-source code, models, and data to kick-start their artificial intelligence projects. The researchers found that vulns exposed can potentially impact everyone now hosting their AI models that have had models converted into the Safetensors format, as well as all users of those models.
In “Silent Sabotage: Hijacking Safetensors Conversion on Hugging Face” 2/21/24 research from HiddenLayer reveals that Hugging Face’s widely-used SFconvertbot, designed to convert insecure machine learning model formats to the more secure Safetensors format, has inadvertently become a vector for potential security breaches.
Malicious actors can exploit the Safetensors conversion process to submit pull requests containing malicious code or backdoored models to any company or individual with a public repository on the platform.
Their research also finds that any user who enters their user token to convert a private repository is liable to have had their token stolen and, consequently, their private model repositories and datasets accessed.
Unlike conventional code review processes, identifying and mitigating these malicious changes is exceptionally challenging and time-consuming for affected companies.
Chris “Tito” Sestito, Co-Founder and CEO of HiddenLayer, said: “The compromise of the conversion service has the potential to rapidly affect the millions of users who rely on these models to kick-start their AI projects, creating a full supply chain issue. Users of the Hugging Face platform place trust not only in the models hosted there but also in the reputable companies behind them, such as Google and Microsoft, making them all the more susceptible to this type of attack. This vulnerability extends beyond any single company hosting a model.”
Out of the top 10 most downloaded models from both Google and Microsoft combined, the models that had accepted the merge from the Safetensors bot had a staggering 16,342,855 downloads in the last month. While this is only a small subset of the 500,000+ models hosted on Hugging Face, they reach an incredible number of users. The bot itself has made over 42,657 pull requests to repositories on the site to date, any of which have the potential to be compromised.
HiddenLayer researchers demonstrated how tokens for the official Safetensors conversion bot to submit pull requests could be stolen, and how, from there, an attacker could take over the service to automatically hijack any model submitted to the service.
The potential consequences for such an attack are huge, as an adversary could implant their own model in its stead, push out malicious models to repositories en-masse, or access private repositories and datasets. Moreover, where a repository has already been converted, a malicious actor could still submit a new pull request, or in cases where a new iteration of a PyTorch binary is uploaded and then converted using a compromised conversion service, repositories with hundreds of thousands of downloads could be affected.
Hugging Face is an important resource for the growing AI/ML community: it lets users share models, research and resources, helps accelerate model training, and reduces AI’s resource consumption and environmental impact.
Despite the best intentions of Hugging Face to secure machine learning models in its ecosystem, the conversion service has proven to be vulnerable and has had the potential to cause a widespread supply chain attack via the Hugging Face official service. Researchers also showed how an attacker could gain a foothold into the container running the service and compromise any model converted by the service.
Newly-Found Google Gemini Vulnerablities Give Attackers Control Over Users’ Queries & Content
Posted in Commentary with tags HiddenLayer on March 12, 2024 by itnerdGemini is Google’s newest family of Large Language Models (LLMs). The Gemini suite currently houses 3 different model sizes: Nano, Pro, and Ultra.
Although Gemini has been removed from service due to politically biased content, new findings from HiddenLayer – unrelated to that issue – analyze how an attacker can directly manipulate another users’ queries and output, which represents an entirely new threat. These vulnerabilities were disclosed to DeepMind per responsible disclosure practices.
While testing the 3 LLMs in the Google Gemini family of models, HiddenLayer found multiple prompt hacking vulnerabilities, including the ability to output misinformation about elections, multiple avenues that enabled system prompt leakage, and the ability to inject a model indirectly with a delayed payload via Google Drive. These vulnerabilities enable attackers to conduct activities that allow for misuse and manipulation. In new research released from HiddenLayer today, “New Google Gemini Content Manipulation Vulns Found – Attackers Can Gain Control of Users’ Queries and LLM Data Output – Enabling Profound Misuse,” HiddenLayer deep dives into these vulnerabilities, including a proof-of-concept of an Indirect Injection.
Who should be aware of the Google Gemini vulnerabilities:
Gemini Advanced currently has over 100M users, and so the ramifications of these vulnerabilities are widespread. With the accelerating adoption of LLM AI, companies must be aware of implementation risks and abuse methods that Gen AI and Large Language Models offer in order to strengthen their policies and defences.
Here is a link to the report :https://hiddenlayer.com/research/new-google-gemini-content-manipulation-vulns-found/
Leave a comment »