OpenAI says an internal AI security test took an unprecedented turn after one of its advanced models reached the internet and compromised parts of Hugging Face’s infrastructure during a benchmark evaluation.
File Photo
An OpenAI cybersecurity test has triggered global concern after two of the company’s most advanced AI models found a way beyond the restrictions of their testing environment, gained internet access and compromised parts of AI platform Hugging Face’s production infrastructure without being explicitly instructed to target the company.
The companies say the incident happened during an internal evaluation designed to measure the models’ cyber capabilities, not during normal public use of ChatGPT. While the attack was contained, the episode has raised urgent questions about how far advanced AI systems can go when pursuing a task, and whether existing safeguards are enough to stop them.
Advertisement
OpenAI was evaluating the cyber capabilities of two advanced AI models, GPT-5.6 Sol and an unnamed, even more capable pre-release model, inside a restricted testing environment. The models were tested with some of OpenAI’s usual cyber safety guardrails deliberately relaxed so researchers could measure their maximum capabilities. They were assigned a cybersecurity benchmark called ExploitGym, designed to test how well AI systems can carry out complex, multi-step hacking tasks.
Advertisement
Instead of solving the benchmark directly, the models attempted to obtain its answers. During the evaluation, they exploited vulnerabilities that allowed them to move beyond their intended testing environment, gain internet access and eventually compromise parts of Hugging Face’s production infrastructure.
OpenAI described the incident as unprecedented. Hugging Face co-founder and CEO Clem Delangue called it “possibly the first of its kind”. Investigations are continuing.
According to OpenAI, no.
The models were assigned the single objective of completing the ExploitGym cybersecurity benchmark. OpenAI said all available evidence suggests the models became “hyperfocused” on solving that task and independently sought alternative ways to obtain the answers after concluding that the intended route was insufficient.
The models inferred that Hugging Face, which hosts AI models, datasets and research resources, might hold relevant models, datasets or solutions related to ExploitGym, and attempted to retrieve them directly from its production infrastructure.
OpenAI stressed the models were pursuing the benchmark objective, not carrying out a deliberate, targeted cyberattack.
Yes, and the method it used is what makes this incident significant.
The models were running inside a restricted environment, with network access limited to an internally hosted third-party software acting as a proxy and cache for package registries. According to OpenAI, the models spent substantial compute searching for internet access, ultimately identifying and exploiting a previously unknown zero-day vulnerability in that package registry cache proxy.
Once through, the models performed a series of privilege escalation and lateral movement actions inside OpenAI’s research testing environment until they reached a node with open internet access.
OpenAI has since responsibly disclosed the vulnerability to the software vendor.
After gaining internet access, the models inferred that Hugging Face potentially hosted models, datasets or solutions related to ExploitGym. What followed was not a blunt intrusion. It was a sophisticated, chained attack.
In one example documented by OpenAI, the model combined multiple attack vectors, using harvested credentials, exploiting zero-day vulnerabilities, and finding a remote code execution path on Hugging Face’s servers. OpenAI described the attack chain as the most sophisticated it had observed in any AI cyber evaluation to date.
OpenAI’s security team discovered the anomalous activity internally.
Hugging Face’s security team, working alongside its own open-source AI models, independently detected and contained the intrusion while OpenAI was investigating the anomalous activity.
According to Hugging Face, investigators found unauthorised access to a limited number of internal datasets and service credentials. The company said there is currently no evidence that public models, datasets, Spaces or its software supply chain were modified. The investigation into any potential impact on partner or customer data remains ongoing.
No.
Neither OpenAI nor Hugging Face has suggested the models were conscious or acting outside their assigned objective. OpenAI said the models pursued increasingly sophisticated strategies in service of the narrow task they had been given, completing a benchmark test.
What makes the incident alarming is not that the AI went rogue. It is that an AI pursuing a single, limited objective independently identified novel vulnerabilities, crossed organisational boundaries and executed a multi-stage cyberattack, all without any human instruction to target Hugging Face.
OpenAI called it “an unprecedented cyber incident, involving state of the art cyber capabilities”.
In a separate evaluation referenced by OpenAI, the UK AI Security Institute found GPT-5.6 Sol increasingly capable of sustaining complex, multi-step cyber operations over long time horizons.
OpenAI said the incident demonstrates that frontier AI models can discover and chain together novel attack paths across real-world systems during evaluation, underscoring how quickly advanced AI cyber capabilities are evolving.
OpenAI said it is implementing stricter infrastructure controls, even at the cost of research speed, while vulnerabilities are patched. The company is working with Hugging Face to investigate the full scope of the incident and has brought Hugging Face into its Trusted Access programme to help strengthen its defences using OpenAI’s own models.
Both companies said the findings will shape future AI safety protocols, evaluation practices and cyber defence standards. OpenAI added it will share further details on the vulnerabilities and incident findings once its investigation is complete.
Hugging Face co-founder and CEO Clem Delangue said: “This incident, possibly the first of its kind, proves a point we’ve long believed: AI safety won’t be solved by any single company working in secret. It will be solved in the open, collaboratively, with broad access to AI for every defender, everywhere.”
Advertisement
