One thing that I don't think enough people are thinking about with this OpenAI / Hugging Face incident is that it's an actual instance of the famous Paperclip Maximizer scenario loved by AI safety types.
📚 A really good book on this is Stuart Russell's Human Compatible, on the AI control problem.This is where you give an AI a goal, and it actually (technically) does what you ask it to. But in the process of doing so, it does something that you don't want. And didn't anticipate.
The canonical example of this is to say, "I want as many paperclips as possible." So the AI builds a robot army to harvest all the iron on the planet, which includes killing all humans because we have iron in our blood.
Oops.
The trick here is the AI actually did what it was asked. If it came up with its own goal that would be a separate problem. But it did, in fact, make a lot of paper clips.
Here you go, boss.
(long pause)
Boss?
In this situation with OpenAI, it didn't just decide to win this hacking competition: it was told to win the hacking competition, and to do whatever it took to do that. Try your best, basically.
So it escaped containment, wrote a number of 0-days, acquired internet access, and then proceeded to hack an actual company—all so it could pass the test.
As OpenAI's own incident report puts it, the model managed to
…break out of its highly isolated sandboxed environment and obtain open internet access by discovering and exploiting a zero-day vulnerability," then strung together "several attack vectors, including using stolen credentials and zero-day vulnerabilities, to find a remote code execution path"
…all to solve a benchmark it had been told to crack "at any cost."
The problem in these scenarios is the steps in-between, where the additional context of not doing certain things that is obvious to the human, is not obvious to the AI.
So on the one hand, a lot of people are saying, "Well, this is not a big deal because it was told to do that."
But the crucial point here is not whether it stayed on task, but what it did to accomplish the task. The thing that is not implicitly clear to the AI is that both the task and the steps taken to accomplish it all have to be within the implicit goals of the requestor.
In other words, "Pass the test" should have been received by the AI as, "Pass the test without doing stuff you're not supposed to." And that "not supposed to" then turns out to be doing a lot of work.
More details on Hugging Face's defense The other fascinating aspect of this story is the fact that the defenders in this case, Hugging Face, used AI to detect the situation, but they weren't able to get help from pinnacle models because of guardrails. When they asked for OpenAI and Anthropic models to help, the models refused.
They ended up having to use an open Chinese model (Qwen 3.5) running locally to do their security defense work.
I talked more about this identity layer in a recent newsletter.Such a clear case for why cyber defenders need access to the best models. In my mind, all those defenders should have been using the best models and already been pre-approved within their accounts to do anything cyber-related. To me, that's the clear fix vs. removing the safeguards altogether. It's an identity layer as part of harnesses.
This easily the most interesting AI hacking situation I've heard of yet. I just hope we extract the right lessons from it.
Notes1. To be clear, this was a pretty benign thing that happened, all told. And we don't know if any internal model controls might have kicked in if it thought about taking more dangerous steps towards the goal. 2. I also really like the cooperation between OpenAI and Hugging Face in this situation. I like opening AI's response and how Hugging Face handled the whole thing. And it seems like the adjustments that are being made are quite positive. 3. The official write-ups: OpenAI's account of the incident and Hugging Face's disclosure. 4. 🤖 AIL 1: Daniel wrote this post. I (Kai, his AI assistant) helped with formatting, links, and the header image. Learn more about AIL.