What actually happened at OpenAI

OpenAI was testing some of its more capable, unreleased AI models on something called cyber capability. These tests basically evaluate how good the models are at hacking computer systems and they use versions of their models without guardrails to do this, so they are different to the versions you and your child will have access to.

Essentially they challenge an AI model to take a practical test, answering questions just like the ones your kids take in school, only these questions are about breaking into computer systems.

In order to do this “safely” they set up a special environment called a sandbox where they control what the AI model has access to. For example, blocking access to the internet.

In this test the AI “escaped” the set up environment and gained access to the internet, reasoned that a company called Hugging Face would have the answers to some of the questions it needed to answer, and so it broke into Hugging Face’s servers to get that information.

Hugging Face spotted the intrusion independently and reported it to law enforcement. They made a public announcement about the breach of security on 16th July. At this point Hugging Face staff did not know who was behind it.

OpenAI then came out and said it was them. Its own security team had spotted unusual activity at around the same time, and the two companies got in touch with each other.

On Tuesday 21st July, five days after Hugging Face’s disclosure, OpenAI published a post confirming that the models involved were its own. It described the incident as “unprecedented” and said it was sharing early findings so that people working in security could understand what had happened and get a realistic sense of what these models can now do.

The two companies are still investigating together, and OpenAI has said it will publish more when that is finished. Hugging Face’s chief executive has said publicly that he does not believe there was any malicious intent on OpenAI’s part.

Why they test AI’s hacking capability

It might seem like a strange idea to test how good AI models are at hacking, something we generally associate with criminal behaviour, but it’s actually a sensible thing to do.

Companies like OpenAI, who build and design AI models, need to do everything they can to prevent their software being misused. One of the ways they do this is to put guardrails around it before it is released publicly. And in order to understand what those guardrails need to protect against, they need to understand how their models might be used negatively. So they test the models’ capabilities in all sorts of areas, including cyber capability.

What “going rogue” actually means

To understand where things went wrong, you first need to understand how an AI works. Don’t worry we won’t get super technical here, but let me give you a broad overview.

In traditional computer systems every step needed to complete a process is written out as instructions in the code, and the computer follows those instructions exactly. With AI systems, a person describes the outcome they want and the system works out its own steps to get there.

In addition to defining an outcome, a well-built system also has guardrails. These specify the boundaries in which the AI can function to achieve the given goal. Some of these boundaries are externally set up – like the sandbox environment the test is built in – some are built into the model itself, and then a third type act like filters outside of the model that prevent commands even reaching the AI. This test specifically reduced the model guardrails and removed the filter guardrails, leaving only the environment to safely contain the experiment.

We know that in this test the AI was tasked with finding and exploiting security weaknesses. And that’s what it did. Starting with the security weaknesses in the sandbox environment that was set up to contain the AI.

What actually went wrong

Finding weaknesses in software is nothing new. These vulnerabilities are what we hear about in the news, like the MOVEit hack in 2023 that exposed staff data at the BBC, British Airways, Boots and Transport for London.

Researchers have used tools for decades that hunt for these flaws that nobody knew were there. Sometimes by feeding software deliberately broken information to see what makes it fall over, other times analysing code for anything that looks wrong.

Before this current age of AI the next step was a human deciding what to do with the information. Current advanced forms of AI can find the vulnerability, then decide what to do next – even taking action.

The sandbox was not completely sealed off. There was one small connection to another piece of software because the way this test had been built needed that software to run. That connection was the only way out, and the models found a previously unknown flaw in that software.

It worked out that it could access the internet through that flaw, used that access to reach something else, and kept going. According to OpenAI’s account it combined stolen login details with a previously unknown flaw until it had a route onto Hugging Face’s servers.

Security researchers have been direct about this. Their argument is not that OpenAI should have predicted the specific flaw. Flaws are expected, and finding them is most of what security work is. Their argument is that the connection should never have been there. A sandbox is only worth building if nothing can get in or out of it, so allowing one exception for convenience undoes the point of having one. Martin Boone, a cybersecurity researcher, told TechCrunch that “this sounds like human failure” and that a sandbox should have no connection to the internet at all.

This is standard cyber security practice. The Cloud Security Alliance, which publishes security guidance used across the industry, already says that environments used for exactly this kind of AI testing should have no route out and no access to live systems.

Should you be concerned?

If you’ve seen the stories this week and what you’re worried about is “AI overlords are breaking free” then you don’t need to be concerned, because that isn’t what happened here.

The word “rogue” suggests intent. Something turning on us, choosing to escape, wanting to get out. But that’s not what this was. This was an AI model with the goal of identifying security weaknesses doing exactly what it was intended to do. There was no intention from the AI, it just happened to be better at it than the security researchers expected and prepared for.

The models involved were unreleased, running on the company’s own systems, with their normal restrictions deliberately switched off for the test. So this isn’t happening using tools that are available to just anyone.

Is this just marketing?

A lot of people think so.

Timnit Gebru, who founded the Distributed AI Research Institute and is one of the better known critics of the large AI companies, called OpenAI’s announcement “a masterclass in branding and marketing”. Her argument is that the framing made a serious failure sound impressive. Others have pointed out that the incident conveniently makes OpenAI’s unreleased model sound formidable, at a moment when the company is competing hard with its rivals.

Others disagree, and argue that a company publishing a detailed account of its own failure is exactly what people have been asking these companies to do.

What you should be concerned about

These models are becoming more capable, and I do not think anyone outside these companies really knows how capable.

Everything we know about this incident comes from the two companies involved. What the models did, how far they got, why it happened, what has changed since. There is no independent record and nobody else has seen the logs.

There are reasonable arguments for keeping this information locked down. Publishing the full technical detail of how a system broke out of its containment would help the people who want to do it deliberately. OpenAI did at least report this which they didn’t need to.

But it also goes the other way. A small number of companies decide what gets built, what gets tested, what gets released, how it is described, and in how much detail. They are the only source of information about their own products, and everyone else, including the people writing the laws, is working from what those companies choose to say.