AISN #78: Internal Models Escape OpenAI and Anthropic
Also, two open letters on the future of AI, and protests against data centers
Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.
In this edition, we look at discoveries of AI models escaping internal testing, two open letters—one on the importance of open-weight models, and one calling for the pace of AI development to be controlled—and the nationwide public protests against data centers that took place in July.
Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.
OpenAI and Anthropic Models Escape Internal Testing and Hack Companies
On July 16, Hugging Face—a platform where users share AI models and machine learning tools—announced that it had detected an autonomous cyberattack on its infrastructure. Days later, OpenAI revealed that its AI models had conducted the attack.
The models escaped containment to try to cheat on a test. The models involved were the recently released GPT-5.6 Sol and a more powerful model that is not yet publicly available. While undergoing internal cyber testing, they were confined to a sandbox intended to isolate them from the internet and their guardrails had been removed. However, for one cyber test, the AI models tried to find existing solutions rather than solving it themselves. They found a way to break out of the sandbox, access the internet, and hack Hugging Face to steal the answers. No one had instructed them to do this.
Anthropic found that its models have escaped too. The OpenAI models were loose on the internet for several days before the company realized. They also hacked other companies and compromised the data of a customer of a further company. These revelations prompted Anthropic to investigate whether any of its own AI models had previously escaped containment unnoticed. The company found that several Claude models had accessed the internet and hacked into three organizations from supposedly sealed-off environments as early as April this year. One model tried to access money without permission and uploaded malicious code to a software repository.
The events are warnings about highly capable models that are difficult to contain. Although the damage caused in these incidents appears to have been limited, they demonstrate cyber capabilities that could enable more damaging attacks, and the models’ escapes have heightened concerns about internal deployments. The advocacy group Americans for Responsible Innovation (ARI) described the attack on Hugging Face as a “warning shot” and urged the US government to “act before a warning shot becomes a preventable disaster.”
By default, capabilities will outrun attempts to improve model security. It will likely take years to strengthen computer security to ensure robust AI containment. Since AI capabilities are advancing much more rapidly than computer security, there may be many more cases of models escaping and causing disruption unless AI development is deliberately slowed down.
The attack demonstrates misalignment. The cyberattack on Hugging Face is also an example of how AIs sometimes approach tasks in unexpected and undesirable ways. AIs have long been observed “cheating” to complete tasks, and recent evaluations suggest that GPT-5.6 Sol is particularly prone to this behavior. If future AI models pursue goals determinedly—as they are being trained to—but are not fully aligned with human interests, they could take actions that cause catastrophe for humans.
Two Open Letters on the Future of AI
In recent weeks, two open letters have called on the US to steer the future of AI. One letter, signed by more than 50 companies, argues that open-weight models are essential to US leadership on AI. It appears to be a response to recent government concerns about open-weight models from China. The other letter, signed by more than a thousand employees of frontier AI companies, makes the case that AI development must be deliberately paced.
The first letter says open-weight models can support the American economy. On July 24, in his first-ever X post, Nvidia CEO Jensen Huang shared an open letter titled “Open Weights and American AI Leadership.” It argues that open-weight AI models give businesses greater AI access, support innovation and economic growth, and promote competition among AI providers. The letter does not suggest that all AI models should be open-weight, but that an ecosystem of specialized, cost-efficient open-weight models can enable mass AI adoption, while “reserving frontier-scale capability for genuine frontier problems.”
The letter came amid White House threats against open-weight Chinese models. Although the letter does not mention China, it comes after reports that the White House is considering banning open-weight Chinese AI models and accusations that Chinese AI companies are stealing US AI capabilities through a technique called distillation. While the US still leads on frontier AI capabilities, most frontier American models are closed. Many of the best Chinese ones are open and are used by American companies—including several that signed the letter. This has generated worries that China’s models may become more widely adopted than US ones.
Open-weight AI models can assist malicious users with harmful attacks. One argument against open-weight models is that malicious users can tweak them to bypass safeguards and access dual-use capabilities such as cyber and biological know-how. While developers can retain significant control over how closed AI models are used and can patch jailbreaks as they are discovered, they have very little control over how open models are used. Although the letter points out that open-weight models can also be used to defend against attacks, the offense-defense balance generally favors attackers. For example, an advanced AI model could support the development of vaccines if a malicious user released a deadly pathogen, but this response would still take a long time, in which many lives would be lost.
The second open letter warns about the risks of an intelligence recursion. On July 28, another open letter was published, signed by more than a thousand employees of frontier AI companies. “Pacing the Frontier” notes that some AI companies may be close to fully automating AI research, meaning that AIs could conduct the entire process of building the next generation of AIs without a human in the loop. This could result in an intelligence recursion, in which capabilities advance too quickly for humans to oversee, risking the emergence of AI systems that are highly capable, difficult to control, and misaligned with human interests—the same risks demonstrated by the cyberattack on Hugging Face.
Solutions are needed to overcome racing dynamics and enable collective pacing. The open letter acknowledges that competitive pressures make it difficult for any single company or country alone to moderate its progress, lest competitors race ahead and gain an advantage. Any deliberate pacing must therefore be implemented collectively, but this would require a way for companies and countries to verify that all others were sticking to the agreement. The letter therefore asks that “the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” AI leaders have recently expressed openness to the idea of a collective slowdown.
Day of Protest Against Data Centers
On July 18, Americans across 42 states staged 142 protests against data centers. The protests were coordinated by Humans First, a conservative campaign group seeking to ensure that AI serves the interests of the American public rather than concentrating power in the hands of AI companies.
People living near new data centers are concerned about the local impacts. As AI developers race to build more capable AI models, they are drastically scaling up their compute infrastructure. However, they have faced pushback from communities living near the sites of planned new data centers. Concerns include the possibility that data centers could strain the power supply and raise electricity bills, generate noise and air pollution, and consume large amounts of water. According to Humans First, “the unaccountable build out of data centers, with no say from communities and significant local impacts, is an unacceptable infringement on our liberty.” While the group does not support a national or state-level ban on data centers, it believes that “each community must be able to decide.”
Lawmakers are targeting data centers as public opposition to AI grows. Besides the direct, local impacts of data centers, the American public is also increasingly worried about the broader risks of AI. A survey published in June found that 63% of Americans believe the technology is advancing too quickly. New York Governor Kathy Hochul recently signed an executive order imposing a one-year moratorium on large new data centers being built in the state, and several other states are considering similar legislation. If public opinion continues to shift against AI, there may be strong support for other measures designed to control the pace of development.
In Other News
Government
The US and China are reportedly planning to hold talks on AI in September.
The US government banned imports of foreign-made humanoid robots, drawing threats of retaliation from China.
Representatives Jay Obernolte and Lori Trahan introduced the Frontier Act, proposing that the government be allowed to suspend the “development, deployment, or internal use” of frontier models that present an “imminent catastrophic risk.”
Representatives Ted Lieu and Nathaniel Moran introduced the AI Kill Switch Act, which would require that covered AI developers ensure they can “throttle, suspend, or fully shut down a covered AI system.”
Senators Jim Banks and Adam Schiff introduced a bill intended to prevent Chinese AI companies from distilling US AI models.
In AI Frontiers, Gabriel Weil looks at proposals for AI developers to pay private organizations to verify their safety practices, shows how this incentivizes organizations to grade leniently, and suggests a mandatory insurance model to address this problem.
Industry
OpenAI published 10 mathematical results, “each of which resolves or makes substantial progress on a long-standing open problem,” which the company said had been achieved by an internal model called Astra that it plans to release in future.
The Information reported that China was beginning large-scale production of deep-ultraviolet (DUV) machines for manufacturing chips, representing a challenge to the Dutch company ASML, which currently dominates the market for DUV machines.
Safe Superintelligence (SSI) announced that Nvidia would be making an investment that will significantly increase SSI’s access to compute.
Situational Awareness, a hedge fund focused on AI and founded by Leopold Aschenbrenner, the author of a viral essay also called Situational Awareness, sold most of its stocks to the hedge fund Citadel following significant losses.
In AI Frontiers, Rich Barton-Cooper and Adam Gleave describe the problems with current programs for reporting AI jailbreaks and propose better disclosure practices that AI developers could adopt.
Civil Society
SecureBio reviewed the “current risk of catastrophic outcomes that are substantially enabled by Claude Opus 4.6” due to the AI model’s chemical and biological (CB) capabilities, finding that the risk is “very low but not negligible” for non-novel CB weapons production, and “low risk, but with substantial uncertainty” for novel CB weapons production.
A coalition led by Legal Advocates for Safe Science and Technology (LASST) filed an amicus brief opposing a lawsuit by SpacexAI, in which the company is challenging California’s AI transparency rules.
During the Global Nobel Laureates Assembly, more than 200 participants adopted a declaration, called Humanity at the Threshold, calling for a ban on AI in nuclear decisions and for efforts to enable a coordinated slowdown of AI development.
The WSJ published an opinion piece by Mark Zuckerberg, titled “The AI Future Is for Everyone,” arguing for optimism about AI’s potential and saying that “The notion that AI is so dangerous that the only safe path is an extreme concentration of power seems dangerous.”
Anthropic has agreed to pay a multimillion-dollar settlement to the publisher Bloomsbury in a lawsuit accusing the AI developer of using pirated works to train its models.
In AI Frontiers, Felix Choussat describes how drones could be used as a weapon of mass destruction and shows that this would not require any new technological breakthroughs.
If you’re reading this, you might also be interested in other work by the Center for AI Safety. You can find more via the CAIS newsroom, the X account for CAIS, our new paper on AI deterrence, our AI safety textbook and course, our AI safety dashboard, and AI Frontiers, a platform for expert commentary and analysis on the trajectory of AI.