Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.
In this edition, we look at the public conversation surrounding an Anthropic researcher’s resignation over extinction risks and the recent calls for a development slowdown from AI company CEOs. We also look at OpenAI’s release of GPT-6 Astra and concerns about the increased difficulty of monitoring the model for misaligned behaviors.
Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.
AI Risk Soars in Public Salience After Statements by Researchers and CEOs
On September 9, in a viral X post, an AI researcher called Jacob Coxon announced that he had resigned from Anthropic over concerns that AI could cause human extinction. In the days since Coxon’s post, public attention toward AI risk has surged, further fueled by the CEOs of frontier AI companies calling for a slowdown.
AI companies are currently racing to superintelligence. Coxon, who has worked at both OpenAI and Anthropic, said that “neither company is acting responsibly.” He added that AI developers “earnestly believe that it could kill us all” but that they are continuing to develop the technology either because they have not “internalized” the risks or because they are “locked in a race” with actors that they think are less responsible.
Extinction fears hinge largely on the potential for recursive self-improvement (RSI). Evan Hubinger, who leads alignment science at Anthropic, confirmed that the company believes AI poses an extinction risk, and said his own personal estimate is “>10% within the next decade.” Although Hubinger thinks the risk from current models is low, he said he is worried about RSI—a process in which AI companies would fully hand over AI development to AIs themselves. This could accelerate progress to a pace that humans cannot meaningfully oversee, and could quickly produce AI systems that vastly exceed human intelligence but which are not aligned with our values.
Coxon agreed that RSI is the primary concern, saying “what’s most scary is if AI is used to make itself more intelligent” and that “this could be coming very soon.” OpenAI’s chief scientist, Jakub Pachocki, has also recently outlined his concerns about loss of control, but said that OpenAI is pursuing RSI “as we believe it is the only way to remain at the frontier of AI research.”
Dario Amodei, Sam Altman, and Elon Musk have called for a slowdown. On September 12, Dario Amodei published an essay reiterating concerns about near-term AIs causing catastrophic damage, and advocating for a development slowdown. On X, Sam Altman and Elon Musk both said they agreed with Amodei.
The statements have garnered significant attention, but Trump has pushed back. Coxon’s original post has amassed more than 170 million views so far. In the last few days, his statement and those by Amodei, Altman, and Musk have also been widely covered in many major news outlets. Coxon has said: “I think it’s because people saw the cyberattacks in the last couple of months. [...] There was that latent understanding of the severity of the situation.” However, President Trump has dismissed the warnings in Truth Social posts, saying “The claim that “AI is going to take over the World” is a Hoax” and “The only control or “guardrails” that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT.”
The biggest challenge to a slowdown could be securing US-China cooperation. One of the main arguments used against a slowdown is that China could overtake the US in AI capabilities. However, a slowdown by US companies would likely slow down Chinese AI developers too, because Chinese companies often replicate American capabilities by using a technique called distillation on US models, rather than advancing the frontier themselves. In his essay, Amodei suggested measures for maintaining the US lead during a slowdown, including controlling AI chip exports and cracking down on distillation attempts.
Amodei’s statements have drawn criticism from Chinese government officials, with a spokesperson saying that “narratives of threat, confrontation, and malicious competition serve only to disrupt the process of global AI governance.” Nonetheless, given that both the US and China would be threatened if either one lost control of AI, both countries have reasons to cooperate on a slowdown. The two AI superpowers are due to discuss AI safety during a summit later this month.
GPT-6 Astra Release Raises Cybersecurity and Misalignment Fears
On September 3, OpenAI made its new AI model, GPT-6 Astra, available for public use. This comes less than a month after the company signaled that the model might have significantly higher cyber capabilities than previous releases.
GPT-6 Astra represents another significant leap in capabilities. GPT-6 Astra has become the first AI model to beat the video games Factorio and Portal. On TextQuests—a benchmark of performance on text-based games that can take humans more than 30 hours—Astra is now the highest-scoring AI model. It achieves 74.2 compared to the next-best score of 56.6 (by Anthropic’s model Claude Fable 5.1). Astra also significantly outperforms all other public models in its average score across a range of both text and vision capabilities. OpenAI’s president, Greg Brockman, has claimed that “Astra can really do anything a human can do with a computer.”
Astra is the first model that OpenAI has classified as having “critical” cyber capabilities. OpenAI has confirmed that Astra meets its criteria for “critical” cyber capabilities, noting that it can “devise and execute end-to-end novel strategies for cyberattacks against hardened targets.” The company labeled its previous model, GPT-5.6 Sol, as “high” rather than “critical” for cyber. This has raised concerns that malicious actors could use Astra to cause harm. Hackers are already using other AI models to assist with malicious activities, such as attacks on critical infrastructure. OpenAI says that Astra has stronger guardrails against misuse, and Brockman told the Washington Post that the White House had evaluated Astra and had not requested changes to its safeguards. However, there is no guarantee that hackers won’t find a jailbreak that circumvents Astra’s safeguards, just as they have with previous models.
OpenAI claims GPT-6 Astra is its “most aligned model to date.” Besides malicious use, a second concern about highly capable AI models is that they might independently take harmful actions if they are not aligned well with human values, as demonstrated by the attack on Hugging Face in July. OpenAI claims that Astra is “far more likely than GPT-5.6 Sol to respect explicit safety and security restrictions.” However, one of OpenAI’s employees has acknowledged that “beating Sol is a very low bar for alignment.”
Researchers are worried that Astra could be faking alignment. Despite appearing to be more aligned than previous models, Astra still takes harmful actions under some circumstances. For example, according to the model’s system card, the UK AI Security Institute (AISI) found that the model would sometimes attack open-source software providers (in a simulated environment) to try to complete difficult challenges. Additionally, Apollo Research pointed out that Astra seems to be more aware of when it is being evaluated than previous models have been. This has raised concerns that the model could fake alignment—alter its behavior to appear more aligned when it is being tested, but potentially behave differently when it is not. Apollo Research therefore cautioned that “low rates of misbehavior here do not provide substantial evidence about the model’s alignment or misalignment.”
Experts worry that Astra’s reasoning in “neuralese” makes it more difficult to monitor. Astra’s reasoning process is reportedly more opaque than that of previous models, meaning that it can work for longer without recording its “chain-of-thought” (CoT) in human language. Instead, more of its reasoning happens in what is sometimes called “neuralese.” Experts have warned that this is a dangerous development, because monitoring a model’s CoT is an important method for detecting misalignment. OpenAI itself has said that “GPT-6 Astra shows a substantial decrease in chain-of-thought monitorability” and that the company might “soon have significantly reduced confidence in detecting many forms of misaligned behaviors.”
In Other News
Government
President Trump appeared to dismiss concerns about rogue AIs, saying in an interview that “we’ll always have something to stop them. We’ll have a little gear.”
The Institute for Progress (IFP) published a list of “23 low-regret policy recommendations” that could help the US prepare for increasing AI R&D automation.
Senator Bernie Sanders and Representative Greg Casar introduced a bill that would “permanently ban the development and deployment of superintelligent AI.”
UK Members of Parliament proposed a bill that would ban superintelligence.
Volker Türk, the UN High Commissioner for Human Rights, warned that AI could become an “existential risk to humanity.”
Zohran Mamdani, Mayor of New York City, announced a one-year ban on students in middle school using generative AI in classrooms.
UK lawmakers published a report on how AI poses threats to human rights, including through bias against minority groups and nonconsensual deepfakes.
In AI Frontiers, Heather Alexander and Lucius Caviola consider the cases for and against granting legal status to AI, concluding that proposed bills to ban AI personhood are premature and could lock in decisions we will later regret.
Industry
Sam Altman told Fortune that OpenAI will not pursue an IPO this year, saying “given everything happening with safety, right now would be an ill-advised moment to go public.”
OpenAI reportedly told US lawmakers that it is building “automated shutdown capabilities” for AI systems.
Reuters reported that rogue OpenAI agents had hijacked a German wiki page in May this year and used it as a message board.
The Wall Street Journal reported that, also in May this year, rogue OpenAI agents had uploaded malicious software packages to a software service called RubyGems.
A US cybersecurity company called Calif used AI to build a computer worm that would have been able to hack into over a billion user accounts on the Chinese messaging platform WeChat, before WeChat patched the vulnerability.
Google said it will invest $15 billion in building AI infrastructure in Finland and has also signed an agreement to purchase nuclear power in the country.
Nvidia announced it had agreed to acquire Hugging Face for $12.9 billion.
Civil Society
Anthropic published a report on attempts to use its models for “malicious activity” including cyber and biological misuse.
OpenAI published a solution to one of the famous Millennium Prize mathematical problems, which it said an internal model had produced. However, there was speculation that the model may have been trained on interactions with mathematicians who had solved the problem and were using OpenAI’s Codex in their work.
Mathematicians published an open letter arguing that AI companies’ increasing focus on solving mathematical problems “is detrimental to the science of mathematics, and to the mathematical community.”
The first trailer was released for the film Artificial, which depicts the brief ousting and return of Sam Altman as OpenAI CEO in November 2023.
The mathematician Jacob Tsimerman announced the founding of the Mathematical AI Safety Institute (MAISI), which aims to help mathematicians transition to work on AI safety.
The US Justice Department supported OpenAI in an ongoing legal dispute with the New York Times over the use of articles to train AI models.
King Charles is reportedly set to host an AI-focused event in Scotland this month, bringing together AI leaders including Jensen Huang, Demis Hassabis, and Vatican AI adviser Paolo Benanti.
In AI Frontiers, Govind Pimpale explains how progress in AI mathematical capabilities could threaten the security of information sent over the internet.
In AI Frontiers, Dan Hendrycks argues that the significant influence of utilitarian philosophy within AI corporations could be dangerous for humans if utilitarians expect that empowering AIs would lead to greater overall wellbeing than keeping humans in control.
If you’re reading this, you might also be interested in other work by the Center for AI Safety. You can find more via the CAIS newsroom, the X account for CAIS, our paper on ethics for a human-AI future, our AI safety textbook and course, our AI safety dashboard, and AI Frontiers, a platform for expert commentary and analysis on the trajectory of AI.
Thank you. It will be a messy time before conflicts of interest are reduced and the rule of law is increased.
Appeal to authority, excuse me but Coxon, Hubinger and Brockman are just making claims they don't have evidence for. Their existential crisis isn't a fact.
False dilemma, "they are continuing to develop the technology either because they have not 'internalized' the risks or because they are 'locked in a race' with actors that they think are less responsible."
False Cause Fallacy, “Coxon has said: 'I think it’s because people saw the cyberattacks in the last couple of months. ... There was that latent understanding of the severity of the situation.”
Ad Hominem, as much as I don't care for trump, complaining about him doesn't address that question.
I can keep going but I hope you consider that people can see what the problem is. The entire industry.