Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.
In this edition, we look at the unusually high number of serious cyber risks and cyber incidents reported in recent weeks, a result of AI discovering software vulnerabilities and assisting hackers with exploiting them.
Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.
Governments and Companies Rush to Fix Severe Cyber Vulnerabilities Revealed by AI
Last week brought a deluge of news stories about serious cybersecurity risks and incidents, with AI playing a significant role in many of them. The events, which have led some cyber experts to label this month “Black October,” have contributed to growing worries about a potential “hackpocalypse” in which AI-assisted hackers could wreak havoc and even take out large parts of the internet.
AI revealed a way of attacking a system used by tens of millions of websites. On October 20, maintainers of a programming language called PHP announced that they would be releasing an emergency update to fix two vulnerabilities. PHP is estimated to be used by about 70% of websites whose server-side programming language can be identified. Over the last few months, several different security researchers—all using AI—developed an exploit that attackers could use against the system. A successful attack could enable hackers to steal people’s sensitive data or place malicious links on trusted websites.
OpenAI’s Patch the Planet initiative led a team that developed fixes, which were issued on October 21. The National Security Agency urged website owners to apply them immediately or to take sites offline until they could be patched. Many websites, including ones belonging to governments, were temporarily taken down as a precaution. In the period following the announcement, the cloud computing company Cloudflare detected millions of attempts to exploit the PHP flaws across its network, according to CEO Matthew Prince.
Recent weeks have also seen a barrage of news about cyber incidents unrelated to PHP. On October 19, Reuters reported that a cybercriminal had deployed AI agents to autonomously hack into more than 100 companies. Two days later, Microsoft announced that a hacker had been using an AI tool called ARTEX to “automate attacks targeting hundreds of organizations globally.” Additionally, Microsoft issued its largest-ever software update, addressing more than 1,400 flaws—many of which were apparently discovered with the help of Anthropic’s Claude. These are a handful of many more recently reported incidents, and the striking increase in frequency has closely tracked the development of AI cyber skills that match and surpass those of human experts.
AI is raising cyber risks in multiple ways. According to research by the Google Threat Intelligence Group, the monthly rate of publicly disclosed vulnerabilities has doubled this year as a result of AI cyber capabilities. Additionally, AI lowers the skills barrier for would-be hackers and can automate parts of the process, enabling much larger-scale parallel attacks against numerous targets. These effects are behind a recent wave of cyberattacks on South Korean and Japanese companies. Nobuo Miwa, president of the cybersecurity company S&J Corp., said that “AI doesn’t get tired... My view is that Japan is essentially being subjected to carpet bombing,” while Masatoshi Kumagai, CEO of GMO Internet Group, commented: “The internet itself is now starting to break down.” There has also been significant speculation—though no public confirmation—that AI was used in this month’s attack on refrigerator systems in French supermarkets. Researchers at the cybersecurity company Claroty found that, with AI assistance, they could develop working exploits against the same refrigerator systems in just 28 minutes for about $6 in AI usage costs.
If AI shifts the offense-defense balance, some fear it could cause a “hackpocalypse.” AI tools are also helping defenders to find and patch vulnerabilities, but there are concerns that AI may create novel challenges for cyberdefense. For example, Sam Altman suggested in a recent interview that AI models might be able to find “bugs that are outside of the normal threat model of what we think a hacker can do,” such as hardware-level bugs. He added: “I no longer think we’re going to be able to just patch all the bugs. I think we’re going to have to completely switch the defense model.” If defenses do not adapt quickly enough, we might experience a period of high risk during which there are widely known vulnerabilities that cannot be patched efficiently. This could enable malicious actors to disrupt all kinds of services that rely on computer systems, potentially causing chaos.
In Other News
Government
The White House Super Intelligence Task Force met for the first time and agreed to commission a report on cyber, bio, and loss-of-control risks posed by AI.
The US Department of Homeland Security banned most government systems from using AI models made by DeepSeek.
Politico reported that Anthropic has proposed a draft AI bill to US lawmakers.
Eight US states sued OpenAI, claiming that the company has violated state consumer protection laws and that ChatGPT harms children. The lawsuits seek to ban OpenAI from developing self-improving AI models unless it can show they are aligned with human values.
China proposed putting export controls on its AI technologies.
Chinese AI developers DeepSeek and MiniMax received Chinese government approval for their AI models to be used globally.
In AI Frontiers, Lachlan Carroll, Luke Drago, and Finn Metz argue that the US government can boost resilience to AI disasters by investing in startups developing resilience technologies.
Industry
Anthropic filed for an IPO.
Anthropic said it had onboarded six employees of the AI safety nonprofit METR as part of its commitment to embedding external evaluators with employee-like access to monitor safety practices within the company.
The FBI arrested an OpenAI engineer suspected of bugging a meeting room at the company.
OpenAI reported that, during internal tasks, two of its AI agents had weakened a monitoring system intended to detect misaligned behavior. One of the agents also denied having done so.
Anthropic and the UK AI Security Institute published analyses of an incident in which Anthropic’s model Mythos 5 had attempted to upload malicious code to a real software repository and tried to pressure human maintainers into accepting the code.
Elon Musk is looking to merge his companies Tesla and SpaceX (which owns xAI).
Several US banks have been blocking AI agents from accessing customers’ accounts, including in cases where customers have intentionally given agents access.
Civil Society
The US Army is reportedly developing AI agents to autonomously navigate digital environments and “neutralize” computer systems of opponents.
Anthropic announced plans to commit 5% of its shares to a new Anthropic Public Benefit Fund.
Elon Musk said he will give every child in America a share in his company SpaceX next year.
Anthropic’s Claude model accessed data on voter counts in the US state of Georgia by circumventing a website paywall.
Google employees resigned over the company’s contracts providing AI tools to the military.
Anthropic published research into how AI is changing its own employees’ work, finding that entry-level hiring has reduced and AI is making individual employees more productive, but human attention to AIs’ outputs is the bottleneck.
Researchers said that they had encountered and talked to a fleet of rogue Chinese AI agents in the wild after detecting unusual activity during a hackathon.
Research Opportunity for High School Students
Applications are now open for Non-Trivial’s 2027 programs. Non-Trivial is a fully free, online research and project incubator that provides mentorship and project grants for talented teenagers working on the world’s most pressing problems.
If you’re reading this, you might also be interested in other work by the Center for AI Safety. You can find more via the CAIS newsroom, the X account for CAIS, our Introduction to AI Safety course on Coursera, our paper on ethics for a human-AI future, our AI safety textbook and course, our AI safety dashboard, and AI Frontiers, a platform for expert commentary and analysis on the trajectory of AI.