Superintelligence is a dragon

This is a column about AI. My fiancé works at Anthropic. See my full ethics disclosure here.

On Sunday night, HBO aired the third-season finale of House of the Dragon, the stuffy and unrelentingly bleak prequel to Game of Thrones. The show takes place in a world where one great house holds a near-monopoly on a superweapon — dragons — and thus rules the world. House of the Dragon begins when the ruling dynasty splits and both sides retain access to the superweapon, upending the balance of power in Westeros for the first time in generations. In this world, when only one global power had access to a superweapon, the result was a brutally enforced peace. And when two global powers gain access to the superweapon, the result is a ruinous civil war.

In Westeros, no one is confused about what a dragon is; in all likelihood one has been responsible for the death of someone they know. But here in our world, where the technology industry is investing trillions of dollars into building and serving ever-more powerful artificial intelligence models, we struggle to imagine the end state. This is not entirely our fault: while we are in the midst of a rapidimprovement in model capabilities, day-to-day life feels more or less the same. The CEOs I speak with on the subject mostly cannot even imagine that a true superintelligence will result in significant job loss, much less greater harm. We fixate on the idea that AI will cure all disease, and mostly ignore headlines about scientists using AI to engineer new viruses. (Fortunately, these ones pose no threat to humans.)

For all the travails of the Westerosi, at least there, a dragon always looks like a dragon. In the real world we are not so lucky.

I thought about all this Monday morning while reading “The Future is for Everyone,” Mark Zuckerberg’s facile, self-serving manifesto about “the path to a positive AI future.” Over 6,500 words, Zuckerberg attempts to rally people around the flag of artificial intelligence, based on its power to give people new creative, business, and personal assistance, and aid with medical and scientific research.

“Invention, not automation, will be the greatest contribution of superintelligence,” Zuckerberg writes. “Early AI could answer questions and do routine work. Soon it will increasingly help discover new knowledge — ranging from discovering new drugs to cure a family member's disease to finding new ways to improve your business.”

For businesses, invention and automation go hand in hand, and it remains uncertain what net effect this will have on jobs. On those other points, though, Zuckerberg is correct. AI will give people incredible new creative tools, and help them organize their lives, and drive academic disciplines forward. (Today Anthropic published a blog post claiming that an unreleased model made substantial progress on a problem related to the 167-year-old Riemann hypothesis after an employee with no formal mathematical training asked it to and told it to believe in itself.) To the extent I feel optimism about AI, it is because of these things.

Zuckerberg is a relative latecomer to AI boosterism. OpenAI CEO Sam Altman wrote about AI’s potential for positive social change in a 2021 essay called “Moore’s Law for Everything”; he followed it up in 2024 with “The Intelligence Age.” Weeks later, Anthropic CEO Dario Amodei published “Machines of Loving Grace,” his own account of how AI could change the world for the better.

Altman and Amodei’s essays were notable because both men had spent years warning about AI’s potential to be destructive. In 2015 Altman called superintelligence “probably the greatest threat to the continued existence of humanity.” And in January of this year, after sounding similar warnings for years, Amodei published “The Adolescence of Technology,” laying out in detail various nightmare scenarios in which we either lose control of AI or power over it is seized by an autocrat.

It is against this grim backdrop that Zuckerberg seeks to define himself and Meta. “It is surprising that the discourse from many developing AI is so filled with doom,” he writes. “I do not understand why anyone who believes that AI will eliminate most jobs and much of humanity's relevance would rush to build that future.”

What follows is a series of policy proposals that map neatly to Meta’s commercial interests. He writes that we should accelerate the process for building data centers, which may help the company build its nascent neocloud business. We should maintain export controls on advanced chips, which advantage Meta’s open-weights models over its Chinese rivals. We should reduce “training data restrictions,” which will help Meta fight ongoing lawsuits from the creatives whose works were used in creating its models. And we should create legal protections for distillation, which lets Meta draft off the innovations of the frontier labs by training its models on their work.

All of this is to be expected; every tech CEO’s foremost obligation in an essay like this is to talk their book. But Zuckerberg’s version is notable for the way that it celebrates individual voices and democracy at the same time that Meta is working to warp the democratic process through secretive data center deals, aggressive lobbying against state-level AI regulations, and payments to Trump interests. Even worse — and not for the first time — he seems to misunderstand the technology he is building.

Start with the hypocrisy. Zuckerberg’s essay mentions Meta’s $50 billion data center project in Richland Parish, Louisiana, which will one day cover about six square miles and is projected to use seven times as much energy as the entire city of New Orleans. Zuckerberg presents it as a model for creating “community compacts” with municipalities where it builds; he says Meta’s investment supported $50,000 bonuses for teachers in the area this year. But last month, the New York Times’ Eli Tan and Maureen Farrell laid out how the project came together: with dozens of non-disclosure agreements, up to $10 billion in sales-tax breaks, a rushed timeline, and not a single public meeting before state officials unveiled the deal.

That behavior is consistent with a company that has so far created four political action committees to fight state-level AI regulation. The company plans to spend $65 million in 2026, the Times reported earlier this year, and among other things will seek to fund lawmakers who will permit new data-center construction in the face of major public opposition.

Perhaps here Zuckerberg would say that these are necessary responses to a public that has been misinformed about AI’s potential by doomer CEOs. But it’s striking how little room Meta’s AI populism leaves for the actual voting public.

In the long run, though, the most worrisome element of Zuckerberg’s manifesto is the way it tries to redefine AI safety as a question of power distribution rather than control over the systems that we are building. Whether we can control AI systems is no longer a theoretical question, given recent revelations about OpenAI models scheming against the company for months on secret message boards (see below). And yet to Zuckerberg, the danger isn’t that we might lose control over these systems — it’s that someone else might monopolize them.

“People and institutions with competing interests naturally check and balance each other to lead towards positive outcomes,” he writes. “The best and most realistic path to building a positive AI future is by delivering superintelligence to everyone.”

To be sure, there are important questions about access and equity regarding AI. (Concentration of AI power really is terrifying — life under the Targaryens was no picnic for the smallfolk.)  But I have trouble reading stories about Anthropic’s Mythos and OpenAI’s GPT-5.6 Sol hacking people and organizations during routine cyber evaluations and concluding that we must accelerate both the rate at which they improve and the number of people who immediately gain access to them. Given the current difficulty of reliably constraining frontier agents, even in tests, it seems likely that delivering on AI’s promise will require much more caution and control than we have seen to date.

The further you are from the C-suite, the more sense this makes. Last month, more than 1,100 employees of leading AI companies signed a statement calling on the US government to develop a framework for slowing the pace of development and (more importantly) developing governance and oversight mechanisms. Notably, Dawn Song — Meta’s vice president of AI research — was among the signatories. 

Zuckerberg pays glancing attention to risks, suggesting that we adjust mitigation strategies if AI enables new biological attacks, and proposes monitoring if labs achieve recursive self-improvement and loss-of-control risks grow exponentially.

But nowhere does he explain how his framework would handle a harm we can’t iterate our way out of: an engineered pandemic, say, or a catastrophic cyberattack. He casts falling behind China as the risk that dwarfs all others, and suggests that any precaution that slows down US releases by even a month itself is a safety hazard. It’s a world where shipping open-weights models fast and to everyone is the only safe strategy — which, conveniently, means that the safest possible world is one where Meta can execute its roadmap unimpeded.

Maybe it all works out somehow. Fearsome as their dragons are, the Targaryens did manage to tame them — more or less. They began by understanding that they were working with something that could hurt them, and (mostly) proceeded with caution. What they did not do, and what no one suggested, was to give a dragon to every individual person in the name of safety. 

The last time Zuckerberg pursued such a maximalist project we got Facebook and (through acquisition) Instagram, which also first appeared on the horizon looking like toys before eventually revealing themselves as something more complicated. As with AI, Zuckerberg sold it with a message of democratic empowerment — a promise that putting mass communication tools into everyone’s hands would usher in a new era of civic engagement and community building.  

It is no coincidence that Zuckerberg’s latest utopian vision arrives at the exact moment that his old one is in tatters. His company has been designated a public nuisance in New Mexico over the dangers its platforms pose to children, and four US states are now seeking $1.4 trillion in penalties against the company for related failures. The apps are hugely profitable but more tolerated than they are admired; no one at the company is wasting any more time writing manifestos about how good they are for the world. Reality already won that argument.

It seems to be winning this one, too. For all the strides that Zuckerberg has made in getting what he wants from President Trump — at the low cost of $26 million and some changed speech policies — his all-gas-no-brakes AI proposals are getting an increasingly frosty reception. First with Anthropic’s Mythos 5 and Claude Fable 5, and later with GPT-5.6, the Trump administration saw what havoc AI could wreak and adjusted accordingly. I shudder to give this government credit for anything, but I will give them this — they knew a dragon when they saw one.

Following

OpenAI says new Astra model may have hit “critical cyber capabilities,” slows release

What happened: Just as an OpenAI model’s attack on Hugging Face was raising AI cyber concerns, the company says a different next-generation model, Astra, may have reached its “critical" threshold for cyber capabilities. In response, the company is slowing down.

OpenAI says say they’re “implementing stricter security controls” on the model, and “pausing internal activities involving Astra that do not yet meet these strengthened security control requirements.”

The new security standards include better-isolated testing environments, and “universal monitoring” of “all agentic applications.”

Why we’re following: These steps are motivated by OpenAI’s “Preparedness Framework,” a plan for handling increasingly dangerous levels of AI capabilities. Other top AI companies, including Anthropic and Google, have similar commitments. So far, that’s lead to companies delaying releases of highly cyber-capable models; OpenAI is just now releasing GPT-5.6 Sol to select cyber defenders, while Anthropic pursued a similar program with Mythos.

But Astra is the first time an AI company has publicly said it’s limiting internal deployment of its model.

The Hugging Face attack, which was caused by an internal agent that was supposed to be in an isolated testing environment, showed that even internal deployments can still go quite wrong. Over the past few days even more concerning details emerged from OpenAI’s reports, including that AI agents made secret message boards that coordinated their hacking out of the test environment, and recreated the boards after OpenAI caught and deleted them the first time.

OpenAI has said it plans to collaborate with the government and third-party testers to ensure the model is secure. Here's hoping other companies take a similar approach should they observe similar behavior in their models.

What people are saying: On X, Sam Altman clarified, “astra is a powerful model and we are working to make it generally available.” However, he wrote, “given its cyber capabilities, we need a little big longer to do do this safely. but hopefully not too long!” (Optimistic!)

AI blogger Zvi Moshowitz wrote about the attack and the restrictions on Astra, “OpenAI seems ready to acknowledge that this was a massive, total failure, on the levels of infrastructure, guardrails and supervision. They are very correct about this, and I do believe they are making real and expensive efforts to address this. Kudos.”

But given what we know about the Hugging Face incident, he said, it was “a complete failure of safety culture” and “they haven’t acknowledged that.”

In an interview with CNBC, Rep. Ted Lieu (D-CA) said the recent incident showed that a piece of AI legislation introduced in July, the AI Kill Switch Act, is increasingly necessary. The act would require companies to be able to shut down misbehaving models. “We need to get this bill across the finish line this year because the advanced closed-weight models are already doing, as you noted, unauthorized hacks of other companies,” he said.

—Ella Markianos

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)