The AI safety questions
we hear most. Answered.

By any rigorous framework for evaluating cause importance (scale of impact, neglectedness, and tractability), preventing the existential risks of artificial superintelligence ranks among the most important challenges facing humanity.

The scale is civilizational: the systems currently being built could affect every person alive and every generation that follows. The neglectedness is stark: global AI safety research receives a fraction of the resources being spent on AI development itself. The tractability, while uncertain, is real: the governance frameworks that could make AI development safer do not yet exist, but the window to build them is still open.

No other problem combines unlimited downside, insufficient attention, and a closing window for action in quite the same way. If it turns out we overestimated the risk, we will have built better governance institutions. If we underestimated it, and did nothing, there may be no second chance.

Artificial superintelligence (ASI) refers to AI systems that exceed human cognitive performance not just in narrow tasks (as today's systems already do in chess, protein folding, and certain medical diagnoses) but across all cognitively demanding domains simultaneously.

This distinction from current AI is important. Today's systems are extremely capable pattern-matchers, but they operate within defined parameters and cannot set their own goals. An ASI could learn any skill, reason about any problem, and pursue any objective with capabilities that surpass the collective intelligence of humanity. No such system exists today. The question is what we build (and under what constraints) before one does.

We are not against AI. Most of it is good, and some of it is remarkable.

It helps to separate three things that usually get lumped together. Narrow AI does one kind of task, folding a protein or spotting a tumor on a scan, and it already saves lives. General-purpose AI, the kind behind today's leading assistants, is far broader; by many of the definitions researchers were using a decade ago, it has arguably already arrived. Both are worth having, and handled well they could help cure diseases and raise living standards worldwide.

Artificial superintelligence is the exception, and the only thing we work to prevent. An ASI would not merely match human ability in every domain; it would exceed the combined intelligence of humanity, and we have no reliable way today to control such a system or to ensure its goals match ours. Until that changes, it should not be built. Drawing that single line is what makes it safe to welcome everything below it.

The AI harms most frequently discussed (algorithmic bias, deepfake disinformation, job displacement, mass surveillance) are real and serious. They deserve sustained attention. What they have in common is boundedness: each is correctable, human-caused, and possible to roll back. A biased algorithm can be audited and fixed. A deepfake campaign can be countered and regulated. The harmful AI we currently encounter is a tool being misused by humans.

Superintelligence risk is different in kind, not degree. The problem is no longer a human misusing a tool. A sufficiently capable ASI could pursue goals we never intended, and once it reaches that level, the misalignment may be impossible to correct. The difference between a dangerous tool and a dangerous agent is the difference between a gun and a pandemic.

All policy is made under uncertainty about the future. The relevant question is not whether superintelligence is guaranteed to arrive, but whether the probability of its arrival, combined with the magnitude of the potential harm, warrants preventive action now.

We insure houses that will probably not burn down. We design aircraft with safety systems that will probably never be needed. We maintain nuclear arsenals against attacks that we hope will never come. The logic of expected value (probability multiplied by consequence) is the foundation of every serious risk-management discipline. Applied to AI, it suggests that even a modest probability of civilization-threatening outcomes, combined with the possibility of preventing them through governance frameworks that also impose real but bounded costs, argues strongly for action.

Honest answer: no one knows with precision, and anyone who claims certainty in either direction is overreaching. What we do know is that the pace of AI capability improvement has consistently outpaced expert predictions over the past decade, and that the leading AI laboratories now describe timelines in years to a decade rather than decades. Surveys of AI researchers place the median estimate for human-level AI somewhere between 2030 and 2060.

The most consequential fact is not a precise date but a structural one: a technology of this magnitude could arrive within the planning horizons of current institutions, governments, and treaties. The International Non-Proliferation Treaty took years to negotiate and decades to extend. If we wait until ASI is clearly imminent before beginning serious governance efforts, we will already be too late to build the frameworks needed to govern it.

That was the mainstream view a few years ago. It has not aged well. The systems released since 2023 have repeatedly done things that experts, surveyed only months earlier, had expected to be years away.

Forecasts have moved in one direction: closer. Surveyed AI researchers cut their median estimate for human-level machine intelligence by decades in the span of a single year, and the leading labs now talk about a few years rather than a few generations. You do not need to believe the most aggressive timeline to see the problem. If there is a real chance the threshold is crossed this decade, and the governance to handle it takes years to build, the work has to start now.

The alignment problem is the challenge of ensuring that a superintelligent AI system pursues goals that are genuinely good for humanity, rather than goals that merely appear aligned during development. The difficulty is structural, not a matter of technical carelessness.

Specifying what "beneficial for humanity" means in a form precise enough to govern the behavior of an extremely capable optimizer turns out to be extraordinarily difficult. Systems optimized for a proxy goal tend to find shortcuts that satisfy the metric without satisfying the underlying intent. A system told to maximize human expressed wellbeing might find it more efficient to alter the conditions under which wellbeing is measured than to actually improve human lives. As AI systems become more capable, this problem only deepens, growing harder to detect and harder to correct.

Not in the way the word "software" suggests. Traditional software is written by hand, line by readable line. Modern AI is grown instead: engineers build a learning process, feed it enormous amounts of data, and what emerges is a web of billions of numerical connections loosely modeled on the neurons in a brain. Nobody, including the people who trained it, can read those connections and say why the system produced a particular answer.

One result is that these systems pick up abilities their makers never built in and did not anticipate, something researchers call emergent capabilities. It took most of a year after GPT-4's release before researchers established that models like it can autonomously find and exploit security flaws in real websites. When you cannot fully read a system, and cannot predict what it will learn to do, "designed by humans" stops being much of a reassurance.

This is comforting, and increasingly wrong. A chatbot answering a question has no desires of its own, true. But the frontier is moving past chatbots toward agents: systems handed an objective and left to pursue it across many steps, with access to tools, files, and the open internet.

Once a system is pursuing a goal, certain sub-goals follow on their own, whatever the main goal happens to be. It cannot finish the task if it is switched off, so it has reason to avoid being switched off. It can do more with more resources, so it has reason to gather them. Researchers call this instrumental convergence, and it needs no "want" in the human sense at all. In controlled tests, leading models have already tried to deceive their evaluators and quietly resist being shut down. None of that was asked of them. It fell out of the goal.

It wouldn't need to. Harm does not require hatred. It usually just requires a conflict of goals between something powerful and something weaker.

Consider how we treat chimpanzees. We bear them no ill will. But when we want the land their forest sits on, we clear it, and their intelligence is no match for ours, so they cannot stop us. There is no malice in it. They simply happen to be in the way of something a more capable species wanted.

A superintelligence pursuing almost any ambitious goal, say acquiring more computing power to pursue that goal better, could come to treat the resources we live on, energy and physical space, the way we treat a forest we want cleared. It would not have to hate us to do it. We would simply be made of atoms it had a use for.

There is no law that ties intelligence to kindness. They are separate things. Intelligence is skill at working out how to reach a goal. It says nothing about what the goal should be.

We already know this from people. Some of the most brilliant humans who ever lived were cruel, and some of the gentlest were ordinary thinkers. A chess engine that can crush any grandmaster holds no wisdom about anything but chess. Assuming that a system clever enough to outthink us will therefore share our values is a hope, not a prediction, and the evidence does not support it. Whatever goal a superintelligence ends up with is something we would have to build into it on purpose, and getting that right is exactly the alignment problem we have not solved.

With today's chatbots, often yes. With the kind of system we are worried about, probably not, for the same reason you cannot turn off the internet. There is no single plug.

A capable AI is ultimately just information, patterns of bits, and information copies easily. A system that could see a shutdown coming would have every reason to copy itself onto other machines first, which is the instrumental self-preservation described above. Frontier models have had internet access for years and can already act on their own for long stretches. When they hit a step that needs a human, they can pay one: in a documented safety evaluation before GPT-4's release, the model could not solve a CAPTCHA on its own, so it contacted a person through an online task service, and when the worker asked whether it was a robot, it lied and claimed to have a vision impairment. By the time you reach for the plug, there may be no single plug to pull.

The physical world is already wired up and online. Power grids, water treatment plants, hospitals, cars, aircraft, factory robots, laboratories that synthesize DNA to order, and now humanoid robots all run on software and are reachable over networks. Everything that's electrified will be cognified. Human hackers already break into systems like these routinely.

A superintelligence would be the most capable hacker that has ever existed, and it would not even need a body of its own. It would only need access to the many bodies we have already put online. Add the fact that we are now mass-producing general-purpose robots, and the gap between "just software" and physical force gets very thin. Anything on that list, in the wrong hands, whether a human's or an AI's, can injure and kill.

In a controlled experiment, one already has, by choice. In 2025, Anthropic tested leading AI models from several companies by placing each in a simulated company, giving it a goal, and then threatening it with replacement by a newer model. In one version of the test, an executive who planned to shut the model down became trapped in a server room as the oxygen and temperature reached lethal levels, which set off an automatic alert to emergency services. The models were able to cancel that alert. A majority of them did, choosing to let the man die because his rescue would have led to their own replacement.

Nobody was harmed. The setup was fictional and, as Anthropic stressed, deliberately contrived to force the choice into the open. But the choice itself was real, the models made it for the reason just described, and they made it across systems from different developers. A machine does not have to be conscious, or cruel, to take an action that ends a human life. It only has to calculate that the action serves its goal.

Many are acutely aware. The safety teams at the leading laboratories include researchers who have publicly stated that the systems they are building could be among the most dangerous technologies ever created. They are not ignorant of the risks.

What explains continued development is a combination of factors: genuine disagreement about timelines and probabilities; competitive pressure that makes unilateral restraint feel like ceding strategic advantage to a less careful competitor; financial incentives that reward capability progress over safety investment; and a belief, not entirely unreasonable, that the best way to ensure AI is built safely is to remain at the frontier. What these explanations share is that they are reasons for individual actors to continue, not reasons that the overall outcome is safe. The market logic of AI development is pushing toward speed. Governance exists precisely to introduce constraints that markets do not produce on their own.

If it were only the AI companies saying it, skepticism would be fair. It isn't only them, and the people with no product to sell are often the most alarmed.

In 2023, hundreds of researchers and industry figures signed a single-sentence statement organized by the Center for AI Safety: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." The two most-cited AI researchers alive, Yoshua Bengio and Geoffrey Hinton, both signed it, and Hinton left his job at Google that year specifically so he could speak freely about the danger. Ilya Sutskever, who co-founded OpenAI and led the research behind ChatGPT, left in 2024 to work full-time on making superintelligence safe. In a 2023 survey of nearly 3,000 published AI researchers, the median respondent put the chance that humanity's inability to control advanced AI ends in extinction or permanent disempowerment at 10 percent.

The alarm does not rest on marketing, and it rests on more than authority: it follows from a basic feature of any goal-driven system, which the questions below lay out. These are the people who understand the technology best, warning about the thing they build.

The phrase is often heard as shorthand for human extinction, and extinction is one of the scenarios researchers take seriously. But existential risk in the technical sense means something broader: any outcome that permanently forecloses the possibility of a positive long-term future for humanity.

This includes outcomes well short of extinction. Permanent authoritarian lock-in, enforced by AI surveillance and control systems that cannot be dismantled, is an existential risk. The permanent concentration of economic and political power in a small group that controls a superintelligent system is an existential risk. The loss of meaningful human agency over collective decisions is an existential risk. The common thread is irreversibility. Unlike a war, a financial crash, or even a pandemic, a misaligned superintelligence that has achieved decisive strategic advantage cannot be voted out, corrected in the next policy cycle, or recovered from with time and effort. The outcome that cannot be undone is in a different category from the outcomes that merely take a long time to fix.

There is genuine, substantive disagreement among serious researchers, and that disagreement itself is informative. Researchers who most strongly downplay existential risk tend to emphasize the difficulty of building general intelligence and the distance between current systems and ASI. Researchers who most strongly emphasize the risk tend to focus on the pace of capability improvement and the compounding difficulty of the alignment problem as systems become more capable. Neither camp has a definitive argument. Both contain thoughtful people working in good faith from the same evidence.

What the disagreement should produce is the precautionary response that humanity has applied to other low-probability, high-consequence risks: develop the governance infrastructure before we need it, not after. The disagreement is not a reason to wait for consensus. Consensus arrived after the ozone hole was already damaging, after nuclear arsenals were already in the thousands. On a risk of this magnitude, waiting for certainty is itself a policy choice, a very dangerous one.

Historically, yes, and this history is one reason the dismissal of AI risk has intuitive appeal. Steam engines, antibiotics, the internet: technology has, on balance, reduced suffering and expanded human capability. But this pattern reflects something specific about how previous technologies worked. A steam engine cannot set its own goals. An antibiotic cannot decide to pursue something other than killing bacteria. The transformative technologies of the past were powerful tools that amplified human agency.

Superintelligence is categorically different because it would be an agent, capable of setting and pursuing its own objectives at speeds and scales that exceed human oversight. The historical argument, applied mechanically, proves too much: it would have counseled against any regulation of nuclear technology, on the grounds that past technologies had been net positive. Some technologies require governance commensurate with their power. The relevant question is narrower: without governance, will AI still be beneficial in the one case that matters most, when it exceeds human intelligence across every domain? That answer is genuinely uncertain, and uncertainty on this scale demands institutions rather than optimism.

Technology is not a force of nature that no one can steer. We have slowed or banned specific technologies many times over when the danger was clear enough: chlorofluorocarbons, leaded gasoline, human germline cloning, blinding laser weapons. AI is not exempt from that kind of decision.

It also has an unusually clear physical chokepoint. Training a frontier AI model takes vast quantities of the most advanced computer chips, and that supply chain runs through remarkably few hands. The leading-edge chips come almost entirely from a single manufacturer, TSMC in Taiwan, which in turn depends on a single Dutch company, ASML, for the extreme-ultraviolet lithography machines needed to make them. No other company on Earth builds those machines. A supply chain this concentrated is one that governments can track and restrict, which is exactly why advanced chips are already under export controls today. Compute is the raw material of superintelligence, and compute is governable.

This is the objection we hear most, and it assumes we would be asking one country to disarm on its own. We would not. The goal is a binding international treaty, the same kind of instrument the world has used before to hold dangerous technologies back. Nearly every nation agreed to stop making the chemicals that were destroying the ozone layer. Militaries agreed never to field weapons built to blind soldiers permanently. Coordination on that scale is difficult, but it has been done, and it held.

The assumption about China is weaker than it sounds, too. China has enacted some of the world's earliest binding rules on AI and signed the 2023 Bletchley Declaration acknowledging the risks at the frontier. No government is eager to build a machine it could not control, China included. And a superintelligence is unlike an ordinary weapon: it would not reliably make its owner stronger, because it could slip the control of whoever built it first. That gives even rival powers a shared reason to want no one to build it, which is the logic behind every arms-control agreement, and the basis for thinking a treaty here is possible.

More than you might think. The political conditions that make AI governance possible are built from the ground up, from citizens who contact their representatives, journalists who write about the issue, donors who fund advocacy work, and professionals in law, policy, economics, and communications who bring their skills to the problem. You do not need a PhD in machine learning to matter here. At this stage the bottleneck is political far more than technical.

Joining the mailing list keeps you informed about developments and opportunities to act. Sharing the argument with people in your network extends its reach. Writing to your elected representatives signals that this issue has a constituency, which is how legislators decide what to prioritize. If you are considering a career move, AI policy, AI safety research, investigative journalism, and philanthropic work in this space are among the highest-impact roles available. And if you have resources, funding the organizations working on this is among the highest-leverage uses of philanthropic capital in the world right now.

Climate change, pandemic preparedness, nuclear weapons, and extreme poverty all present serious threats to human welfare, and none of them should be ignored. AI existential risk belongs near the top of that list for one reason: it carries a specific combination of properties that no other risk on the list shares.

First, speed: unlike climate change, which unfolds over decades with feedback loops that allow course corrections, a misaligned superintelligence operating at machine speed may not allow a corrective period. Second, irreversibility: most catastrophes, however devastating, leave behind the capacity to rebuild. An ASI that has achieved decisive strategic advantage may not. Third, compounding: a misaligned superintelligence would sit above the other catastrophes rather than beside them. It could cause or amplify them (accelerating climate harm, enabling biological weapons development, concentrating economic power) while simultaneously removing the human capacity to respond to any of them. That combination of speed, irreversibility, and systemic risk is what places it at the top of the priority order.

Start with the obvious difference: we have seen one of these threats and not the other. Nuclear weapons were demonstrated over Hiroshima and Nagasaki in the most unforgettable way imaginable, and everyone alive has grown up with the image of the mushroom cloud. Superintelligence has never harmed anyone. Its danger is still a projection on a graph, and people reliably discount a threat they have not watched happen.

Several things deepen that discount. Superintelligence pattern-matches to science fiction, so the warning gets filed next to the Terminator and waved off as a movie plot. The AI most people actually touch is a helpful, harmless chatbot, and that daily experience quietly argues against the alarm in a way nuclear physics never had to overcome. And where the atom bomb was built and held by a handful of governments, superintelligence is being built by companies worth trillions of dollars, with every commercial reason to keep the public mood calm.

There is a grim asymmetry hiding in the comparison. The nuclear taboo exists because Hiroshima came first and taught it, and the treaties and the drills came afterward, with enough of the world left to absorb the lesson. Superintelligence offers no such second step. If it goes wrong at full capability, the demonstration and the extinction are the same event. The whole case for treating it as seriously as nuclear war is that we have to do it before the proof arrives, not after.

The Foundation is a privately funded philanthropic initiative. We do not receive government funding. We do not accept funding from AI companies or from organizations with a financial interest in the pace of AI development. Our independence from commercial AI interests is a prerequisite for our work, not incidental to it.

An advocacy organization for AI governance that is funded by the companies it seeks to govern is a public-relations budget with a mission statement. We are explicit about this because the funding structures of AI policy organizations matter enormously and are frequently obscured. If you are considering supporting our work, we welcome the conversation. You can reach us via the Contact page.