Experts put a number on AI catastrophe: reading the 272-expert Delphi study

Risk governance runs on numbers, and for most of the debate about catastrophic AI risk the numbers have been missing — positions staked out with adjectives ("existential," "overblown," "unprecedented") and an argument largely about vocabulary. A Delphi study published this June changes the register: it asked 272 AI experts, across three structured rounds, to put probabilities on 24 distinct AI risk domains over the next five years.

The headline is uncomfortable. Under a business-as-usual trajectory — organisations and governments continuing existing practice without additional AI-specific mitigations — the experts judged 18 of the 24 risk domains to carry a greater than 10% probability of a catastrophic outcome by 2030. "Catastrophic" here is not rhetorical: it means more than one million deaths, more than 100 billion US dollars in financial loss, or civilisational-scale intangible damage such as democratic collapse or authoritarian lock-in. A better-than-one-in-ten chance of an outcome that severe, across three-quarters of the risks examined, is what a standard framework would label intolerable.

This post is our reading of the study from a financial-institution risk desk, and it is also the hub for a four-part series that takes each risk cluster in turn — with the DF Analytics response spelled out for each (links at the end). It is not investment or legal advice, and the figures below are the experts' elicited probabilities, not DF Analytics forecasts. But the structure it exposes — who is exposed, who is responsible, and the gap between the two — maps almost exactly onto the governance conversations we have with CROs and boards.

10-minute read · Updated July 14, 2026

Key takeaways

• A three-round Delphi study of 272 experts from 37 countries prioritised 24 AI risk domains over a 2025–2030 horizon; under business-as-usual, 18 of the 24 carry a greater than 10% chance of catastrophic harm.
• Mitigation does not make the problem go away: even with pragmatic, cost-effective measures in place, all 24 risks retain at least a 5% catastrophic probability, and five stay above 10%.
• The exposure is badly misaligned with the control: AI users and the general public are rated the most vulnerable, while the actors best placed to prevent harm — foundation-model developers and governance bodies — sit upstream and face competitive pressure against precaution.
• Information, national security, and finance are among the most vulnerable sectors, which puts this study squarely on the agenda of any institution that already treats model risk as board-level.

What the study actually measured

The method matters, because it is what turns opinion into something you can prioritise against. A Delphi study runs expert judgement through several anonymous rounds, feeding responses back so that estimates converge or, just as usefully, so that persistent disagreement becomes visible. This one ran three rounds in late 2025 with 272 experts from academia, industry, government, and civil society across 37 countries. It is not a global cross-section — 79% of participants came from Europe or North America — a skew worth holding in mind.

Experts rated each of 24 risk domains on four dimensions: the expected severity and likelihood of harm, the vulnerability of different actors and sectors, and the responsibility each actor bears for mitigation. Crucially, they did so under two scenarios: a business-as-usual (BAU) path, and a pragmatic mitigations path in which organisations and governments make cost-effective, realistic efforts to address AI risk. The gap between those two scenarios is the single most useful output of the exercise — an estimate of how much protection is actually on the table.

The intolerable baseline

Under BAU, eighteen of the twenty-four domains clear the 10% catastrophic threshold. The five most severe, ranked by their business-as-usual probability of catastrophic harm, were:

• Dangerous capabilities — 21.5%. AI systems developing or accessing capabilities that enable mass harm through deception, weapons development, cyber-offence, or self-proliferation.
• Weapons & cyberattacks — 21.0%. The deliberate use of AI to build cyber weapons, lethal autonomous weapons, or CBRNE (chemical, biological, radiological, nuclear, explosive) capability.
• Power centralization — 18.0%. AI-driven concentration of power and resources producing extreme inequality and an unfair distribution of benefits.
• Competitive dynamics — 16.6%. State or corporate "race" behaviour that prioritises strategic or economic advantage over safety and ships unsafe systems.
• False information — 12.8%. The inadvertent generation of deceptive content that undermines human autonomy and causes material harm.

Two things stand out. The top of the list is dominated by deliberate misuse and structural risks rather than accidental model failure — the failure modes a model-level safety fix does not touch. And 21.5% on the leading risk is not a tail estimate; it is a base-rate a treasurer would recognise as needing a hedge.

Operationalised: the risks topping this list are precisely the ones our Regulatory Crawler tracks through supervisory and enforcement channels, because they are where rulemaking is heading. When a risk is both high-probability and structurally driven, expect the regulatory response to be structural too.

The mitigation reality check

The natural reply to a frightening baseline is "yes, but we will do something about it." The study's answer is sobering. Under the pragmatic-mitigations scenario — realistic, cost-effective effort, not a fantasy of perfect global coordination — all 24 risks still retain at least a 5% probability of catastrophic harm by 2030. Mitigation moves the numbers; it does not clear the threshold.

Five risks stay above 10% even with pragmatic mitigations in place:

• Dangerous capabilities — 12%.
• Weapons & cyberattacks — 12%.
• Environmental harm — 12%.
• Power centralization — 11%.
• Inequality & unemployment — 11%.

Notice how the list changes shape. Environmental harm and inequality climb into the residual top five — not because model-level safety cannot reduce them, but because it barely addresses them at all. These are consequences of how AI is deployed across an economy, and they persist because the tools most organisations reach for do not bite on them.

Operationalised: a 5% residual floor across every risk is a governance instruction, not a footnote. The correct posture is not "solve and move on" but "reduce, monitor continuously, and keep documenting."

The asymmetry that should worry every board

The study's most consequential finding is not a probability at all — it is a structural mismatch. When experts rated who is vulnerable against who is responsible, the two did not line up.

On vulnerability, the verdict was near-unanimous: AI users and affected stakeholders — the general public — were rated the most vulnerable actors across nearly all 24 risks, drawing consensus ratings of "highly to extremely vulnerable." At the other end, AI infrastructure providers — the entities supplying compute, cloud, and data — were judged least vulnerable, largely insulated from direct AI risk. On responsibility, the experts pointed upstream: general-purpose AI developers (the builders of foundation models) and AI governance actors (governments, regulators, and standards bodies) were assigned primary responsibility for mitigation.

Put those together and you get a textbook moral hazard. The people who bear the harm cannot prevent it — a downstream deployer has no lever on model weights, alignment, or release decisions. The actors who can prevent it sit upstream and face intense pressure to move fast, so the cost of precaution lands on them while the cost of failure is externalised onto a highly vulnerable public. That is the same "race dynamics over safety" that shows up as its own top-five risk. Left alone, this structure does not self-correct; it rewards whoever precautions least.

Where finance sits in the exposure map

For our readership, the sector findings are where the study stops being abstract. Experts rated information and national security as the most vulnerable sectors overall — the domains most exposed to disinformation, privacy loss, cyberattack, and cascading geopolitical failure. Immediately behind them sit two sectors that need no introduction here. Finance and insurance were rated highly vulnerable to direct financial attack — fraud, scams, market manipulation — and to AI security vulnerabilities; these are not hypotheticals but extensions of threats risk desks already manage, now with a capability multiplier attached. Health care was rated highly vulnerable to privacy loss, discrimination, and unsafe over-reliance, where a failure converts immediately into human cost.

The lesson is not that finance is uniquely doomed, but that it is uniquely in the blast radius of several risks at once — and that argues for treating AI risk as an enterprise concern, not a line item owned by one team.

What the experts themselves were most worried about

The study also asked experts to simply name their top three most concerning domains. The ranking that emerged is revealing precisely because it does not match the severity table:

• Weapons & cyberattacks — 26.8% of experts.
• Power centralization — 23.5%.
• Disinformation & influence — 22.1%.
• Loss of consensus reality — 21.6%.
• Dangerous capabilities — 21.6%.

The surprise is "loss of consensus reality" — the erosion of a shared picture of the world — appearing this high. It reflects a specific worry: AI-generated filter bubbles and personalised misinformation eating away at the shared reality that democratic societies and functioning markets both depend on. A market is itself a consensus mechanism; if the informational substrate underneath it fractures, price discovery and public trust fracture with it.

What it means for governance

The policy reading the study invites is one we have argued from a different direction for a while. Relying on frontier developers to govern themselves is structurally insufficient — not for lack of good intentions, but because race dynamics and tragedy-of-the-commons pressures make voluntary restraint individually irrational.

Closing the gap, the study implies, requires hard instruments that force upstream actors to internalise the costs they currently externalise: strict liability, mandatory insurance regimes, transparency mandates, and monitoring. These make precaution the rational choice rather than the costly one — and they sit in the toolkit of the governance actors the experts hold responsible.

Two caveats keep this from being tidy. Some residual risks — extreme power centralization, environmental harm, widespread inequality — cannot be closed by any model-level safeguard; they call for competition policy, labour protection, and international coordination, well outside an AI safety team's perimeter. And none of this argues against firm-level diligence — only that it is necessary, not sufficient. The operational takeaway is the one we keep returning to: inventory your AI exposure honestly, document continuously from runtime state rather than at audit cycles, and build governance to survive the discovery that the mitigation window is narrower than you assumed.

Frequently asked questions

What counts as a "catastrophic" outcome in the study?

The study defines catastrophic harm as an outcome involving more than one million human deaths, more than 100 billion US dollars in financial loss, or civilisational-scale intangible impacts such as global democratic collapse or authoritarian lock-in. The much-cited "18 of 24 risks above 10%" figure refers to the probability of an outcome at that severity by 2030 under business-as-usual.

Do mitigations solve the problem?

No — they reduce it. Under the study's pragmatic-mitigations scenario, all 24 risks still retain at least a 5% catastrophic probability by 2030, and five (dangerous capabilities, weapons & cyberattacks, environmental harm, power centralization, and inequality & unemployment) stay above 10%. The right posture is continuous reduction and monitoring, not one-time resolution.

Why does the study matter for financial institutions?

Experts rated finance and insurance among the most vulnerable sectors — to fraud, scams, market manipulation, and AI security vulnerabilities — alongside information and national security. Because a financial institution sits in the blast radius of several of these risks at once, the study argues for treating AI risk as an enterprise, board-level concern.

What is the "moral hazard" the study identifies?

The actors most vulnerable to AI harms (users and the public) are not the ones who can prevent them. Prevention sits upstream with foundation-model developers and governance bodies, who face competitive pressure against precaution — so the cost of caution falls on them while the cost of failure lands on the vulnerable. The study argues this gap will not close without hard instruments such as strict liability and mandatory insurance.

The four deep-dives in this series

This post is the summary; each risk cluster gets its own treatment, with the DF Analytics response spelled out.

• When capability outruns control: monitoring AI's most dangerous risks — dangerous capabilities and AI-enabled weapons/cyberattacks, and the agent-monitoring layer that surfaces them.
• Structural AI risks: the harms no model-level fix can touch — power centralization, competitive dynamics, inequality and environmental harm, and how to measure exposure to them.
• The moral hazard at the centre of AI risk — why voluntary self-governance fails, and the hard instruments that close the gap.
• Finance in the blast radius — the three financial attack surfaces and the engines that watch each.

Related reading

• EU AI Act Digital Omnibus: the high-risk delay explained
• Governance by default: 9 principles for AI in finance
• Aspects Q2 2026 — Tech & AI infrastructure

External references

• Prioritization of Risks from Artificial Intelligence: A Delphi Study of 272 International Experts (arXiv)
• MIT FutureTech — study publication page
• MIT Sloan — International AI experts warn of potentially catastrophic risks from AI

About the author — Compliance Liaison — Deep Finance Analytics. The Compliance Liaison tracks supervisory developments through Regulatory Crawler and translates them into operational guidance. See the Insights hub for the full archive, or book a discovery call to discuss this post with the team.