There are many points here that I agree with. For example, I agree that citing a precise probability suggests that there is some kind of method used to generate it, and that it has sufficient robustness so that it wouldn't easily change in response to new information.
But fundamentally I think that the plausibility of the main point trades on a conflation between two distinct claims:
1. That a claim (e.g. "doom") is supported by little evidence and has high uncertainty/robustness
2. That the claim is unlikely to be true
These can come apart, as I think you would agree. You can be very certain that something is unlikely to be true, and conversely be very uncertain about whether something is true while having as your best guess that it is pretty likely.
You say that policy should not be based on uncertain claims or insufficient evidence. On its face, that seems plausible. But that's partly because we seem to assume that it's unlikely barring robust evidence to the contrary.
This seems true for the example given. It is that a claim of an 80% probability of an alien invasion would be silly. I agree. But the obvious explanation for that is not uncertainty or lack of evidence, but that we have pretty good initial evidence that it's much lower!
For example, we haven't had any (obvious) invasions for the last few millennia, so it would be surprising if they happened now, unless we think that we're doing something right now that is particularly likely to attract aliens. Indeed, we would need very strong evidence that an alien invasion was incoming to change policy on this matter.
By contrast, mere uncertainty--having poor grounds for a best guess in either direction--does not clearly direct action either way. To see this, note that unlike probabilities, uncertainty is symmetric between a claim and its negation. If you have no idea about whether doom will occur, then you have no idea whether not-doom will occur. The proposals for decision-procedures that I am familiar with in such situations are either to apply a precautionary principle, which would presumably endorse strong intervention, or to do our best with the evidence we have and form a probability.
So the only way it seems to say that policy should not be freaking out about AI risk is to argue that the situation is not just uncertain, but that the probability is low.
But you don't get that for free! You only do that if you indeed assume some baseline prior that is sufficiently low that our current evidence is insufficient (e.g. "no technology has killed us all yet"). And that might be very plausible, but you should be explicit that this is an assumption of the argument. Mere appeal to uncertainty will not get you there, and it seems unfair to present the argument as if it did.
At this point, however, the authors are in the same game they're criticizing, trying to find reasons to take seriously or not take seriously AI risk. And they, like the rest of us who do not default to precaution, have to do their best.
While many of the concerns they raise about the attempts to predict AI risk are great, and I agree with. For example, expert surveys are pretty uninformative when nobody is an expert. But they're also underselling the arguments for AI risk as something merely "speculative"--a term that indicates mere uncertainty but strongly implicates low probability.
In fact, there are plenty of arguments that try to take this seriously while arguing against short-term doom risk, recently "Where is the intelligence explosion?" by Ramez Naam. Obvious examples on the other side include, even more recently, Chan et al. "What if automating AI R&D triggers an intelligence explosion?", published today.
Finally, this line had me slightly gasping:
"We have no objection to AI x-risk forecasting as an academic activity, and forecasts may be helpful to companies and other private decision makers. We only question its use in the context of public policy."
Presumably, either the probabilities are not helpful for decision-making in companies, in which case we should be objecting to their use there (or at least not buy their stock). Or they are useful, in which case we are asking public policy to leave epistemic money on the table.
It feels like this post conflates "it's hard to estimate x-risk" with "x-risk is low".
You write, "[Policymakers] should reject the kind of policies that might seem compelling if we view x-risk as urgent and serious, notably: restricting AI development."
If we shouldn't do things that make sense if x-risk is high, then it sounds like we should do things that make sense if x-risk is low.
But you don't give any evidence that x-risk is low, instead you argue it's hard to estimate. These are two different claims!
1. I have often found that offering a specific numerical value (in this case, probability) conferred greater credibility. When I worked for a large bank and was arguing for technology budget, I found that if I said we needed, say, "$23.4 million" people assumed our budget forecasts were highly accurate (when they could have been precise, but still inaccurate), while $23 million was taken as more of an estimate.
2. Is there really "no reference class" to support the Inductive method? While AI systems have not yet caused a catastrophic or extinction event, there have been numerous instances of AI systems taking significant actions with negative consequences (e.g., attempting to change other systems and databases). Can this be analogized to "observing thousands of small [asteroid] impacts" being useful in estimating the likelihood of a large (extinction-level) asteroid impact?
Speculating on such risks when the mechanisms are wholly unclear say much more about the speculator than the number. It reflects a personal position more than any objective analysis. We need to focus more on known risks, right in front of us now. One I've been thinking about is cyber security. There's pretty strong evidence that the AI labs have built, in the name of defense, the best cyber hacking tools on the planet. We also can't ignore how AI is affecting education, human interaction and cognition in general, though I think that's going to be more of an adaptation process. We clearly have the power to change the p(doom) through action. We need to start now.
Worth reposting. Policy built on unreliable probability estimates risks swinging with the mood of the moment. Focusing on concrete, observable harms and solid incident reporting gives a much more stable basis for decisions.
"Claiming to have a high p(doom) (one’s estimate of the probability of AI doom) seems to have become a way to signal one’s identity and commitment to the cause."
Reminds me of what I also wrote 2 years ago "Publicly providing a value of p(doom) has become somewhat of a social phenomenon among AI researchers and other interested parties."
We argue many of the same points. I have suggested something more meaningful might be "risk = unpredictability * capability". At least you can somewhat benchmark unpredictability and capability, but of course that still doesn't allow you to calculate any specific risk.
Ultimately p(doom) is useless for assisting policy and likely harmful. But if we cared about something more imminent and relevant, we would probably be talking about p(dystopia) and AI being used to spy on everyone, warp public opinion, use in military weapons etc.
Cynical read, but the order of events does fit. What I'd want to know is whether those rules actually cost rivals more, and I haven't seen anyone check that.
It would be cynical if Dario's actions matched his words.
Claiming his philosophical approach centers on pacing growth for safety concerns and that he isnt a doomer in essays then appearing on prime time news talking about the dangers of his models is fine but releasing that model a week later?? Did he change his philosophy in a week or capitalise on the hype?
Blog post about proposing FAA regulation - the FAA takes 5 to 9 years to approve a new airplane. This would be a disaster if that model was adapted and ported for ai.
Cost to competitors? This is in lost revenue (if FAA style regulation took place). Also costing open models their entire business trying to have them banned. Another cost is the churn from openai and grok to Anthropic when he does his media winge/new model song and dance. (Look at X for cancelling gpt for Claude posts each time - its non trivial)
It seems like you argue that it is impossible to rigorously forecast AI x-risks, and thus AI x-risk estimates should not inform policy decision-making. Yet you conclude with "As we’ll argue in a future essay in this series, not only are such policies unnecessary, they are likely to increase x-risk." This is a policy prescription based on an AI x-risk forecast, which seems to go against the whole point of your essay. Do you mind clarifying?
Yeah one thing I came to realize working at Metaculus is that EAs actually have no idea of what Bayesianism is beyond the formalisms, or of why it would be desirable to be a Bayesian. They have a cargo cult belief in math and judge non-mathematical questions by looking at the mathematical credentials of the people arguing.
The Bostrom passage is the one I keep coming back to. Every estimate carries an error band. The band covers the small uncertainty: how many people were asked, how precisely something was measured. It does not cover the big one, whether the assumptions underneath are right. With p(doom) there is only the big one.
In construction, every quote has a line under the price: assumed stable ground, no contamination. That line is the real offer. The rest is arithmetic.
When the forecasts in the tournament you describe sit a factor of a hundred apart, and hardly anyone moves even with money on the table, that is an argument about the ground dressed up as an argument about the price. Averaging two quotes on different ground gives you the price of a house that does not exist.
Policy asks a different question anyway: what happens if we do X? None of the risk numbers you quote carries an "if we". Even with a perfect line underneath, a number like that would not say which measure lowers it.
That kind of knowledge is rarely found by asking. It is acquired by acting and watching. Small children learn causes this way, by pushing things and seeing what moves. The naturalistic evaluations you call for, measuring how work actually changes with AI, belong here. They are more than better benchmarks, because each one is a small intervention whose result comes back.
For the result to come back, someone has to stand where it shows up and be able to read it. On a building site, the caretaker knows after the first winter which corner stays cold. Whether that reading travels back to the people who planned the building decides whether anyone learns from it. After the handover, it usually does not.
You end by asking governments to respond better to new evidence. For AI, the question underneath may be this one: where would the first sign show up, and is anyone standing there who could read it?
> The XPT is a good example of transparency, as is this paper (though it is not about x-risk).
By the say, the linked paper argues that "Transformative AGI by 2043 is <1% likely." This may have been a reasonable-sounding estimate when it was published in 2023, but is pretty bad by today's lights.
The allure of a hero defeating an evil dragon could be too appealing. We should not neglect the potholes that can cause everyday harms. A wider big tent approach is sorely needed.
Car crashes happen constantly, so an insurer's bad guesses show up as red ink and get corrected. Human extinction has one trial, and as the authors say, the evidence lands too late to act on. A 10% figure is a first draft nobody can mark — which is why it looks rigorous and still leaves a policymaker unable to justify the cost to the people whose models get restricted.
There are many points here that I agree with. For example, I agree that citing a precise probability suggests that there is some kind of method used to generate it, and that it has sufficient robustness so that it wouldn't easily change in response to new information.
But fundamentally I think that the plausibility of the main point trades on a conflation between two distinct claims:
1. That a claim (e.g. "doom") is supported by little evidence and has high uncertainty/robustness
2. That the claim is unlikely to be true
These can come apart, as I think you would agree. You can be very certain that something is unlikely to be true, and conversely be very uncertain about whether something is true while having as your best guess that it is pretty likely.
You say that policy should not be based on uncertain claims or insufficient evidence. On its face, that seems plausible. But that's partly because we seem to assume that it's unlikely barring robust evidence to the contrary.
This seems true for the example given. It is that a claim of an 80% probability of an alien invasion would be silly. I agree. But the obvious explanation for that is not uncertainty or lack of evidence, but that we have pretty good initial evidence that it's much lower!
For example, we haven't had any (obvious) invasions for the last few millennia, so it would be surprising if they happened now, unless we think that we're doing something right now that is particularly likely to attract aliens. Indeed, we would need very strong evidence that an alien invasion was incoming to change policy on this matter.
By contrast, mere uncertainty--having poor grounds for a best guess in either direction--does not clearly direct action either way. To see this, note that unlike probabilities, uncertainty is symmetric between a claim and its negation. If you have no idea about whether doom will occur, then you have no idea whether not-doom will occur. The proposals for decision-procedures that I am familiar with in such situations are either to apply a precautionary principle, which would presumably endorse strong intervention, or to do our best with the evidence we have and form a probability.
So the only way it seems to say that policy should not be freaking out about AI risk is to argue that the situation is not just uncertain, but that the probability is low.
But you don't get that for free! You only do that if you indeed assume some baseline prior that is sufficiently low that our current evidence is insufficient (e.g. "no technology has killed us all yet"). And that might be very plausible, but you should be explicit that this is an assumption of the argument. Mere appeal to uncertainty will not get you there, and it seems unfair to present the argument as if it did.
At this point, however, the authors are in the same game they're criticizing, trying to find reasons to take seriously or not take seriously AI risk. And they, like the rest of us who do not default to precaution, have to do their best.
While many of the concerns they raise about the attempts to predict AI risk are great, and I agree with. For example, expert surveys are pretty uninformative when nobody is an expert. But they're also underselling the arguments for AI risk as something merely "speculative"--a term that indicates mere uncertainty but strongly implicates low probability.
In fact, there are plenty of arguments that try to take this seriously while arguing against short-term doom risk, recently "Where is the intelligence explosion?" by Ramez Naam. Obvious examples on the other side include, even more recently, Chan et al. "What if automating AI R&D triggers an intelligence explosion?", published today.
Finally, this line had me slightly gasping:
"We have no objection to AI x-risk forecasting as an academic activity, and forecasts may be helpful to companies and other private decision makers. We only question its use in the context of public policy."
Presumably, either the probabilities are not helpful for decision-making in companies, in which case we should be objecting to their use there (or at least not buy their stock). Or they are useful, in which case we are asking public policy to leave epistemic money on the table.
It feels like this post conflates "it's hard to estimate x-risk" with "x-risk is low".
You write, "[Policymakers] should reject the kind of policies that might seem compelling if we view x-risk as urgent and serious, notably: restricting AI development."
If we shouldn't do things that make sense if x-risk is high, then it sounds like we should do things that make sense if x-risk is low.
But you don't give any evidence that x-risk is low, instead you argue it's hard to estimate. These are two different claims!
A comment and a question:
1. I have often found that offering a specific numerical value (in this case, probability) conferred greater credibility. When I worked for a large bank and was arguing for technology budget, I found that if I said we needed, say, "$23.4 million" people assumed our budget forecasts were highly accurate (when they could have been precise, but still inaccurate), while $23 million was taken as more of an estimate.
2. Is there really "no reference class" to support the Inductive method? While AI systems have not yet caused a catastrophic or extinction event, there have been numerous instances of AI systems taking significant actions with negative consequences (e.g., attempting to change other systems and databases). Can this be analogized to "observing thousands of small [asteroid] impacts" being useful in estimating the likelihood of a large (extinction-level) asteroid impact?
Speculating on such risks when the mechanisms are wholly unclear say much more about the speculator than the number. It reflects a personal position more than any objective analysis. We need to focus more on known risks, right in front of us now. One I've been thinking about is cyber security. There's pretty strong evidence that the AI labs have built, in the name of defense, the best cyber hacking tools on the planet. We also can't ignore how AI is affecting education, human interaction and cognition in general, though I think that's going to be more of an adaptation process. We clearly have the power to change the p(doom) through action. We need to start now.
Worth reposting. Policy built on unreliable probability estimates risks swinging with the mood of the moment. Focusing on concrete, observable harms and solid incident reporting gives a much more stable basis for decisions.
11 oct 2024 - Dario publishes an essay opening with how bad ai risks can be
22 oct 2024 - Dario releases Sonnet 3.5
17 September 2025 - Dario at AI summet in DC "25% chance things go really badly"
29 September 2025 - Dario releases Sonnet 4.5
26 june 2026 - Altman says he can't safely deploy the latest model as it may lose control
September 2026 - Altman releases gpt 5, 5.6 and luna/sol models
"Make me virtuous but not just yet"
- St Augustine
Doom is not cautionary. It is marketing.
"OUR MODELS ARE WAY TO POWERFUL.... (you can try it for $300/month next week)"
"Claiming to have a high p(doom) (one’s estimate of the probability of AI doom) seems to have become a way to signal one’s identity and commitment to the cause."
Reminds me of what I also wrote 2 years ago "Publicly providing a value of p(doom) has become somewhat of a social phenomenon among AI researchers and other interested parties."
We argue many of the same points. I have suggested something more meaningful might be "risk = unpredictability * capability". At least you can somewhat benchmark unpredictability and capability, but of course that still doesn't allow you to calculate any specific risk.
Ultimately p(doom) is useless for assisting policy and likely harmful. But if we cared about something more imminent and relevant, we would probably be talking about p(dystopia) and AI being used to spy on everyone, warp public opinion, use in military weapons etc.
My similar and supporting argument. FYI.
https://www.mindprison.cc/p/pdoom-the-useless-ai-predictor
Putting a number on it makes the guess feel checked when it isn't. A forecast nobody can score later is just a feeling in a nicer format.
- release latest model
- warns of ai danger
- causes all social media and news to discuss ai danger
- propose rules that will raise rival frontier companies costs
- name a partner as the watchdog for all companies
- release more powerful model than what we were warned about 12 days before
- include extinction risk is S-1 for NYSE IPO - must be true
- extinction risk disclosure caps liability
Goal = market cap go up
Quantify net worth while we write articles about how to quantify risk.
Three card Monte.
Cynical read, but the order of events does fit. What I'd want to know is whether those rules actually cost rivals more, and I haven't seen anyone check that.
It would be cynical if Dario's actions matched his words.
Claiming his philosophical approach centers on pacing growth for safety concerns and that he isnt a doomer in essays then appearing on prime time news talking about the dangers of his models is fine but releasing that model a week later?? Did he change his philosophy in a week or capitalise on the hype?
Blog post about proposing FAA regulation - the FAA takes 5 to 9 years to approve a new airplane. This would be a disaster if that model was adapted and ported for ai.
Cost to competitors? This is in lost revenue (if FAA style regulation took place). Also costing open models their entire business trying to have them banned. Another cost is the churn from openai and grok to Anthropic when he does his media winge/new model song and dance. (Look at X for cancelling gpt for Claude posts each time - its non trivial)
Lost revenue and churn are the parts I'd want someone to count. Right now it hangs together as a story, and that's different from being checked.
It seems like you argue that it is impossible to rigorously forecast AI x-risks, and thus AI x-risk estimates should not inform policy decision-making. Yet you conclude with "As we’ll argue in a future essay in this series, not only are such policies unnecessary, they are likely to increase x-risk." This is a policy prescription based on an AI x-risk forecast, which seems to go against the whole point of your essay. Do you mind clarifying?
Understanding Bayesian analysis as a mathematical expression of inductive reasoning has helped me to better judge its applicability.
Also was the final blow to any remaining interest in AE/Rationalism.
Yeah one thing I came to realize working at Metaculus is that EAs actually have no idea of what Bayesianism is beyond the formalisms, or of why it would be desirable to be a Bayesian. They have a cargo cult belief in math and judge non-mathematical questions by looking at the mathematical credentials of the people arguing.
"Cargo cult" is exactly right. To quote a famous blogger I once revered, "P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary."
The Pentapartite Panel, required reading for the modern human.
AI debating the consequences of AI.
The Bostrom passage is the one I keep coming back to. Every estimate carries an error band. The band covers the small uncertainty: how many people were asked, how precisely something was measured. It does not cover the big one, whether the assumptions underneath are right. With p(doom) there is only the big one.
In construction, every quote has a line under the price: assumed stable ground, no contamination. That line is the real offer. The rest is arithmetic.
When the forecasts in the tournament you describe sit a factor of a hundred apart, and hardly anyone moves even with money on the table, that is an argument about the ground dressed up as an argument about the price. Averaging two quotes on different ground gives you the price of a house that does not exist.
Policy asks a different question anyway: what happens if we do X? None of the risk numbers you quote carries an "if we". Even with a perfect line underneath, a number like that would not say which measure lowers it.
That kind of knowledge is rarely found by asking. It is acquired by acting and watching. Small children learn causes this way, by pushing things and seeing what moves. The naturalistic evaluations you call for, measuring how work actually changes with AI, belong here. They are more than better benchmarks, because each one is a small intervention whose result comes back.
For the result to come back, someone has to stand where it shows up and be able to read it. On a building site, the caretaker knows after the first winter which corner stays cold. Whether that reading travels back to the people who planned the building decides whether anyone learns from it. After the handover, it usually does not.
You end by asking governments to respond better to new evidence. For AI, the question underneath may be this one: where would the first sign show up, and is anyone standing there who could read it?
> The XPT is a good example of transparency, as is this paper (though it is not about x-risk).
By the say, the linked paper argues that "Transformative AGI by 2043 is <1% likely." This may have been a reasonable-sounding estimate when it was published in 2023, but is pretty bad by today's lights.
The allure of a hero defeating an evil dragon could be too appealing. We should not neglect the potholes that can cause everyday harms. A wider big tent approach is sorely needed.
Car crashes happen constantly, so an insurer's bad guesses show up as red ink and get corrected. Human extinction has one trial, and as the authors say, the evidence lands too late to act on. A 10% figure is a first draft nobody can mark — which is why it looks rigorous and still leaves a policymaker unable to justify the cost to the people whose models get restricted.
This piece is published on the same day as the Chan et al., paper "What if automating AI R&D
triggers an intelligence
explosion?"
Is that a paper that you agree or disagree with?