Thanks for the additional context! I think I understand your views better now and I appreciate your feedback.
Just speaking for myself here, I think I can identify some key cruxes between us. I’ll take them one by one:
I think the impact of most actions here is basically chaotic.
I disagree with this. I think it’s better if people have a better understanding of the key issues raised by the emergence of AGI. We don’t have all the answers, but we’ve thought about these issues a lot and have ideas about what kinds of problems are most pressing to address and what some potential solutions are. Communicating these ideas more broadly and to people who may be able to help is just better in expectation than failing to do so (all else equal), even though, as with any problem, you can’t be sure you’re making things better, and there’s some chance you make things worse.
I also think “make the world better in meaningful ways in our usual cause areas before AGI is here” probably helps in many worlds, due to things like AI maybe trying to copy our values, or AI could be controlled by the UN or whatever and it’s good to get as much moral progress in there as possible beforehand, or just updates on the amount of morally aligned training data being used.
I don’t think I agree with this. I think the value of doing work in areas like global health or helping animals is largely in the direct impact of these actions, rather than any impact on what it means for the arrival of AGI. I don’t think even if, in an overwhelming success, we cut malaria deaths in half next year, that will meaningfully increase the likelihood that AGI is aligned or that the training data reflects a better morality. It’s more likely that directly trying to work to create beneficial AI will have these effects. Of course, the case for saving lives from malaria is still strong, because people’s lives matter and are worth saving.
I think that more serious consideration of the Existential Risk Persuasion Tournament leads one to conclude that wildly transformational outcomes just aren’t that likely in the short/medium term.
Recall that the XPT is from 2022, so there’s a lot that’s happened since. Even still, here’s what Ezra Karger noted about expectations of the experts and forecasters views when we interviewed him on the 80k podcast:
One of the pieces of this work that I found most interesting is that even though domain experts and superforecasters disagreed strongly, I would argue, about AI-caused risks, they both believed that AI progress would continue very quickly.
So we did ask superforecasters and domain experts when we would have an advanced AI system, according to a definition that relied on a long list of capabilities. And the domain experts gave a year of 2046, and the superforecasters gave a year of 2060.
My understanding is that XPT was using the definition of AGI used in the Metaculus question cited in Niel’s original post (though see his comment for some caveats about the definition). In March 2022, that forecast was around 2056-2058; it’s now at 2030. The Metaculus question also has over 1500 forecasters, whereas XPT had around 30 superforecasters, I believe. So overall I wouldn’t consider XPT to be strong evidence against short timelines.
I think there is some general “outside view” reason to be sceptical of short timelines. But I think there are good reasons to think that kind of perspective would miss big changes like this, and there is enough reason to believe short timelines are plausible to take action on that basis.
Thanks, I think you’ve done a decent job of identifying cruxes, and I appreciate the additional info too. Your comment about the XPT being from 2022 does update me somewhat.
One thing I’ll highlight and will be thinking about: there’s some tension between the two positions of
a) “recent AI developments are very surprising, so therefore we should update our p|doom to be significantly higher than superforecasters from 2022” and
b) “in 2022, superforecasters thought AI progress would continue very quickly beyond current day levels”
This is potentially partially resolved by the statement:
c) “superforecasters though AI progress would be fast, but it’s actually very fast, so therefore we are right to update to be significantly higher”.
This is a sensible take, and is supported by things like the Metaculus survey you cite. However, I think that if they thought it was already going to be fast, and yet still only had a small chance of extinction in 2022, then recent developments would make them give a higher probability, but not significantly higher. The exact amount it has changed, and what counts as “significantly higher” vs marginally higher has unfortunately been left as an exercise for the reader, and it’s not the only risk, so I think I do understand your position.
Thanks for the additional context! I think I understand your views better now and I appreciate your feedback.
Just speaking for myself here, I think I can identify some key cruxes between us. I’ll take them one by one:
I disagree with this. I think it’s better if people have a better understanding of the key issues raised by the emergence of AGI. We don’t have all the answers, but we’ve thought about these issues a lot and have ideas about what kinds of problems are most pressing to address and what some potential solutions are. Communicating these ideas more broadly and to people who may be able to help is just better in expectation than failing to do so (all else equal), even though, as with any problem, you can’t be sure you’re making things better, and there’s some chance you make things worse.
I don’t think I agree with this. I think the value of doing work in areas like global health or helping animals is largely in the direct impact of these actions, rather than any impact on what it means for the arrival of AGI. I don’t think even if, in an overwhelming success, we cut malaria deaths in half next year, that will meaningfully increase the likelihood that AGI is aligned or that the training data reflects a better morality. It’s more likely that directly trying to work to create beneficial AI will have these effects. Of course, the case for saving lives from malaria is still strong, because people’s lives matter and are worth saving.
Recall that the XPT is from 2022, so there’s a lot that’s happened since. Even still, here’s what Ezra Karger noted about expectations of the experts and forecasters views when we interviewed him on the 80k podcast:
My understanding is that XPT was using the definition of AGI used in the Metaculus question cited in Niel’s original post (though see his comment for some caveats about the definition). In March 2022, that forecast was around 2056-2058; it’s now at 2030. The Metaculus question also has over 1500 forecasters, whereas XPT had around 30 superforecasters, I believe. So overall I wouldn’t consider XPT to be strong evidence against short timelines.
I think there is some general “outside view” reason to be sceptical of short timelines. But I think there are good reasons to think that kind of perspective would miss big changes like this, and there is enough reason to believe short timelines are plausible to take action on that basis.
Again, thanks for engaging with all this!
Thanks, I think you’ve done a decent job of identifying cruxes, and I appreciate the additional info too. Your comment about the XPT being from 2022 does update me somewhat.
One thing I’ll highlight and will be thinking about: there’s some tension between the two positions of
a) “recent AI developments are very surprising, so therefore we should update our p|doom to be significantly higher than superforecasters from 2022” and
b) “in 2022, superforecasters thought AI progress would continue very quickly beyond current day levels”
This is potentially partially resolved by the statement:
c) “superforecasters though AI progress would be fast, but it’s actually very fast, so therefore we are right to update to be significantly higher”.
This is a sensible take, and is supported by things like the Metaculus survey you cite. However, I think that if they thought it was already going to be fast, and yet still only had a small chance of extinction in 2022, then recent developments would make them give a higher probability, but not significantly higher. The exact amount it has changed, and what counts as “significantly higher” vs marginally higher has unfortunately been left as an exercise for the reader, and it’s not the only risk, so I think I do understand your position.