Thanks, I think you’ve done a decent job of identifying cruxes, and I appreciate the additional info too. Your comment about the XPT being from 2022 does update me somewhat.
One thing I’ll highlight and will be thinking about: there’s some tension between the two positions of
a) “recent AI developments are very surprising, so therefore we should update our p|doom to be significantly higher than superforecasters from 2022” and
b) “in 2022, superforecasters thought AI progress would continue very quickly beyond current day levels”
This is potentially partially resolved by the statement:
c) “superforecasters though AI progress would be fast, but it’s actually very fast, so therefore we are right to update to be significantly higher”.
This is a sensible take, and is supported by things like the Metaculus survey you cite. However, I think that if they thought it was already going to be fast, and yet still only had a small chance of extinction in 2022, then recent developments would make them give a higher probability, but not significantly higher. The exact amount it has changed, and what counts as “significantly higher” vs marginally higher has unfortunately been left as an exercise for the reader, and it’s not the only risk, so I think I do understand your position.
Thanks, I think you’ve done a decent job of identifying cruxes, and I appreciate the additional info too. Your comment about the XPT being from 2022 does update me somewhat.
One thing I’ll highlight and will be thinking about: there’s some tension between the two positions of
a) “recent AI developments are very surprising, so therefore we should update our p|doom to be significantly higher than superforecasters from 2022” and
b) “in 2022, superforecasters thought AI progress would continue very quickly beyond current day levels”
This is potentially partially resolved by the statement:
c) “superforecasters though AI progress would be fast, but it’s actually very fast, so therefore we are right to update to be significantly higher”.
This is a sensible take, and is supported by things like the Metaculus survey you cite. However, I think that if they thought it was already going to be fast, and yet still only had a small chance of extinction in 2022, then recent developments would make them give a higher probability, but not significantly higher. The exact amount it has changed, and what counts as “significantly higher” vs marginally higher has unfortunately been left as an exercise for the reader, and it’s not the only risk, so I think I do understand your position.