Iāll share some low-confidence answers, plus some reasoning.
1. How likely do you think it is that the overall value of the future will be drastically less than it could have been, as a result of humanity not doing enough technical AI safety research?
My own answer: ~4% (note: this is my all-things-considered belief, not just my independent impression)
Predicted meansurvey answer: 14%
Predicted median survey answer: 6%
---
Reasoning for my own answer:
I previously wrote āConditional on TAI being developed and deployed someday (which rules out e.g. it being impossible or an x-catastrophe occurring before then), I fairly arbitrarily estimate a ~10% chance of that precipitating an existential catastrophe.ā
(That phrasing implies that AI could only cause existential catastrophe once TAI is developed and deployed. I think that this is misleading, though perhaps technically true, in the sense that any AI that causes an x-catastrophe is thereby as transformative as the industrial revolution.)
And I also wrote: āI had a very vague sense that there was a 2% chance of x-catastrophe from anything other than TAI by 2055. This was based on basically nothing. Maybe I was just trying to be broadly consistent with my other views and with e.g. Ordās views, but without checking in detail what consistency should entail.ā
I canāt remember if this was conditioning on there being no TAI-induced x-catastrophe by then, but I think it implicitly was
Iāll defer to that past thinking of mine
This implies something very roughly like a 7% existential risk from AI specifically (accounting for the chance that itās for some reason impossible to build TAI or that an x-catastrophe occurs before itās builtāwhich need not be before 2055)
My interpretation of this question captures only a subset of total x-risk from AI
āDrastically lessā implies existential catastrophe, not just a more minor trajectory change, so the answer has to be equal to or less than total x-risk from AI
And I think existential catastrophe from AI could occur even if we put a huge civilizational effort into technical AI safety
E.g., we could have an aligned AI but then itās misused (including in ways most humans are happy with but that are still morally horrible or squander our potential)
Before reading Note B, I interpreted the question as āHow likely do you think it is that the overall value of the future will be drastically less than it could have been, with the key reason being that humanity didnāt do enough of the right kinds of technical AI safety research?ā For that I said 2.5%.
Note B made me change my answer to 4%, and also makes me feel that the question is a bit weird.
If we had ten thousand well managed/ācoordinated top researchers collaborating on technical AI safety for 200 subjective years, it seems like theyād end up just also doing a lot of moral philosophy, political science, AI governance, etc. And if we say they have to stick to technical AI safety, theyāll just find ways to do the other things but make it look like technical AI safety. I think fairly early on theyāll notice that the biggest remaining issues arenāt really technical AI safety issues, and that itād be crazy to just keep going further and further on the technical stuff.
---
Reasoning for my predicted survey result:
My impression is that people at MIRI would probably have a mean x-risk from AI estimate of ~50%, while people at the other places you mentioned would have a mean estimate of ~10% and a median of 8%.
With (even) less confidence, Iād say people at MIRI would give a mean of 40% to question 1, and people elsewhere would give a mean of 7% and a median of 5%.
Maybe the survey selects from more pessimistic or more optimistic people than average. But I didnāt try to account for that.
Iām guessing MIRI people will be something like a quarter of your respondents.
This suggests the mean survey response would be ~17.5% (40*0.25 + 10*0.75)
It also suggests the median may be close to the median of the non-MIRI people, i.e. close to 5%.
---
I notice that my all-things-considered belief is decently far from what I predict survey respondents will say, even though I expect survey respondents will know much more about AI x-risk and what technical AI safety research could achieve than I do. This feels a bit weird.
But I think itās less that Iām very confident in my independent impressions /ā inside-views here, and more that I think the survey will overweight MIRI and (less importantly) that I also defer to people who donāt research long-term AI topics specifically. (To be clear, I donāt mean I trust MIRIās judgement on this less than I trust each other groupās judgement, just that I give them less than a third as much weight as all of the other mentioned groups combined.)
---
...I realised at this point Iād become nerd-sniped, and so forbade myself from doing question 2.
My impression is that people at MIRI would probably have a mean x-risk from AI estimate of ~50%, while people at the other places you mentioned would have a mean estimate of ~10% and a median of 8%.
Looking only at people who declared their affiliation: MIRI peopleās mean probability for x-catastrophes from āAI systems not doing/āoptimizing what the people deploying them wanted/āintendedā was 80% (though Iām not sure this is what you mean by āx-risk from AIā here), with median 70%.
People who declared a non-MIRI affiliation had a mean Q2 probability of 27.8%, median 26%.
With (even) less confidence, Iād say people at MIRI would give a mean of 40% to question 1, and people elsewhere would give a mean of 7% and a median of 5%.
For Q1, MIRI-identified people gave mean 70% (and median 80%). Non-MIRI-identified people gave mean ~18.7%, median 10%.
Iām guessing MIRI people will be something like a quarter of your respondents.
5ā27 of respondents who specified an affiliation said they work at MIRI (~19%). (By comparison, 17/ā~117 ~= 15% of recipients work at MIRI.)
Iāll share some low-confidence answers, plus some reasoning.
My own answer: ~4% (note: this is my all-things-considered belief, not just my independent impression)
Predicted mean survey answer: 14%
Predicted median survey answer: 6%
---
Reasoning for my own answer:
I previously wrote āConditional on TAI being developed and deployed someday (which rules out e.g. it being impossible or an x-catastrophe occurring before then), I fairly arbitrarily estimate a ~10% chance of that precipitating an existential catastrophe.ā
(That phrasing implies that AI could only cause existential catastrophe once TAI is developed and deployed. I think that this is misleading, though perhaps technically true, in the sense that any AI that causes an x-catastrophe is thereby as transformative as the industrial revolution.)
And I also wrote: āI had a very vague sense that there was a 2% chance of x-catastrophe from anything other than TAI by 2055. This was based on basically nothing. Maybe I was just trying to be broadly consistent with my other views and with e.g. Ordās views, but without checking in detail what consistency should entail.ā
I canāt remember if this was conditioning on there being no TAI-induced x-catastrophe by then, but I think it implicitly was
Iāll defer to that past thinking of mine
This implies something very roughly like a 7% existential risk from AI specifically (accounting for the chance that itās for some reason impossible to build TAI or that an x-catastrophe occurs before itās builtāwhich need not be before 2055)
My interpretation of this question captures only a subset of total x-risk from AI
āDrastically lessā implies existential catastrophe, not just a more minor trajectory change, so the answer has to be equal to or less than total x-risk from AI
And I think existential catastrophe from AI could occur even if we put a huge civilizational effort into technical AI safety
E.g., we could have an aligned AI but then itās misused (including in ways most humans are happy with but that are still morally horrible or squander our potential)
Before reading Note B, I interpreted the question as āHow likely do you think it is that the overall value of the future will be drastically less than it could have been, with the key reason being that humanity didnāt do enough of the right kinds of technical AI safety research?ā For that I said 2.5%.
Note B made me change my answer to 4%, and also makes me feel that the question is a bit weird.
If we had ten thousand well managed/ācoordinated top researchers collaborating on technical AI safety for 200 subjective years, it seems like theyād end up just also doing a lot of moral philosophy, political science, AI governance, etc. And if we say they have to stick to technical AI safety, theyāll just find ways to do the other things but make it look like technical AI safety. I think fairly early on theyāll notice that the biggest remaining issues arenāt really technical AI safety issues, and that itād be crazy to just keep going further and further on the technical stuff.
---
Reasoning for my predicted survey result:
My impression is that people at MIRI would probably have a mean x-risk from AI estimate of ~50%, while people at the other places you mentioned would have a mean estimate of ~10% and a median of 8%.
With (even) less confidence, Iād say people at MIRI would give a mean of 40% to question 1, and people elsewhere would give a mean of 7% and a median of 5%.
Maybe the survey selects from more pessimistic or more optimistic people than average. But I didnāt try to account for that.
Iām guessing MIRI people will be something like a quarter of your respondents.
This suggests the mean survey response would be ~17.5% (40*0.25 + 10*0.75)
It also suggests the median may be close to the median of the non-MIRI people, i.e. close to 5%.
---
I notice that my all-things-considered belief is decently far from what I predict survey respondents will say, even though I expect survey respondents will know much more about AI x-risk and what technical AI safety research could achieve than I do. This feels a bit weird.
But I think itās less that Iām very confident in my independent impressions /ā inside-views here, and more that I think the survey will overweight MIRI and (less importantly) that I also defer to people who donāt research long-term AI topics specifically. (To be clear, I donāt mean I trust MIRIās judgement on this less than I trust each other groupās judgement, just that I give them less than a third as much weight as all of the other mentioned groups combined.)
---
...I realised at this point Iād become nerd-sniped, and so forbade myself from doing question 2.
Thanks for registering your predictions, Michael!
Results (hover to read):
Mean answer for Q1 was ~30.1%, median answer 20%.
Looking only at people who declared their affiliation: MIRI peopleās mean probability for x-catastrophes from āAI systems not doing/āoptimizing what the people deploying them wanted/āintendedā was 80% (though Iām not sure this is what you mean by āx-risk from AIā here), with median 70%.
People who declared a non-MIRI affiliation had a mean Q2 probability of 27.8%, median 26%.
For Q1, MIRI-identified people gave mean 70% (and median 80%). Non-MIRI-identified people gave mean ~18.7%, median 10%.
5ā27 of respondents who specified an affiliation said they work at MIRI (~19%). (By comparison, 17/ā~117 ~= 15% of recipients work at MIRI.)
Interesting, thanks!
(Iāve added some ruminations on my failings and confusions in a comment on your results post.)