Thanks for sharing your thoughts. I feel uncertain about how valuable itâd be to collect quantitative info about peopleâs beliefs on questions like these, and your comment has provided useful a input/âperspective on that matter.
Some thoughts/âquestions in response:
Do you think itâs not even net positive to collect such info (e.g., because people end up anchoring on the results or perceiving the respondents as simplistic thinkers)? Or do you just think itâs unclear that itâs net positive enough to justify the time required (from the survey organiser and from the respondents)?
Do you think such info doesnât even reduce our uncertainty and confusion at all? Or just that it only reduces it by a small amount?
Relatedly, I have an impression that people sometimes deny the value of quantitative estimates/âforecasts in general based on seeming to view us as simply either âuncertainâ or âcertainâ on a given matter (e.g., âweâll still have no idea at allâ). In contrast, I think we always have some but not complete uncertainty, and that we can often/âalways move closer to certainty by small increments.
That said, one can share that view of mine and yet think these estimates/âforecasts (or any other particular thing) donât help us move closer to certainty at all.
It seems to me that those takeaways are not things everyone is (viscerally) aware of, and that theyâre things itâs valuable for people to be (viscerally) aware of. So it seems to me plausible that these seemingly disappointing takeaways actually indicate some value to these efforts. Does that sound right to you?
E.g., I wouldnât be surprised if a large portion of people who donât work at places like FHI wouldnât realise that itâs hard to know how to even operationalise different types of AI risk, and would expect that people at FHI all agree pretty closely on some of these questions.
And I wouldnât be super surprised if even some people who do work at places like FHI thought operationalisations would be relatively easy, agreement would be pretty high, etc. Though I donât really know.
That said, there may be other, cheaper ways to spread those takeaways. E.g., perhaps, simply having a meeting where those points are discussed explicitly but qualitatively, and then releasing a statement on the matter.
Would you apply similar thinking to the question of how valuable existential risk estimates in particular are? Iâd imagine so? Does this mean you see the database of existential risk estimates as of low or negative value?
I ask this question genuinely rather than defensively. Iâm decently confident the database is net positive, but very uncertain about how positive, and open to the idea that itâs net negative.
Do you think itâs not even net positive to collect such info (e.g., because people end up anchoring on the results or perceiving the respondents as simplistic thinkers)? Or do you just think itâs unclear that itâs net positive enough to justify the time required (from the survey organiser and from the respondents)?
Personally, I think itâs net positive but not worth the time investment in most cases. But based on feedback some other people think itâs net negative, at least when not executed exceptionally wellâmostly due to anchoring, projecting a sense of false confidence, risk of numbers being quoted out of context etc.
Do you think such info doesnât even reduce our uncertainty and confusion at all? Or just that it only reduces it by a small amount?
I think an idealized survey would reduce uncertainty a bit. But in practice I think itâs too hard to tell the signal apart from the noise, and so that it basically doesnât reduce object-level uncertainty at all. Iâm more positive about the results providing some high-level takeaways (e.g. âpeople disagree a lotâ) or identifying specific disagreements (e.g. âthese two people disagree a lot on that specific questionâ).
It seems to me that those takeaways are not things everyone is (viscerally) aware of, and that theyâre things itâs valuable for people to be (viscerally) aware of. So it seems to me plausible that these seemingly disappointing takeaways actually indicate some value to these efforts. Does that sound right to you?
Yes, that sounds right to me. I think itâs a bit tricky to get the message right though. I think Iâd want to roughly convey a (more nuanced version of) âwe still need people who can think through questions themselves and form their own views, not just people who seek guidance from some consensus which on many questions may not existâ. (Buckâs post on deference and inside-view models is somewhat related.) But itâs tricky to avoid pessimistic/ânon-constructive impressions like âpeople have no idea what theyâre talking about, so we should stop giving any weight to themâ or âwe donât know anything and so canât do anything about improving the longterm futureâ.
I also do feel a bit torn about the implications myself. After all, the survey issues mostly indicate a failure of a specific way of making beliefs explicit, not necessarily a practical defect in those beliefs themselves. (Weird analogy: if you survey carpenters on weird questions about tables, maybe they also wonât give very useful replies, but they might still be great at building tables.) And especially if weâre pessimistic about the tractability of reducing confusion, then maybe advice along the lines of (e.g.) âtry to do useful AI safety work even if you canât give super clear justifications for what youâre doing and donât fully understand the views of many of your peersâ is among the best generic advice we can give, despite some remaining unease from people who are temperamentally maths/âanalytic philosopher types such as myself.
Would you apply similar thinking to the question of how valuable existential risk estimates in particular are? Iâd imagine so? Does this mean you see the database of existential risk estimates as of low or negative value?
I think a database is valuable precisely because it shows a range of estimates, including the fact that different estimates sometimes diverge a lot.
Regarding existential risk estimates, I do see value in doing research on specific questions that would make us adjust those estimates, and then adjusting them accordingly. But this is probably not among the top criteria Iâd use to pick research questions, and usually Iâd expect most of the value to come from other sources (e.g. identifying potential interventions/âsolutions, field building or other indirect effects, âŚ). The reason mostly is that Iâm skeptical marginal research will change âconsensus estimatesâ by enough that the change in the quantitative probability by itself will have practical consequences. E.g. I think it mostly doesnât matter for practical purposes if you think the risk of extinction from AI this century is, say, 8% or 10% (making up numbers, not my beliefs). If I thought there was a research project that would cause most people to revise that estimate to, say, 0.1% I do think this would be super valuable. But I donât think there is such a research project. (There are already both people whose credences are 0.1% and 10%, respectively, but the issue is they donât fully understand each other, disagree about how to interpret the evidence etc. - and additional research wouldnât significantly change this.)
Again, I do think there are various valuable research projects that would inform our views on how likely extinction from AI is, among other things. But Iâd expect most of the value to come from things other than moving that specific credence.
In any case, all of these things are very different from asking someone who hasnât done such research to fill in a survey. I think surveying more people on what their x-risk credences are will have ~zero or even negative epistemic value for the purpose of improving our x-risk estimates. Instead, weâd need to identify specific research questions, have people spend a long time doing the required research, and then ask those specific people. (So e.g. I think Ordâs estimate have positive epistemic value, and they also would if he stated them in a surveyâthe point is that this is because he has spent a lot of time deriving these specific estimates. But if you survey people, even longtermist researchers, most of them wonât have done such research, and even if they have lots of thoughts on relevant questions if you ask them to give a number they havenât previously derived with great care theyâll essentially âmake it upâ.)
I think I largely agree, except that I think Iâm on the fence about the last paragraph.
Regarding existential risk estimates, I do see value in doing research on specific questions that would make us adjust those estimates, and then adjusting them accordingly.
I agree with what you say in this paragraph. But it seems somewhat separate to the question of how valuable it is to elicit and collate current views?
I think my views are roughly as follows:
âMost relevant experts are fairly confident that certain existential risks (e.g., from AI) as substantially more likely than others (e.g., from asteroids or gamma ray bursts). The vast majority of peopleâand a substantial portion of EAs, longtermists, policymakers, etc. - probably arenât aware experts think that, and might guess that the difference in risk levels is less substantial, or be unable to guess which risks are most likely. (This seems analogous to the situation with large differences in charity cost-effectiveness.) Therefore, eliciting and collecting expertsâ views can provide a useful input into other peopleâs prioritisation decisions.
That said, on the margin, itâll be very hard to shift the relevant expertsâ credences on x-risk levels by more than, for example, a factor of two. And there are often already larger differences in other factors in our decisionsâe.g., tractability of or personal fit for interventions. In addition, we donât know how much weight to put on expertsâ specific credences anyway. So thereâs not that much value in trying to further inform the relevant expertsâ credences on x-risk levels. (Though the same work that would do that might be very valuable for other reasons, like helping those experts build more detailed models of how risks would occur and what the levers for intervention are.)â
Does that roughly match your views?
If I thought there was a research project that would cause most people to revise that estimate to, say, 0.1% I do think this would be super valuable.
Just to check, I assume you mean that thereâd be a lot of value in a research project that would cause most people to revise that estimate to (say) 0.1%, if indeed the best estimate is (say) 0.1%, and that wouldnât cause such a revision otherwise?
One alternative thing you might mean: âI think the best estimate is 0.1%, and I think a research project that would cause most people to realise that would be super valuable.â But Iâm guessing thatâs not what you mean?
Yes, that sounds roughly right. I hadnât thought about the value for communicating with broader audiences.
Just to check, I assume you mean that thereâd be a lot of value in a research project that would cause most people to revise that estimate to (say) 0.1%, if indeed the best estimate is (say) 0.1%, and that wouldnât cause such a revision otherwise?
Yes, thatâs what I meant.
(I think my own estimate is somewhere between 0.1% and 10% FWIW, but also feels quite unstable and like I donât trust that number much.)
Thanks for sharing your thoughts. I feel uncertain about how valuable itâd be to collect quantitative info about peopleâs beliefs on questions like these, and your comment has provided useful a input/âperspective on that matter.
Some thoughts/âquestions in response:
Do you think itâs not even net positive to collect such info (e.g., because people end up anchoring on the results or perceiving the respondents as simplistic thinkers)? Or do you just think itâs unclear that itâs net positive enough to justify the time required (from the survey organiser and from the respondents)?
Do you think such info doesnât even reduce our uncertainty and confusion at all? Or just that it only reduces it by a small amount?
Relatedly, I have an impression that people sometimes deny the value of quantitative estimates/âforecasts in general based on seeming to view us as simply either âuncertainâ or âcertainâ on a given matter (e.g., âweâll still have no idea at allâ). In contrast, I think we always have some but not complete uncertainty, and that we can often/âalways move closer to certainty by small increments.
That said, one can share that view of mine and yet think these estimates/âforecasts (or any other particular thing) donât help us move closer to certainty at all.
It seems to me that those takeaways are not things everyone is (viscerally) aware of, and that theyâre things itâs valuable for people to be (viscerally) aware of. So it seems to me plausible that these seemingly disappointing takeaways actually indicate some value to these efforts. Does that sound right to you?
E.g., I wouldnât be surprised if a large portion of people who donât work at places like FHI wouldnât realise that itâs hard to know how to even operationalise different types of AI risk, and would expect that people at FHI all agree pretty closely on some of these questions.
And I wouldnât be super surprised if even some people who do work at places like FHI thought operationalisations would be relatively easy, agreement would be pretty high, etc. Though I donât really know.
That said, there may be other, cheaper ways to spread those takeaways. E.g., perhaps, simply having a meeting where those points are discussed explicitly but qualitatively, and then releasing a statement on the matter.
Would you apply similar thinking to the question of how valuable existential risk estimates in particular are? Iâd imagine so? Does this mean you see the database of existential risk estimates as of low or negative value?
I ask this question genuinely rather than defensively. Iâm decently confident the database is net positive, but very uncertain about how positive, and open to the idea that itâs net negative.
Personally, I think itâs net positive but not worth the time investment in most cases. But based on feedback some other people think itâs net negative, at least when not executed exceptionally wellâmostly due to anchoring, projecting a sense of false confidence, risk of numbers being quoted out of context etc.
I think an idealized survey would reduce uncertainty a bit. But in practice I think itâs too hard to tell the signal apart from the noise, and so that it basically doesnât reduce object-level uncertainty at all. Iâm more positive about the results providing some high-level takeaways (e.g. âpeople disagree a lotâ) or identifying specific disagreements (e.g. âthese two people disagree a lot on that specific questionâ).
Yes, that sounds right to me. I think itâs a bit tricky to get the message right though. I think Iâd want to roughly convey a (more nuanced version of) âwe still need people who can think through questions themselves and form their own views, not just people who seek guidance from some consensus which on many questions may not existâ. (Buckâs post on deference and inside-view models is somewhat related.) But itâs tricky to avoid pessimistic/ânon-constructive impressions like âpeople have no idea what theyâre talking about, so we should stop giving any weight to themâ or âwe donât know anything and so canât do anything about improving the longterm futureâ.
I also do feel a bit torn about the implications myself. After all, the survey issues mostly indicate a failure of a specific way of making beliefs explicit, not necessarily a practical defect in those beliefs themselves. (Weird analogy: if you survey carpenters on weird questions about tables, maybe they also wonât give very useful replies, but they might still be great at building tables.) And especially if weâre pessimistic about the tractability of reducing confusion, then maybe advice along the lines of (e.g.) âtry to do useful AI safety work even if you canât give super clear justifications for what youâre doing and donât fully understand the views of many of your peersâ is among the best generic advice we can give, despite some remaining unease from people who are temperamentally maths/âanalytic philosopher types such as myself.
I think a database is valuable precisely because it shows a range of estimates, including the fact that different estimates sometimes diverge a lot.
Regarding existential risk estimates, I do see value in doing research on specific questions that would make us adjust those estimates, and then adjusting them accordingly. But this is probably not among the top criteria Iâd use to pick research questions, and usually Iâd expect most of the value to come from other sources (e.g. identifying potential interventions/âsolutions, field building or other indirect effects, âŚ). The reason mostly is that Iâm skeptical marginal research will change âconsensus estimatesâ by enough that the change in the quantitative probability by itself will have practical consequences. E.g. I think it mostly doesnât matter for practical purposes if you think the risk of extinction from AI this century is, say, 8% or 10% (making up numbers, not my beliefs). If I thought there was a research project that would cause most people to revise that estimate to, say, 0.1% I do think this would be super valuable. But I donât think there is such a research project. (There are already both people whose credences are 0.1% and 10%, respectively, but the issue is they donât fully understand each other, disagree about how to interpret the evidence etc. - and additional research wouldnât significantly change this.)
Again, I do think there are various valuable research projects that would inform our views on how likely extinction from AI is, among other things. But Iâd expect most of the value to come from things other than moving that specific credence.
In any case, all of these things are very different from asking someone who hasnât done such research to fill in a survey. I think surveying more people on what their x-risk credences are will have ~zero or even negative epistemic value for the purpose of improving our x-risk estimates. Instead, weâd need to identify specific research questions, have people spend a long time doing the required research, and then ask those specific people. (So e.g. I think Ordâs estimate have positive epistemic value, and they also would if he stated them in a surveyâthe point is that this is because he has spent a lot of time deriving these specific estimates. But if you survey people, even longtermist researchers, most of them wonât have done such research, and even if they have lots of thoughts on relevant questions if you ask them to give a number they havenât previously derived with great care theyâll essentially âmake it upâ.)
Thanks, thatâs all really interesting.
I think I largely agree, except that I think Iâm on the fence about the last paragraph.
I agree with what you say in this paragraph. But it seems somewhat separate to the question of how valuable it is to elicit and collate current views?
I think my views are roughly as follows:
âMost relevant experts are fairly confident that certain existential risks (e.g., from AI) as substantially more likely than others (e.g., from asteroids or gamma ray bursts). The vast majority of peopleâand a substantial portion of EAs, longtermists, policymakers, etc. - probably arenât aware experts think that, and might guess that the difference in risk levels is less substantial, or be unable to guess which risks are most likely. (This seems analogous to the situation with large differences in charity cost-effectiveness.) Therefore, eliciting and collecting expertsâ views can provide a useful input into other peopleâs prioritisation decisions.
That said, on the margin, itâll be very hard to shift the relevant expertsâ credences on x-risk levels by more than, for example, a factor of two. And there are often already larger differences in other factors in our decisionsâe.g., tractability of or personal fit for interventions. In addition, we donât know how much weight to put on expertsâ specific credences anyway. So thereâs not that much value in trying to further inform the relevant expertsâ credences on x-risk levels. (Though the same work that would do that might be very valuable for other reasons, like helping those experts build more detailed models of how risks would occur and what the levers for intervention are.)â
Does that roughly match your views?
Just to check, I assume you mean that thereâd be a lot of value in a research project that would cause most people to revise that estimate to (say) 0.1%, if indeed the best estimate is (say) 0.1%, and that wouldnât cause such a revision otherwise?
One alternative thing you might mean: âI think the best estimate is 0.1%, and I think a research project that would cause most people to realise that would be super valuable.â But Iâm guessing thatâs not what you mean?
Yes, that sounds roughly right. I hadnât thought about the value for communicating with broader audiences.
Yes, thatâs what I meant.
(I think my own estimate is somewhere between 0.1% and 10% FWIW, but also feels quite unstable and like I donât trust that number much.)