I strongly agree on the dubious epistemics. A couple of corroborating experiences:
When I looked at how people understood the orthogonality thesis, my impression was that somewhere from 20-80% of EAs believed that it showed that AI misalignment was extremely likely rather than (as, if anything, it actually shows) that it’s not literally impossible.
These seemingly included an 80k careers advisor who suggested it as a reason why I should be much more concerned about AI; Will MacAskill, who in WWOtF describes ‘The scenario most closely associated with [the book Superintelligence being] one in which a single AI agent designs better and better versions of itself, quickly developing abilities far greater than the abilities of all of humanity combined. Almost certainly, its aims would not be the same as humanity’s aims’; and some or all of the authors of the paper ‘Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence’ (David Denkenberger, Anders Sandberg, Ross John Tieman, and Joshua M. Pearce) which states ‘the goals of the intelligence are essentially arbitrary [48]’, with the reference pointing to Bostrom’s orthogonality thesis essay.
I was invited to comment on 80k’s software development, in which I queried the line ‘With the rise of machine learning, and the huge success of deep learning models like GPT-3, many experts now think it’s reasonably likely that our current machine learning methods could be used to create transformative artificial intelligence’ as implying that the proportion of experts who believed this was increasing without sourcing any evidence. One of the main people behind Anthropic commented on my query ‘it’s increasing meaningfully. i explicitly want to keep this. i wouldn’t want to lose such an important point to measurability bias’ - which sounds to me like carte blanche to claim anything you want. The writer said ’I don’t have good data on this beyond the various AI Impacts surveys. https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/https://aiimpacts.org/ai-timeline-surveys/. But I trust <the anthropic person> and others to be aware of the current state of the field!′ and kept the original claim in, unsourced, verbatim.
Also on incentives, I think there’s a strong status-based and a reasonable financial incentive for academics to find controversial results that merit a ‘more research needed’ line at the end of every paper. So if we lived in a world where the impact of some new technology is likely to be relatively minor, I would still expect a subset of academics to make a name for themselves highlighting all the things that could go wrong (cf for example the debates on the ethics of autonomous vehicles, which seems to be a complete nonissue in practice, but seems to have supported a cottage industry of concerned philosophers).
On industry vs the AI safety community, I think we’re seriously underupdating on the fact that so many people in industry aren’t concerned by this. Google (for eg) engineers are an incredibly intelligent bunch who probably understand AI software substantially better than most AI safety researchers (since there are strong feedback mechanisms on who is employed at Google, whereas success in AI safety seems to be far more subjective). So from that perspective, AI safety is one of the most oversubscribed fields of existential risk.
Lastly, I think going from 90% doom to 10% doom could be more profound than you think. I’m working on a bit of software atm to implement the model I suggested here. I don’t think it’s going to let us conclude anything very confidently, but that’s sort of the point—various plausible inputs seem like they could give virtually any outcome from ‘non-extinction catastrophes are almost as bad as extinction’ to ‘non-extinction catastrophes strongly reduce extinction risk’.
Hi! Wanted to follow up as the author of the 80k software engineering career review, as I don’t think this gives an accurate impression. A few things to say:
I try to have unusually high standards for explaining why I believe the things I write, so I really appreciate people pushing on issues like this.
At the time, when you responded to <the Anthropic person>, you said “I think <the Anthropic person> is probably right” (although you added “I don’t think it’s a good idea to take this sort of claim on trust for important career prioritisation research”).
When I leave claims like this unsourced, it’s usually because I (and my editors) think they’re fairly weak claims, and/or they lack a clear source to reference. That is, the claim is effectively is a piece of research based on general knowledge (e.g. I wouldn’t source the claim “Biden is the President of the USA”) and/or interviews with a range of experts, and the claim is weak or unimportant enough not to investigate further. (FWIW I think it’s likely I should have prioritised writing a longer footnote on why I believe this claim.)
The closest data is the three surveys of NeurIPS researchers, but these are imperfect. They ask how long it will take until there is “human-level machine intelligence”. The median expert asked thought there was an around 1 in 4 chance of this by 2036. Of course, it’s not clear that HLMI and transformative AI are the same thing, or that thinking HLMI being developed soon necessarily means that HLMI will be made by scaling and adapting existing ML methods. In addition, no survey data pre-dates 2016, so it’s hard to say that these views have changed based solely on survey data. (I’ve written more about these surveys and their limitations here, with lots of detail in footnotes; and I discuss the timelines parts of those surveys in the second paragraph here.)
As a result, when I made this claim I was relying on three things. First, that there are likely correlations that make the survey data relevant (i.e., that many people answering the survey think that HLMI will be relatively similar to or cause transformative AI, and that many people answering the survey think that if HLMI is developed soon that suggests it will be ML-based). Second, that people did not think that ML could produce HLMI in the past (e.g. because other approaches like symbolic AI were still being worked on, because texts like Superintelligence do not focus on ML and this was not widely remarked upon at the time despite that book’s popularity, etc.). Third, that people in the AI and ML fields who I spoke to had a reasonable idea of what other experts used to think and how that has changed (note I spoke to many more people than the one person who responded to you in the comments on my piece)!
It’s true that there may be selection bias on this third point. I’m definitely concerned about selection bias for shorter timelines in general in the community, and plan to publish something about this at some point. But in general I think that the best way, as an outsider, to understand what prevailing opinions are in a field, is to talk to people in that field – rather than relying on your own ability to figure out trends across many papers, many of which are difficult to evaluate, many of which may not replicate. I also think that asking about what others in the field think, rather than what the people you’re talking to think, is a decent (if imperfect) way of dealing with that bias.
Overall, I thought the claim I made was weak enough (e.g. “many experts” not “most experts” or “all experts”) that I didn’t feel the need to evaluate this further.
It’s likely, given you’ve raised this, that I should have put this all in a footnote. The only reason I didn’t is that I try to prioritise, and I thought this claim was weak enough to not need much substantiation. I may go back and change that now (depending on how I prioritise this against other work).
Thanks Benjamin, I upvoted. Some things to clarify on my end:
I think the article as a whole was good, or I would have said so!
I did and do think a) that Anthropic Person (AP) was probably right, b) that their attitude was nevertheless irresponsible and epistemically poor and c) that I made it clear that I thought despite them being probably right this needed more justification at the time (the last exchange I have a record of was me reopening the comment thread that you’d resolved to state that I really did think this was important and you re-closing it without further comment)
My concern with poor epistemics was less with reference to you—I presume you were working under time constraints in an area you didn’t have specialist knowledge on—than to AP, who had no such excuse.
I would have had no factual problem with the claim ‘many experts believe’. The phrasing that I challenged, and the grounds I gave for challenging it was that ‘many experts now believe’ (emphasis mine) implies positive change over time—that the proportion of experts who believe this is increasing. That doesn’t seem anything like as self-evident as a comment about the POTUS.
Fwiw I think the rate of change (and possibly even second derivative) of expert beliefs on such a speculative and rapidly evolving subject is much more important than the absolute number or even proportion of experts with the relevant belief, especially since it’s very hard to define who even qualifies as an expert in such a field (per my comment, most of the staff at Google—and other big AI companies—could arguably qualify)
If you’d mentioned that other experts you’d spoken to had a sense that sentiment was changing (and mentioned those conversations as a pseudocitation) I would have been substantially less concerned by the point—though I do think it’s important enough to merit proper research (though such research would have probably been beyond the scope of your 80k piece), and not to imply stronger conclusions than the evidence we have merits.
I am definitely worried about selection bias—the whole concept of ‘AI safety researcher’ screams it (not that I think you only spoke to such people, but any survey which includes them and excludes ‘AI-safety-concern-rejecting researcher/developer’ seems highly likely to get prejudicial results)
But in general I think that the best way, as an outsider, to understand what prevailing opinions are in a field, is to talk to people in that field – rather than relying on your own ability to figure out trends across many papers, many of which are difficult to evaluate, many of which may not replicate. I also think that asking about what others in the field think, rather than what the people you’re talking to think, is a decent (if imperfect) way of dealing with that bias.
I don’t understand what counterclaim you’re making here. I strongly agree that the opinions of experts are very important, hence my entire concern about this exchange!
I think it’s noteworthy that surveys from 2016, 2019, and 2022 have all found roughly similar timelines to AGI (50% by ~2060) for the population of published ML researchers. On the other hand, the EA and AI safety communities seem much more focused on short timelines than they were seven years ago (though I don’t have a source on that).
There are important reasons to think that the change by the EA community is within the measurement error of these surveys, which makes this less noteworthy.
(Like say you put +/- 10 years and +/- 10% on all these answers—note there are loads of reasons why you wouldn’t actually assess the uncertainty like this, (e.g. probabilities can’t go below 0 or above 1), but just to get a feel for the uncertainty this helps. Well, then you get something like:
10%-30% chance of TAI by 2026-2046
40%-60% by 2050-2070
and 75%-95% by 2100
Then many many EA timelines and shifts in EA timelines fall within those errors.)
2. Low response rates + selection biases + not knowing the direction of those biases
The surveys plausibly had a bunch of selection biases in various directions.
This means you need a higher sample to converge on the population means, so the surveys probably aren’t representative. But we’re much less certain in which direction they’re biased.
For example, you might think researchers who go to the top AI conferences are more likely to be optimistic about AI, because they have been selected to think that AI research is doing good. Alternatively, you might think that researchers who are already concerned about AI are more likely to respond to a survey asking about these concerns
3. Other problems, like inconsistent answers in the survey itself
AI impacts wrote some interesting caveats here, including:
Asking people about specific jobs massively changes HLMI forecasts. When we asked some people when AI would be able to do several specific human occupations, and then all human occupations (presumably a subset of all tasks), they gave very much later timelines than when we just asked about HLMI straight out. For people asked to give probabilities for certain years, the difference was a factor of a thousand twenty years out! (10% vs. 0.01%) For people asked to give years for certain probabilities, the normal way of asking put 50% chance 40 years out, while the ‘occupations framing’ put it 90 years out. (These are all based on straightforward medians, not the complicated stuff in the paper.)
People consistently give later forecasts if you ask them for the probability in N years instead of the year that the probability is M. We saw this in the straightforward HLMI question, and most of the tasks and occupations, and also in most of these things when we tested them on mturk people earlier. For HLMI for instance, if you ask when there will be a 50% chance of HLMI you get a median answer of 40 years, yet if you ask what the probability of HLMI is in 40 years, you get a median answer of 30%.
Cheers! You might want to follow up with 80,000 hours on the epistemics point, e.g., alexrjl would probably be interested in hearing about and then potentially addressing your complaints.
I strongly agree on the dubious epistemics. A couple of corroborating experiences:
When I looked at how people understood the orthogonality thesis, my impression was that somewhere from 20-80% of EAs believed that it showed that AI misalignment was extremely likely rather than (as, if anything, it actually shows) that it’s not literally impossible.
These seemingly included an 80k careers advisor who suggested it as a reason why I should be much more concerned about AI; Will MacAskill, who in WWOtF describes ‘The scenario most closely associated with [the book Superintelligence being] one in which a single AI agent designs better and better versions of itself, quickly developing abilities far greater than the abilities of all of humanity combined. Almost certainly, its aims would not be the same as humanity’s aims’; and some or all of the authors of the paper ‘Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence’ (David Denkenberger, Anders Sandberg, Ross John Tieman, and Joshua M. Pearce) which states ‘the goals of the intelligence are essentially arbitrary [48]’, with the reference pointing to Bostrom’s orthogonality thesis essay.
I was invited to comment on 80k’s software development, in which I queried the line ‘With the rise of machine learning, and the huge success of deep learning models like GPT-3, many experts now think it’s reasonably likely that our current machine learning methods could be used to create transformative artificial intelligence’ as implying that the proportion of experts who believed this was increasing without sourcing any evidence. One of the main people behind Anthropic commented on my query ‘it’s increasing meaningfully. i explicitly want to keep this. i wouldn’t want to lose such an important point to measurability bias’ - which sounds to me like carte blanche to claim anything you want. The writer said ’I don’t have good data on this beyond the various AI Impacts surveys. https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/ https://aiimpacts.org/ai-timeline-surveys/. But I trust <the anthropic person> and others to be aware of the current state of the field!′ and kept the original claim in, unsourced, verbatim.
Also on incentives, I think there’s a strong status-based and a reasonable financial incentive for academics to find controversial results that merit a ‘more research needed’ line at the end of every paper. So if we lived in a world where the impact of some new technology is likely to be relatively minor, I would still expect a subset of academics to make a name for themselves highlighting all the things that could go wrong (cf for example the debates on the ethics of autonomous vehicles, which seems to be a complete nonissue in practice, but seems to have supported a cottage industry of concerned philosophers).
On industry vs the AI safety community, I think we’re seriously underupdating on the fact that so many people in industry aren’t concerned by this. Google (for eg) engineers are an incredibly intelligent bunch who probably understand AI software substantially better than most AI safety researchers (since there are strong feedback mechanisms on who is employed at Google, whereas success in AI safety seems to be far more subjective). So from that perspective, AI safety is one of the most oversubscribed fields of existential risk.
Lastly, I think going from 90% doom to 10% doom could be more profound than you think. I’m working on a bit of software atm to implement the model I suggested here. I don’t think it’s going to let us conclude anything very confidently, but that’s sort of the point—various plausible inputs seem like they could give virtually any outcome from ‘non-extinction catastrophes are almost as bad as extinction’ to ‘non-extinction catastrophes strongly reduce extinction risk’.
Hi! Wanted to follow up as the author of the 80k software engineering career review, as I don’t think this gives an accurate impression. A few things to say:
I try to have unusually high standards for explaining why I believe the things I write, so I really appreciate people pushing on issues like this.
At the time, when you responded to <the Anthropic person>, you said “I think <the Anthropic person> is probably right” (although you added “I don’t think it’s a good idea to take this sort of claim on trust for important career prioritisation research”).
When I leave claims like this unsourced, it’s usually because I (and my editors) think they’re fairly weak claims, and/or they lack a clear source to reference. That is, the claim is effectively is a piece of research based on general knowledge (e.g. I wouldn’t source the claim “Biden is the President of the USA”) and/or interviews with a range of experts, and the claim is weak or unimportant enough not to investigate further. (FWIW I think it’s likely I should have prioritised writing a longer footnote on why I believe this claim.)
The closest data is the three surveys of NeurIPS researchers, but these are imperfect. They ask how long it will take until there is “human-level machine intelligence”. The median expert asked thought there was an around 1 in 4 chance of this by 2036. Of course, it’s not clear that HLMI and transformative AI are the same thing, or that thinking HLMI being developed soon necessarily means that HLMI will be made by scaling and adapting existing ML methods. In addition, no survey data pre-dates 2016, so it’s hard to say that these views have changed based solely on survey data. (I’ve written more about these surveys and their limitations here, with lots of detail in footnotes; and I discuss the timelines parts of those surveys in the second paragraph here.)
As a result, when I made this claim I was relying on three things. First, that there are likely correlations that make the survey data relevant (i.e., that many people answering the survey think that HLMI will be relatively similar to or cause transformative AI, and that many people answering the survey think that if HLMI is developed soon that suggests it will be ML-based). Second, that people did not think that ML could produce HLMI in the past (e.g. because other approaches like symbolic AI were still being worked on, because texts like Superintelligence do not focus on ML and this was not widely remarked upon at the time despite that book’s popularity, etc.). Third, that people in the AI and ML fields who I spoke to had a reasonable idea of what other experts used to think and how that has changed (note I spoke to many more people than the one person who responded to you in the comments on my piece)!
It’s true that there may be selection bias on this third point. I’m definitely concerned about selection bias for shorter timelines in general in the community, and plan to publish something about this at some point. But in general I think that the best way, as an outsider, to understand what prevailing opinions are in a field, is to talk to people in that field – rather than relying on your own ability to figure out trends across many papers, many of which are difficult to evaluate, many of which may not replicate. I also think that asking about what others in the field think, rather than what the people you’re talking to think, is a decent (if imperfect) way of dealing with that bias.
Overall, I thought the claim I made was weak enough (e.g. “many experts” not “most experts” or “all experts”) that I didn’t feel the need to evaluate this further.
It’s likely, given you’ve raised this, that I should have put this all in a footnote. The only reason I didn’t is that I try to prioritise, and I thought this claim was weak enough to not need much substantiation. I may go back and change that now (depending on how I prioritise this against other work).
Thanks Benjamin, I upvoted. Some things to clarify on my end:
I think the article as a whole was good, or I would have said so!
I did and do think a) that Anthropic Person (AP) was probably right, b) that their attitude was nevertheless irresponsible and epistemically poor and c) that I made it clear that I thought despite them being probably right this needed more justification at the time (the last exchange I have a record of was me reopening the comment thread that you’d resolved to state that I really did think this was important and you re-closing it without further comment)
My concern with poor epistemics was less with reference to you—I presume you were working under time constraints in an area you didn’t have specialist knowledge on—than to AP, who had no such excuse.
I would have had no factual problem with the claim ‘many experts believe’. The phrasing that I challenged, and the grounds I gave for challenging it was that ‘many experts now believe’ (emphasis mine) implies positive change over time—that the proportion of experts who believe this is increasing. That doesn’t seem anything like as self-evident as a comment about the POTUS.
Fwiw I think the rate of change (and possibly even second derivative) of expert beliefs on such a speculative and rapidly evolving subject is much more important than the absolute number or even proportion of experts with the relevant belief, especially since it’s very hard to define who even qualifies as an expert in such a field (per my comment, most of the staff at Google—and other big AI companies—could arguably qualify)
If you’d mentioned that other experts you’d spoken to had a sense that sentiment was changing (and mentioned those conversations as a pseudocitation) I would have been substantially less concerned by the point—though I do think it’s important enough to merit proper research (though such research would have probably been beyond the scope of your 80k piece), and not to imply stronger conclusions than the evidence we have merits.
I am definitely worried about selection bias—the whole concept of ‘AI safety researcher’ screams it (not that I think you only spoke to such people, but any survey which includes them and excludes ‘AI-safety-concern-rejecting researcher/developer’ seems highly likely to get prejudicial results)
I don’t understand what counterclaim you’re making here. I strongly agree that the opinions of experts are very important, hence my entire concern about this exchange!
Thanks for this! Looks like we actually roughly agree overall :)
I think it’s noteworthy that surveys from 2016, 2019, and 2022 have all found roughly similar timelines to AGI (50% by ~2060) for the population of published ML researchers. On the other hand, the EA and AI safety communities seem much more focused on short timelines than they were seven years ago (though I don’t have a source on that).
There are important reasons to think that the change by the EA community is within the measurement error of these surveys, which makes this less noteworthy.
(Like say you put +/- 10 years and +/- 10% on all these answers—note there are loads of reasons why you wouldn’t actually assess the uncertainty like this, (e.g. probabilities can’t go below 0 or above 1), but just to get a feel for the uncertainty this helps. Well, then you get something like:
10%-30% chance of TAI by 2026-2046
40%-60% by 2050-2070
and 75%-95% by 2100
Then many many EA timelines and shifts in EA timelines fall within those errors.)
Reasons why these surveys have huge error
1. Low response rates.
The response rates were really quite low.
2. Low response rates + selection biases + not knowing the direction of those biases
The surveys plausibly had a bunch of selection biases in various directions.
This means you need a higher sample to converge on the population means, so the surveys probably aren’t representative. But we’re much less certain in which direction they’re biased.
Quoting me:
3. Other problems, like inconsistent answers in the survey itself
AI impacts wrote some interesting caveats here, including:
The 80k podcast on the 2016 survey goes into this too.
Cheers! You might want to follow up with 80,000 hours on the epistemics point, e.g., alexrjl would probably be interested in hearing about and then potentially addressing your complaints.