Why there hasn’t been a consensus/debate between people with contradicting views on the AGI timelines/safety topic?
I know almost nothing about ML/AI and I don’t think I can form an opinion on my own so I try to base my opinion on the opinions of more knowledgeable people that I trust an respect. However what I find problematic is that those opinions vary dramatically, while it is not clear why those people hold their beliefs. I also don’t think I have enough knowledge in the area to be able to extract that information from people myself eg. if I talk to a knowledgeable ‘AGI soon and bad’ person they would very likely convince me in their view and the same would happen if I talk to a knowledgeable ‘AGI not soon and good’ person. Wouldn’t it be good idea to have debates between people with those contradicting views, figure out what the cruxes are and write them down? I understand that some people have vested interests in one side of the questions, for example a CEO of an AI company may not gain much from such debate and thus refuse to participate in it, but I think there are many reasonable people that would be willing to share their opinion and hear other people’s arguments. Forgive me if this has already been done and I have missed it (but I would appreciate if you can point me to it).
Overall, I think a) this would be cool to see more of and b) it would be a service to the community if someone collected all the existing examples together.
Not exactly what you’re describing, but MIRI and other safety researchers did the MIRI conversations and also sort of debated at events. They were helpful and I would be excited about having more, but I think there are at least three obstacles to identifying cruxes:
Yudkowsky just has the pessimism dial set way higher than anyone else (it’s not clear that this is wrong, but this makes it hard to debate whether a plan will work)
Often two research agendas are built in different ontologies, and this causes a lot of friction especially when researcher A’s ontology is unnatural to researcher B. See the comments to this for a long discussion on what counts as inner vs outer alignment.
Much of the disagreement comes down to research taste; see my comment here for an example of differences in opinion driven by taste.
That said, I’d be excited about debates between people with totally different views, e.g. Yudkowsky and Yann Lecun if it could happen...
The debate on this subject has been ongoing between individuals who are within or adjacent to the EA/LessWrong communities (see posts that other comments have linked and other links that are sure to follow). However, these debates often are highly insular and primarily are between people who share core assumptions about:
AGI being an existential risk with a high probability of occurring
Extinction via AGI having a significant probability of occurring within our lifetimes (next 10-50 years)
Other extinction risks (e.g pandemics or nuclear war) not likely manifesting prior to AGI and curtailing AI development such that AGI risk is no longer of relevance in any near-term timeline as a result
AGI being a more deadly existential risk than other existential risks (e.g pandemics or nuclear war)
AI alignment research being neglected and/or tractable
Current work on fairness and transparency improving methods for AI models not being particularly useful towards solving AI alignment
There are many other AI researchers and individuals from other relevant, adjacent disciplines that would disagree with all or most of these assumptions. Debates between that group and people within the EA/LessWrong community who would mostly agree with the above assumptions is something that is sorely lacking, save for some mud-flinging on Twitter between AI ethicists and AI alignment researchers.
Prediction is hard and reading the debate from the vantage point of 14 years in the future it’s clear that in many ways the science and the argument has moved on, but it’s also clear that Eliezer made better predictions than Robin Hanson did, in a way that inclines me to try and learn as much of his worldview as possible so I can analyze other arguments through that frame.
Essentially, Yudkowsky is very worried about AGI (‘we’re dead in 20-30 years’ worried) because he thinks that progress on AI overall will rapidly accelerate as AI helps us make further progress. Hanson was (is?) less worried.
Why there hasn’t been a consensus/debate between people with contradicting views on the AGI timelines/safety topic?
I know almost nothing about ML/AI and I don’t think I can form an opinion on my own so I try to base my opinion on the opinions of more knowledgeable people that I trust an respect. However what I find problematic is that those opinions vary dramatically, while it is not clear why those people hold their beliefs. I also don’t think I have enough knowledge in the area to be able to extract that information from people myself eg. if I talk to a knowledgeable ‘AGI soon and bad’ person they would very likely convince me in their view and the same would happen if I talk to a knowledgeable ‘AGI not soon and good’ person. Wouldn’t it be good idea to have debates between people with those contradicting views, figure out what the cruxes are and write them down? I understand that some people have vested interests in one side of the questions, for example a CEO of an AI company may not gain much from such debate and thus refuse to participate in it, but I think there are many reasonable people that would be willing to share their opinion and hear other people’s arguments. Forgive me if this has already been done and I have missed it (but I would appreciate if you can point me to it).
OpenPhil has commissioned various reviews of its work, e.g. on power-seeking AI.
Less formal, but there was this facebook debate between some big names in AI.
Overall, I think a) this would be cool to see more of and b) it would be a service to the community if someone collected all the existing examples together.
Not exactly what you’re describing, but MIRI and other safety researchers did the MIRI conversations and also sort of debated at events. They were helpful and I would be excited about having more, but I think there are at least three obstacles to identifying cruxes:
Yudkowsky just has the pessimism dial set way higher than anyone else (it’s not clear that this is wrong, but this makes it hard to debate whether a plan will work)
Often two research agendas are built in different ontologies, and this causes a lot of friction especially when researcher A’s ontology is unnatural to researcher B. See the comments to this for a long discussion on what counts as inner vs outer alignment.
Much of the disagreement comes down to research taste; see my comment here for an example of differences in opinion driven by taste.
That said, I’d be excited about debates between people with totally different views, e.g. Yudkowsky and Yann Lecun if it could happen...
The debate on this subject has been ongoing between individuals who are within or adjacent to the EA/LessWrong communities (see posts that other comments have linked and other links that are sure to follow). However, these debates often are highly insular and primarily are between people who share core assumptions about:
AGI being an existential risk with a high probability of occurring
Extinction via AGI having a significant probability of occurring within our lifetimes (next 10-50 years)
Other extinction risks (e.g pandemics or nuclear war) not likely manifesting prior to AGI and curtailing AI development such that AGI risk is no longer of relevance in any near-term timeline as a result
AGI being a more deadly existential risk than other existential risks (e.g pandemics or nuclear war)
AI alignment research being neglected and/or tractable
Current work on fairness and transparency improving methods for AI models not being particularly useful towards solving AI alignment
There are many other AI researchers and individuals from other relevant, adjacent disciplines that would disagree with all or most of these assumptions. Debates between that group and people within the EA/LessWrong community who would mostly agree with the above assumptions is something that is sorely lacking, save for some mud-flinging on Twitter between AI ethicists and AI alignment researchers.
There was a prominent debate between Eliezer Yudkowsky and Robin Hanson back in 2008 which is a part of the EA/rationalist communities’ origin story, link here: https://wiki.lesswrong.com/index.php?title=The_Hanson-Yudkowsky_AI-Foom_Debate
Prediction is hard and reading the debate from the vantage point of 14 years in the future it’s clear that in many ways the science and the argument has moved on, but it’s also clear that Eliezer made better predictions than Robin Hanson did, in a way that inclines me to try and learn as much of his worldview as possible so I can analyze other arguments through that frame.
This link could also be useful for learning how Yudkowsky & Hanson think about the issue: https://intelligence.org/ai-foom-debate
Essentially, Yudkowsky is very worried about AGI (‘we’re dead in 20-30 years’ worried) because he thinks that progress on AI overall will rapidly accelerate as AI helps us make further progress. Hanson was (is?) less worried.