I’m missing some important object-level knowledge, so it’s nice to see this spelled out more explicitly by someone with more exposure to the relevant community. Hope you get some engagement!
I know several people from outside EA (Ivy League, Ex-FANG, work in ML, startup, Bay Area) and they share the same “skepticism” (in quotes because it’s not the right word, their view is more negative).
I suspect one aspect of the problem is the sort of “Timnit Gebru”-style yelling and also sneering, often from the leftist community that is opposed to EA more broadly (but much of this leftist sneering was nucleated by the Bay Area communities).
This gives proponents of AI-safety an easy target, funneling online discourse into a cul de sac of tribalism. I suspect this dynamic is deliberately cultivated on both sides, a system ultimately supported by a lot of crypto/tech wealth. This leads to where we are today, where someone like Bruce (not to mention many young people) get confused.
I don’t follow Timnit closely, but I’m fairly unconvinced by much of what I think you’re referring to RE: “Timnit Gebru-style yelling / sneering”, and I don’t want to give the impression that my uncertainties are strongly influenced by this, or by AI-safety community pushback to those kinds of sneering. I’d be hesitant to agree that I share these views that you are attributing to me, since I don’t really know what you are referring to RE: folks who share “the same skepticism” (but some more negative version).
When I talk about uncertainty, some of these are really just the things that Nuno is pointing out in this post. Concrete examples of what some of these uncertainties look like in practice for me personally include:
I don’t have a good inside view on timelines, but when EY says our probability of survival is ~0% this seems like an extraordinary claim that doesn’t seem to be very well supported or argued for, and something I intuitively want to reject outright, but don’t have the object level expertise to meaningfully do so. I don’t know the extent to which EY’s views are representative or highly influential in current AI safety efforts, and I can imagine a world where there’s too much deferring going on. It seems like some within the community have similar thoughts.
When Metaculus suggests a median prediction of 2041 for AGI and a lower quartile prediction of 2030, I don’t feel like I have a good way of working out how much I should defer to this.
I was surprised to find that there seems to be a pretty strong correlation between people who work on AI safety and people who have been involved in either the EA/LW community, and there are many people outside of the EA/LW space who have a much lower P(doom) than those inside these spaces, even those who prima facie have strong incentives to have an accurate gauge of what P(doom) actually is.
To use a highly imperfect analogy, if being an EA doctor was highly correlated with a belief that [disease X] is made up, and if a significant majority of mainstream medicine believe [disease X] is a real phenomena, this would make me more uncertain about whether EAs are right, as I have to take into account the likelihood that a significant majority of mainstream medicine is wrong, against the possibility that EA doctors somehow have a way of interpreting noise / information in a much less biased way to non-EA doctors.
Of course, it’s plausible that EAs are on to something legit, and everyone else is underweighting the risks. But all I’m saying is that this is an added uncertainty.
It doesn’t help that, unlike COVID (another place EAs say they were ahead of the curve in), it’s much easier to retreat into some unfalsifiable position when it comes to P(doom) and AI safety, and also (very reasonably) not helpful to point to base rates.
While I wouldn’t suggest that 80,000 hours has “gone off the guardrails” in the way Nuno suggests CFAR did, on particularly uncharitable days I do get the sense that 80,000 hours feels more like a recruitment platform for AI and longtermist careers for a narrow readership that fit a particular philosophical view, which was not what it felt like back in ~2015 when I first read it (at that time, perhaps a more open, worldview-diversified career guide for a wider audience). Perhaps this reflects a deliberate shift in strategy, but I do get the sense that if this is the case, additional transparency about this shift would be helpful, since the target audience are often fairly impressionable, idealistic high school students or undergrads looking for a highly impactful career.
Another reason I’m slightly worried about this, and related to the earlier point about deferral, is that Ben Todd says: “For instance, I agree it’s really important for EA to attract people who are very open minded and curious, to keep EA alive as a question. And one way to do that is to broadcast ideas that aren’t widely accepted.” [emphasis added]
But my view is that this approach does not meaningfully differentiate between “attracting open-minded and curious people” VS “attracting highly deferring + easily influenced people”, and 80,000 hours may inadvertently contribute to the latter if not careful.
Relatedly, there have been some recent discussions around deference around things like AI timelines, as well as in the EA community generally.
------------
I don’t want this comment thread to get into a discussion about Timnit, because I think that detracts from engagement with Nuno’s post, but a quick comment:[1]
Regardless of one’s views on the quality of Timnit-style engagement, it’s useful to consider the extent the AI safety community may need to coordinate with people who might be influenced by that category of “skepticism”. This could either be at the object-level, or in terms of considering why some classes of “skepticism” seems to gain so much traction despite AI safety proponents’ disagreements with them. It might point to better ways for the AI safety community when it comes to communicating their ideas, or to help maximise buy-in from key stakeholders and focus on shared goals. (Thinking about some of these things have been directly influential in how some folks view ongoing UN engagement processes, for example).
Thank you for this useful content and explaining your beliefs.
don’t follow Timnit closely, but I’m fairly unconvinced by much of what I think you’re referring to RE: “Timnit Gebru-style yelling / sneering”, and I don’t want to give the impression that my uncertainties are strongly influenced by this, or by AI-safety community pushback to those kinds of sneering.
My comment is claiming a dynamic that is upstream of, and produces the information environment you are in. This produces your “skepticism” or “uncertainty”. To expand on this, without this dynamic, the facts and truth would be clearer and you would not be uncertain or feel the need to update your beliefs in response to a forum post.
My comment is not implying you are influenced by “Gebru-style” content directly. It is sort of implying the opposite/orthogonal. The fact you felt it necessary to distance yourself from Gebru several times in your comment, essentially because a comment mentioned her name, makes this very point itself.
don’t really know what you are referring to RE: folks who share “the same skepticism” (but some more negative version), but I’d be hesitant to agree that I share these views that you are attributing to me, since I don’t know what their views are.
Yes, I affirm that “skepticism” or “uncertainty” are my words. (I think the nature of this “skepticism” is secondary to the main point in my comment and the fact you brought this up is symptomatic of the point I made).
At the same time, I think the rest of your comment suggests my beliefs/representation of you was fair (e.g. sometimes uncharitably seeing 80KH as a recruitment platform for “AI/LT” would be consistent with skepticism).
In some sense, my comment is not a direct reply to you (you are even mentioned in third person). I’m OK with this, or even find the resulting response desirable, and it may have been hard to achieve in any other way.
Here is a post by Scott Alexander which I think might be pointing to a similar phenomenon as what you are hinting at. Money quote:
Consider the war on terror. They say that every time the United States bombs Pakistan or Afghanistan or somewhere, all we’re doing is radicalizing the young people there and making more terrorists. Those terrorists then go on to kill Americans, which makes Americans get very angry and call for more bombing of Pakistan and Afghanistan.
Taken as a meme, it’s a single parasite with two hosts and two forms. In an Afghan host, it appears in a form called ‘jihad’, and hijacks its host into killing himself in order to spread it to its second, American host. In the American host it morphs in a form called ‘the war on terror’, and it hijacks the Americans into giving their own lives (and tax dollars) to spread it back to its Afghan host in the form of bombs.
From the human point of view, jihad and the War on Terror are opposing forces. From the memetic point of view, they’re as complementary as caterpillars and butterflies. Instead of judging, we just note that somehow we accidentally created a replicator, and replicators are going to replicate until something makes them stop.
”I don’t have a good inside view on timelines, but when EY says our probability of survival is ~0% this seems like an extraordinary claim that doesn’t seem to be very well supported or argued for, and something I intuitively want to reject outright, but don’t have the object level expertise to meaningfully do so. I don’t know the extent to which EY’s views are representative or highly influential in current AI safety efforts, and I can imagine a world where there’s too much deferring going on. It seems like some within the community have similar thoughts.”
EY’s view of doom being basically certain are fairly marginal. They definitely are part of the conversation, and he certainly is not the only person who holds them. But most people who are actively working on AI safety see the odds of survival as much higher than roughly 0% -- and I think most people see the P(doom) as actually much lower than 80%.
The key motivating argument for AI safety being important, even if you think that EY’s model of the world might be false (though it also might be true) is that while it is easy to come up with plausible reasons to think that P(doom) is much less than 1, it is very hard to dismiss enough of the arguments for it to get p(doom) close to zero.
Yes, I think it’s good that there is basically consensus here on AGI doom being a serious problem; the argument seems to be one of degree. Even OP says p(AGI doom by 2070) ~ 10%.
I think Gary Marcus seems to play the role of an “anti-AI-doom” figurehead much more than Timnit Gebru. I don’t even know what his views on doom are, but he has established himself as a prominent critic of “AI is improving fast” views and seemingly gets lots of engagement from the safety community.
I also think Marcus’ criticisms aren’t very compelling, and so the discourse they generate isn’t terribly valuable. I think similarly of Gebru’s criticism (I think it’s worse than Marcus’, actually), but I just don’t think it has as much impact on the safety community.
I know several people from outside EA (Ivy League, Ex-FANG, work in ML, startup, Bay Area) and they share the same “skepticism” (in quotes because it’s not the right word, their view is more negative).
I suspect one aspect of the problem is the sort of “Timnit Gebru”-style yelling and also sneering, often from the leftist community that is opposed to EA more broadly (but much of this leftist sneering was nucleated by the Bay Area communities).
This gives proponents of AI-safety an easy target, funneling online discourse into a cul de sac of tribalism. I suspect this dynamic is deliberately cultivated on both sides, a system ultimately supported by a lot of crypto/tech wealth. This leads to where we are today, where someone like Bruce (not to mention many young people) get confused.
I don’t follow Timnit closely, but I’m fairly unconvinced by much of what I think you’re referring to RE: “Timnit Gebru-style yelling / sneering”, and I don’t want to give the impression that my uncertainties are strongly influenced by this, or by AI-safety community pushback to those kinds of sneering. I’d be hesitant to agree that I share these views that you are attributing to me, since I don’t really know what you are referring to RE: folks who share “the same skepticism” (but some more negative version).
When I talk about uncertainty, some of these are really just the things that Nuno is pointing out in this post. Concrete examples of what some of these uncertainties look like in practice for me personally include:
I don’t have a good inside view on timelines, but when EY says our probability of survival is ~0% this seems like an extraordinary claim that doesn’t seem to be very well supported or argued for, and something I intuitively want to reject outright, but don’t have the object level expertise to meaningfully do so. I don’t know the extent to which EY’s views are representative or highly influential in current AI safety efforts, and I can imagine a world where there’s too much deferring going on. It seems like some within the community have similar thoughts.
When Metaculus suggests a median prediction of 2041 for AGI and a lower quartile prediction of 2030, I don’t feel like I have a good way of working out how much I should defer to this.
I was surprised to find that there seems to be a pretty strong correlation between people who work on AI safety and people who have been involved in either the EA/LW community, and there are many people outside of the EA/LW space who have a much lower P(doom) than those inside these spaces, even those who prima facie have strong incentives to have an accurate gauge of what P(doom) actually is.
To use a highly imperfect analogy, if being an EA doctor was highly correlated with a belief that [disease X] is made up, and if a significant majority of mainstream medicine believe [disease X] is a real phenomena, this would make me more uncertain about whether EAs are right, as I have to take into account the likelihood that a significant majority of mainstream medicine is wrong, against the possibility that EA doctors somehow have a way of interpreting noise / information in a much less biased way to non-EA doctors.
Of course, it’s plausible that EAs are on to something legit, and everyone else is underweighting the risks. But all I’m saying is that this is an added uncertainty.
It doesn’t help that, unlike COVID (another place EAs say they were ahead of the curve in), it’s much easier to retreat into some unfalsifiable position when it comes to P(doom) and AI safety, and also (very reasonably) not helpful to point to base rates.
While I wouldn’t suggest that 80,000 hours has “gone off the guardrails” in the way Nuno suggests CFAR did, on particularly uncharitable days I do get the sense that 80,000 hours feels more like a recruitment platform for AI and longtermist careers for a narrow readership that fit a particular philosophical view, which was not what it felt like back in ~2015 when I first read it (at that time, perhaps a more open, worldview-diversified career guide for a wider audience). Perhaps this reflects a deliberate shift in strategy, but I do get the sense that if this is the case, additional transparency about this shift would be helpful, since the target audience are often fairly impressionable, idealistic high school students or undergrads looking for a highly impactful career.
Another reason I’m slightly worried about this, and related to the earlier point about deferral, is that Ben Todd says: “For instance, I agree it’s really important for EA to attract people who are very open minded and curious, to keep EA alive as a question. And one way to do that is to broadcast ideas that aren’t widely accepted.” [emphasis added]
But my view is that this approach does not meaningfully differentiate between “attracting open-minded and curious people” VS “attracting highly deferring + easily influenced people”, and 80,000 hours may inadvertently contribute to the latter if not careful.
Relatedly, there have been some recent discussions around deference around things like AI timelines, as well as in the EA community generally.
------------
I don’t want this comment thread to get into a discussion about Timnit, because I think that detracts from engagement with Nuno’s post, but a quick comment:[1]
Regardless of one’s views on the quality of Timnit-style engagement, it’s useful to consider the extent the AI safety community may need to coordinate with people who might be influenced by that category of “skepticism”. This could either be at the object-level, or in terms of considering why some classes of “skepticism” seems to gain so much traction despite AI safety proponents’ disagreements with them. It might point to better ways for the AI safety community when it comes to communicating their ideas, or to help maximise buy-in from key stakeholders and focus on shared goals. (Thinking about some of these things have been directly influential in how some folks view ongoing UN engagement processes, for example).
Which should not be read as an endorsement of Timnit-style engagement.
Thank you for this useful content and explaining your beliefs.
My comment is claiming a dynamic that is upstream of, and produces the information environment you are in. This produces your “skepticism” or “uncertainty”. To expand on this, without this dynamic, the facts and truth would be clearer and you would not be uncertain or feel the need to update your beliefs in response to a forum post.
My comment is not implying you are influenced by “Gebru-style” content directly. It is sort of implying the opposite/orthogonal. The fact you felt it necessary to distance yourself from Gebru several times in your comment, essentially because a comment mentioned her name, makes this very point itself.
Yes, I affirm that “skepticism” or “uncertainty” are my words. (I think the nature of this “skepticism” is secondary to the main point in my comment and the fact you brought this up is symptomatic of the point I made).
At the same time, I think the rest of your comment suggests my beliefs/representation of you was fair (e.g. sometimes uncharitably seeing 80KH as a recruitment platform for “AI/LT” would be consistent with skepticism).
In some sense, my comment is not a direct reply to you (you are even mentioned in third person). I’m OK with this, or even find the resulting response desirable, and it may have been hard to achieve in any other way.
Here is a post by Scott Alexander which I think might be pointing to a similar phenomenon as what you are hinting at. Money quote:
Object level point:
”I don’t have a good inside view on timelines, but when EY says our probability of survival is ~0% this seems like an extraordinary claim that doesn’t seem to be very well supported or argued for, and something I intuitively want to reject outright, but don’t have the object level expertise to meaningfully do so. I don’t know the extent to which EY’s views are representative or highly influential in current AI safety efforts, and I can imagine a world where there’s too much deferring going on. It seems like some within the community have similar thoughts.”
EY’s view of doom being basically certain are fairly marginal. They definitely are part of the conversation, and he certainly is not the only person who holds them. But most people who are actively working on AI safety see the odds of survival as much higher than roughly 0% -- and I think most people see the P(doom) as actually much lower than 80%.
The key motivating argument for AI safety being important, even if you think that EY’s model of the world might be false (though it also might be true) is that while it is easy to come up with plausible reasons to think that P(doom) is much less than 1, it is very hard to dismiss enough of the arguments for it to get p(doom) close to zero.
Yes, I think it’s good that there is basically consensus here on AGI doom being a serious problem; the argument seems to be one of degree. Even OP says p(AGI doom by 2070) ~ 10%.
I think Gary Marcus seems to play the role of an “anti-AI-doom” figurehead much more than Timnit Gebru. I don’t even know what his views on doom are, but he has established himself as a prominent critic of “AI is improving fast” views and seemingly gets lots of engagement from the safety community.
I also think Marcus’ criticisms aren’t very compelling, and so the discourse they generate isn’t terribly valuable. I think similarly of Gebru’s criticism (I think it’s worse than Marcus’, actually), but I just don’t think it has as much impact on the safety community.
Brutal comment, love it