Just wanted to flag the group is heavily selected for belief alignment with something like “EA/Constellation/Trajan House” views, and “AI enabled human takeovers” was promoted as agenda to prioritize in multiple widely read memos by high statues people in the community (which the organisers prioritized in the reading list).
I dislike the “echo chambre” effect where the steps are: - invite people partially based on alignment with the idea cluster - tell them to read memos advocating something written by some of the most central people in the cluster - poll attendees - results are framed as “leaders and key thinkers in the x-risk and AI safety communities agree”
It is in some sense useful, but in my view the cluster of people invited represents maybe ~30% of thinking about x-risk and AI safety, and its mostly an amplification of existing voices.
”The slight lean against misaligned AI takeover resources is perhaps the most surprising result for this audience, and merits closer examination.”
This is unsurprising given the marginal and somewhat confusing nature of the question. My wild guess is— some attendees voted for everything; it is unclear what does it mean on the margin, probably to grow everything, and prioritize more neglected topics? - some attendees understood the marginal question as “assuming fixed pie, how to change the allocation”—with this understanding you need to assign something negative weight for consistency
Just wanted to flag the group is heavily selected for belief alignment with something like “EA/Constellation/Trajan House” views
As an event focused on x-risk, yes, I think this is fair.
“AI enabled human takeovers” was promoted as agenda to prioritize in multiple widely read memos by high statues people in the community (which the organisers prioritized in the reading list).
It’s true that:
The agenda featured some talks emphasising risks from AI-enabled human takeover.
Some of the most popular memos also emphasised this risk category.
Some people took the survey after reading the agenda and the memos.
But I don’t think attendees were as strongly influenced as you seem to imply:
We highlighted some memos to read at the beginning, but soon after launching the memo platform, we prioritized memos by votes from attendees. Memos making the case for more emphasis on risks from aligned AI were heavily upvoted, and some memos that we highlighted from the beginning received fewer upvotes.
The survey was in part motivated by disagreements I’d heard about how the AIS community was allocating resources. While I’m sure some attendees were influenced by information they recently encountered, many will have thought about these questions in advance of the survey.
I don’t have the full data, but I think it’s likely that many attendees completed the survey before engaging with the memos and before the full agenda was published.
I do think you’re pointing to a real effect to be aware of, and thanks for pointing it out, but I don’t think it’s as significant as you make out (though maybe you don’t think it’s super significant).
Results are framed as “leaders and key thinkers in the x-risk and AI safety communities agree”
I think the areas of broad consensus accurately (if roughly) reflect the data we have here and what we saw in memos. FWIW, my overall takeaway from running this survey is that leaders and key thinkers have a wide range of views and I think this post captures and conveys this.
As an event focused on x-risk, yes, I think this is fair.
This seems like a misinterpretation of Jan’s point. There are multiple intellectual clusters which at least claim to care about x-risk which aren’t well-described as the “EA/Constellation/Trajan House” cluster. The main ones which come to mind are:
The MIRI cluster
The Pause AI cluster
The academic ML cluster
The multi-agent/sociopolitical safety cluster (which isn’t very well-defined right now but I’d put both Jan and myself in this, broadly speaking)
The Anthropic cluster (which e.g. is more positive on racing than the EA/Constellation cluster, though I’m not claiming that there’s a coherent intellectual worldview behind this)
I would only describe a few people in each cluster as actual thought leaders or key thinkers. So compared with Jan my concern is less about who gets invited, and more that sampling any gathering as large as the Summit averages together responses from people with too wide a range of levels of leadership to be accurately described as “AI safety leaders”.
Yeah, I expected as much. Though as per my comment above, I’m much more concerned about representation of thought leaders. A better proxy for intellectual diversity is something like “are the few people from each of these clusters who are the biggest critics of the consensus view invited?” E.g. for the Pause AI cluster that’d probably be Holly; for the MIRI cluster that’d probably be Yudkowsky and Habryka; for the academic ML cluster that’d probably be Dan Hendrycks; for the sociopolitical safety cluster that’d probably be Ben Hoffman and Michael Vassar.
I don’t know exactly who was invited but I expect that the Summit gets a medium score on this metric: not great, not terrible.
Just wanted to flag the group is heavily selected for belief alignment with something like “EA/Constellation/Trajan House” views, and “AI enabled human takeovers” was promoted as agenda to prioritize in multiple widely read memos by high statues people in the community (which the organisers prioritized in the reading list).
I dislike the “echo chambre” effect where the steps are:
- invite people partially based on alignment with the idea cluster
- tell them to read memos advocating something written by some of the most central people in the cluster
- poll attendees
- results are framed as “leaders and key thinkers in the x-risk and AI safety communities agree”
It is in some sense useful, but in my view the cluster of people invited represents maybe ~30% of thinking about x-risk and AI safety, and its mostly an amplification of existing voices.
”The slight lean against misaligned AI takeover resources is perhaps the most surprising result for this audience, and merits closer examination.”
This is unsurprising given the marginal and somewhat confusing nature of the question. My wild guess is—
some attendees voted for everything; it is unclear what does it mean on the margin, probably to grow everything, and prioritize more neglected topics?
- some attendees understood the marginal question as “assuming fixed pie, how to change the allocation”—with this understanding you need to assign something negative weight for consistency
Thanks Jan, I appreciate the pushback.
As an event focused on x-risk, yes, I think this is fair.
It’s true that:
The agenda featured some talks emphasising risks from AI-enabled human takeover.
Some of the most popular memos also emphasised this risk category.
Some people took the survey after reading the agenda and the memos.
But I don’t think attendees were as strongly influenced as you seem to imply:
We highlighted some memos to read at the beginning, but soon after launching the memo platform, we prioritized memos by votes from attendees. Memos making the case for more emphasis on risks from aligned AI were heavily upvoted, and some memos that we highlighted from the beginning received fewer upvotes.
The survey was in part motivated by disagreements I’d heard about how the AIS community was allocating resources. While I’m sure some attendees were influenced by information they recently encountered, many will have thought about these questions in advance of the survey.
I don’t have the full data, but I think it’s likely that many attendees completed the survey before engaging with the memos and before the full agenda was published.
I do think you’re pointing to a real effect to be aware of, and thanks for pointing it out, but I don’t think it’s as significant as you make out (though maybe you don’t think it’s super significant).
I think the areas of broad consensus accurately (if roughly) reflect the data we have here and what we saw in memos. FWIW, my overall takeaway from running this survey is that leaders and key thinkers have a wide range of views and I think this post captures and conveys this.
This seems like a misinterpretation of Jan’s point. There are multiple intellectual clusters which at least claim to care about x-risk which aren’t well-described as the “EA/Constellation/Trajan House” cluster. The main ones which come to mind are:
The MIRI cluster
The Pause AI cluster
The academic ML cluster
The multi-agent/sociopolitical safety cluster (which isn’t very well-defined right now but I’d put both Jan and myself in this, broadly speaking)
The Anthropic cluster (which e.g. is more positive on racing than the EA/Constellation cluster, though I’m not claiming that there’s a coherent intellectual worldview behind this)
I would only describe a few people in each cluster as actual thought leaders or key thinkers. So compared with Jan my concern is less about who gets invited, and more that sampling any gathering as large as the Summit averages together responses from people with too wide a range of levels of leadership to be accurately described as “AI safety leaders”.
In case it’s helpful: as an attendee of this event I would say ~2.5 of these 5 were like “decently” represented (not saying that’s sufficient)
Yeah, I expected as much. Though as per my comment above, I’m much more concerned about representation of thought leaders. A better proxy for intellectual diversity is something like “are the few people from each of these clusters who are the biggest critics of the consensus view invited?” E.g. for the Pause AI cluster that’d probably be Holly; for the MIRI cluster that’d probably be Yudkowsky and Habryka; for the academic ML cluster that’d probably be Dan Hendrycks; for the sociopolitical safety cluster that’d probably be Ben Hoffman and Michael Vassar.
I don’t know exactly who was invited but I expect that the Summit gets a medium score on this metric: not great, not terrible.