Thanks for your feedback on including the eventās focus as a limitation of the survey. Thatās something weāll consider if we run a similar survey and decide to publish the data next year.
Some of the organizations you listed had representatives invited who either did not attend or did not fill out the survey. (The survey was emailed to all invitees, and some of those who filled it out didnāt attend the event.) If everyone invited had filled it out, I think the list of represented organizations would look more diverse by your criteria.
Thanks Aaron. Glad to hear the invitee list included a broader list of organizations, and that youāll consider a more explicit discussion of potential selection bias effects going forward.
(I was the interim director of CEA during Leaders Forum, and Iām now the executive director.)
I think that CEA has a history of pushing longtermism in somewhat underhand ways (e.g. I think that I made a mistake when I published an āEA handbookā without sufficiently consulting non-longtermist researchers, and in a way that probably over-represented AI safety and under-represented material outside of traditional EA cause areas, resulting in a product that appeared to represent EA, without accurately doing so). Given this background, I think itās reasonable to be suspicious of CEAās cause prioritisation.
(Iāll be writing more about this in the future, and it feels a bit odd to get into this in a comment when itās a major-ish update to CEAās strategy, but I think itās better to share more rather than less.) In the future, Iād like CEA to take a more agnostic approach to cause prioritisation, trying to construct non-gameable mechanisms for making decisions about how much we talk about different causes. An example of how this might work is that we might pay for an independent contractor to try to figure out who has spent more than two years full time thinking about cause prioritization, and then surveying those people. Obviously that project would be complicatedāitās hard to figure out exactly what ācause prioā means, it would be important to reach out through diverse networks to make sure there arenāt network biases etc.
Anyway, given this background of pushing longtermism, I think itās reasonable to be skeptical of CEAās approach on this sort of thing.
When I look at the list of organizations that were surveyed, it doesnāt look like the list of organizations most involved in movement building and coordination. It looks much more like a specific subset of that type of org: those focused on longtermism or x-risk (especially AI) and based in one of the main hubs (London accounts for ~50% of respondents, and the Bay accounts for ~30%).* Those that prioritize global poverty, and to a lesser extent animal welfare, seem notably missing. Itās possible the list of organizations that didnāt respond or werenāt named looks a lot different, but if thatās the case it seems worth calling attention to and possibly trying to rectify (e.g. did you email the survey to anyone or was it all done in person at the Leaders Forum?)
I think youāre probably right that there are some biases here. How the invite process worked this year was that Amy Labenz, who runs the event, draws up a longlist of potential attendees (asking some external advisors for suggestions about who should be invited). Then Amy, Julia Wise, and I voted yes/āno/āmaybe on all of the individuals on the longlist (often adding comments). Amy made a final call about who to invite, based on those votes. I expect that all of this means that the final invite list is somewhat biased by our networks, and some background assumptions we have about individuals and orgs.
Given this, I think that it would be fair to view the attendees of the event as āsome people who CEA staff think it would be useful to get together for a few daysā rather than āthe definitive list of EA leadersā. I think that we were also somewhat loose about what the criteria for inviting people should be, and Iād like us to be a bit clearer on that in the future (see a couple of paragraphs below). Given this, I think that calling the event āEA Leaders Forumā is probably a mistake, but others on the team think that changing the name could be confusing and have transition costsāweāre still talking about this, and havenāt reached resolution about whether weāll keep the name for next year.
I also think CEA made some mistakes in the way we framed this post (not just the author, since it went through other readers before publication.) I think the post kind of frames this as āEA leaders think Xā, which I expect would be the sort of thing that lots of EAs should update on. (Even though I think it does try to explicitly disavow this interpretation (see the section on āWhat this data does and does not representā, I think the title suggests something thatās more like āEA leaders think these are the prioritiesāprobably you should update towards these being the prioritiesā). I think that the reality is more like āsome people that CEA staff think itās useful to get together for an event think Xā, which is something that people should update on less.
Weāre currently at a team retreat where weāre talking more about what the goals of the event should be in the future. I think that itās possible that the event looks pretty different in future years, and weāre not yet sure how. But I think that whatever we decide, we should think more carefully about the criteria for attendees, and that will include thinking carefully about the approach to cause prioritization.
Thank you for taking the time to respond, Max. I appreciate your engagement, your explanation of how the invitation process worked this year, and your willingness to acknowledge that CEA may have historically been too aggressive in how it has pushed longtermism and how it has framed the results of past surveys.
In the future, Iād like CEA to take a more agnostic approach to cause prioritisation, trying to construct non-gameable mechanisms for making decisions about how much we talk about different causes.
Very glad to hear this. As you note, implementing this sort of thing in practice can be tricky. As CEA starts designing new mechanisms, Iād love to see you gather input (as early possible) from people who have expressed concern about CEAās representativeness in the past (Iād be happy to offer opinions if youād like). These also might be good people to serve as āexternal advisorsā who generate suggestions for the invite list.
Good luck with the retreat! I look forward to seeing your strategy update once thatās written up.
Some of the organizations you listed had representatives invited who either did not attend or did not fill out the survey. [...] If everyone invited had filled it out, I think the list of represented organizations would look more diverse by your criteria.
Depends a bit on how much you mean to stretch the word āsomeāā¦ This is false as far as I can tell.. at best I would describe your comment as highly misleading.
Iām not sure what you mean, Peter, but Iāll try to be more clear. Of the seven organizations listed in the comment to which I replied, three of them had people invited, according to the list of people who were recorded as having been sent the invite email.
How did you interpret the word āsomeā? Is there another sense in which you saw the comment as misleading?
Iām sorry. I was reading uncharitably and wrote too quickly. Your latest response sounds clear and fair to me. Thanks for providing the numbers and Iām sorry for misjudging the situation.
Thanks for your feedback on including the eventās focus as a limitation of the survey. Thatās something weāll consider if we run a similar survey and decide to publish the data next year.
Some of the organizations you listed had representatives invited who either did not attend or did not fill out the survey. (The survey was emailed to all invitees, and some of those who filled it out didnāt attend the event.) If everyone invited had filled it out, I think the list of represented organizations would look more diverse by your criteria.
Thanks Aaron. Glad to hear the invitee list included a broader list of organizations, and that youāll consider a more explicit discussion of potential selection bias effects going forward.
(I was the interim director of CEA during Leaders Forum, and Iām now the executive director.)
I think that CEA has a history of pushing longtermism in somewhat underhand ways (e.g. I think that I made a mistake when I published an āEA handbookā without sufficiently consulting non-longtermist researchers, and in a way that probably over-represented AI safety and under-represented material outside of traditional EA cause areas, resulting in a product that appeared to represent EA, without accurately doing so). Given this background, I think itās reasonable to be suspicious of CEAās cause prioritisation.
(Iāll be writing more about this in the future, and it feels a bit odd to get into this in a comment when itās a major-ish update to CEAās strategy, but I think itās better to share more rather than less.) In the future, Iād like CEA to take a more agnostic approach to cause prioritisation, trying to construct non-gameable mechanisms for making decisions about how much we talk about different causes. An example of how this might work is that we might pay for an independent contractor to try to figure out who has spent more than two years full time thinking about cause prioritization, and then surveying those people. Obviously that project would be complicatedāitās hard to figure out exactly what ācause prioā means, it would be important to reach out through diverse networks to make sure there arenāt network biases etc.
Anyway, given this background of pushing longtermism, I think itās reasonable to be skeptical of CEAās approach on this sort of thing.
I think youāre probably right that there are some biases here. How the invite process worked this year was that Amy Labenz, who runs the event, draws up a longlist of potential attendees (asking some external advisors for suggestions about who should be invited). Then Amy, Julia Wise, and I voted yes/āno/āmaybe on all of the individuals on the longlist (often adding comments). Amy made a final call about who to invite, based on those votes. I expect that all of this means that the final invite list is somewhat biased by our networks, and some background assumptions we have about individuals and orgs.
Given this, I think that it would be fair to view the attendees of the event as āsome people who CEA staff think it would be useful to get together for a few daysā rather than āthe definitive list of EA leadersā. I think that we were also somewhat loose about what the criteria for inviting people should be, and Iād like us to be a bit clearer on that in the future (see a couple of paragraphs below). Given this, I think that calling the event āEA Leaders Forumā is probably a mistake, but others on the team think that changing the name could be confusing and have transition costsāweāre still talking about this, and havenāt reached resolution about whether weāll keep the name for next year.
I also think CEA made some mistakes in the way we framed this post (not just the author, since it went through other readers before publication.) I think the post kind of frames this as āEA leaders think Xā, which I expect would be the sort of thing that lots of EAs should update on. (Even though I think it does try to explicitly disavow this interpretation (see the section on āWhat this data does and does not representā, I think the title suggests something thatās more like āEA leaders think these are the prioritiesāprobably you should update towards these being the prioritiesā). I think that the reality is more like āsome people that CEA staff think itās useful to get together for an event think Xā, which is something that people should update on less.
Weāre currently at a team retreat where weāre talking more about what the goals of the event should be in the future. I think that itās possible that the event looks pretty different in future years, and weāre not yet sure how. But I think that whatever we decide, we should think more carefully about the criteria for attendees, and that will include thinking carefully about the approach to cause prioritization.
Thank you for taking the time to respond, Max. I appreciate your engagement, your explanation of how the invitation process worked this year, and your willingness to acknowledge that CEA may have historically been too aggressive in how it has pushed longtermism and how it has framed the results of past surveys.
Very glad to hear this. As you note, implementing this sort of thing in practice can be tricky. As CEA starts designing new mechanisms, Iād love to see you gather input (as early possible) from people who have expressed concern about CEAās representativeness in the past (Iād be happy to offer opinions if youād like). These also might be good people to serve as āexternal advisorsā who generate suggestions for the invite list.
Good luck with the retreat! I look forward to seeing your strategy update once thatās written up.
Depends a bit on how much you mean to stretch the word āsomeāā¦ This is false as far as I can tell.. at best I would describe your comment as highly misleading.
Iām not sure what you mean, Peter, but Iāll try to be more clear. Of the seven organizations listed in the comment to which I replied, three of them had people invited, according to the list of people who were recorded as having been sent the invite email.
How did you interpret the word āsomeā? Is there another sense in which you saw the comment as misleading?
Iām sorry. I was reading uncharitably and wrote too quickly. Your latest response sounds clear and fair to me. Thanks for providing the numbers and Iām sorry for misjudging the situation.