What criteria were used to decide which orgs/âindividuals should be invited?
A small team of CEA staffers (I was not one of them) selected an initial invite list (58 people). At present, we see Leaders Forum as an event focused on movement building and coordination. We focus on inviting people who play a role in trying to shape the overall direction of the EA movement (whatever cause area they focus on), rather than people who mostly focus on direct research within a particular cause area. As youâd imagine, this distinction can be somewhat fuzzy, but thatâs the mindset with which CEA approaches invites (though other factors can play a role).
To give a specific example, while the Against Malaria Foundation is an important charity for people in EA who want to support global health, Iâm not aware of any AMF staffers who have both a strong interest in the EA movement as a whole and some relevant movement-building experience. I donât think that, say, Rob Mather (AMFâs CEO), or a representative from the Gates Foundation, would get much value from the vast majority of conversations/âsessions at the event.
I should also note that the event has gotten a bit smaller over time. The first Leaders Forum (2016) had ~100 invitees and 62 attendees and wasnât as focused on any particular topic. The next year, we shifted to a stronger focus on movement-building in particular (including community health, movement strategy, and risks to EA), which naturally led to a smaller, more focused invite list.
As with any other CEA program, Leaders Forum may continue to change over time; we arenât yet sure how many people weâll invite next year.
Because of this, I donât think it really makes sense to aggregate data over all cause areas.
I mostly agree! For several reasons, I wouldnât put much stock in the cause-area data. Most participants likely arrived at their answers very quickly, and the numbers are of course dependent on the backgrounds of the people who both (a) were invited and (b) took the time to respond. However, because we did conduct the survey, it felt appropriate to share what information came out of it, even if the value of that information is limited.
I do, however, think itâs good to have this information to check whether certain âextremeâ conditions are present â for example, it would have been surprising and notable if wild animal welfare had wound up with a median score of â0â, as that would seem to imply that most attendees think the cause doesnât matter at all.
As stated, some orgs are small and so were not named, but still responded. Maybe a breakdown by the cause area for all the respondents would be more useful with the data you have already?
Given the limited utility of the prioritization data, I donât know how much more helpful a cause-area breakdown would be. (Also, many if not most respondents currently work on more than one of the areas mentioned, but not necessarily with an even split between areas â any number I came up with would be fairly subjective.)
It seems weird to me that DeepMind and the Good Food Institute are on this list, but not, say, the Against Malaria Foundation, GiveDirectly, Giving What We Can, J-PAL, IPA, or the Humane League.
In addition to what I noted above about the types of attendees we aimed for, Iâll note that the list of respondent organizations doesnât perfectly match who we invited; quite a few other organizations had invitees who didnât fill out the survey. However, I will note that Giving What We Can (which is a project of CEA and was represented by CEA staff) did have representatives there.
As for organizations like DeepMind or GFI: While some of the orgs on the list are focused on a single narrow area, the employees we invited often had backgrounds in EA movement building and (in some cases) direct experience in other cause areas. (One invitee has run at least three major EA-aligned projects in three different areas.)
This wasnât necessarily the case for every attendee (as I mentioned, we considered factors other than community-building experience), but itâs an important reason that the org list looks the way it does.
At present, we see Leaders Forum as an event focused on movement building and coordination. We focus on inviting people who play a role in trying to shape the overall direction of the EA movement (whatever cause area they focus on), rather than people who mostly focus on direct research within a particular cause area. As youâd imagine, this distinction can be somewhat fuzzy, but thatâs the mindset with which CEA approaches invites (though other factors can play a role).
I really wish this had been included in the OP, in the section that discusses the weaknesses of the data. That section seems to frame the data as a more or less random subset of leaders of EA organizations (âThese results shouldnât be taken as an authoritative or consensus view of effective altruism as a whole. They donât represent everyone in EA, or even every leader of an EA organization.â)
When I look at the list of organizations that were surveyed, it doesnât look like the list of organizations most involved in movement building and coordination. It looks much more like a specific subset of that type of org: those focused on longtermism or x-risk (especially AI) and based in one of the main hubs (London accounts for ~50% of respondents, and the Bay accounts for ~30%).* Those that prioritize global poverty, and to a lesser extent animal welfare, seem notably missing. Itâs possible the list of organizations that didnât respond or werenât named looks a lot different, but if thatâs the case it seems worth calling attention to and possibly trying to rectify (e.g. did you email the survey to anyone or was it all done in person at the Leaders Forum?)
Some of the organizations Iâd have expected to see included, even if the focus was movement building/âcoordination: GiveWell (strategy/âgrowth staff, not pure research staff), LEAN, Charity Entrepreneurship, Vegan Outreach, Rethink Priorities, One for the World, Founders Pledge, etc. Most EAs would see these as EA organizations involved to some degree with movement building. But weâre not learning what they think, while we are apparently hearing from at least one org/âperson who âwant to avoid being connected explicitly to the EA movementâfor example, if almost all their work happens in non-EA circles, where EA might have a mixed reputation.â
Iâm worried that people who read this report are likely to misinterpret the data being presented as more broadly representative than it actually is (e.g. the implications of respondents believing ~30% of EA resources should go to AI work over the next 5 years are radically different if those respondents disproportionally omit people who favor other causes). I have the same concerns about this survey was presented as Jacy Reese expressed about how the leaders survey from 2 years ago (which also captured a narrow set of opinions) was presented:
My main general thought here is just that we shouldnât depend on so much from the reader. Most people, even most thoughtful EAs, wonât read in full and come up with all the qualifications on their own, so itâs important for article writers to include those themselves, and to include those upfront and center in their articles.
Lastly, Iâll note that thereâs a certain irony in surveying only a narrow set of people, given that even among those respondents: âThe most common theme in these answers [about problems in the EA community] seems to be the desire for EA to be more inclusive and welcoming. Respondents saw a lot of room for improvement on intellectual diversity, humility, and outreach, whether to distinct groups with different views or to the general population.â I suspect if a more diverse set of leaders had been surveyed, this theme would have been expressed even more strongly.
* GFI and Effective Giving both have London offices, but Iâve assumed their respondents were from other locations.
I agree with you that the orgs you mentioned (e.g. One for the World) are more focused on movement building than some of the other orgs that were invited.
I talked with Amy Labenz (who organized the event) in the course of writing my original reply. We want to clarify that when we said: âAt present, we see Leaders Forum as an event focused on movement building and coordination. We focus on inviting people who play a role in trying to shape the overall direction of the EA movement (whatever cause area they focus on)â. We didnât mean to over-emphasize âmovement buildingâ (in the sense of âbringing more people to EAâ) relative to âpeople shaping the overall direction of the EA movement (in the sense of âfiguring out what the movement should prioritize, growth or otherwiseâ).
My use of the term âmovement buildingâ was my slight misinterpretation of an internal document written by Amy. The eventâs purpose was closer to discussing the goals, health, and trajectory of the movement (e.g. âhow should we prioritize growth vs. other things?â) than discussing how to grow/âbuild the movement (e.g. âhow should we introduce EA to new people?â)
Thanks Aaron, thatâs a helpful clarification. Focusing on âpeople shaping the overall direction of the EA movementâ rather than just movement building seems like a sensible decision. But one drawback is that coming up with a list of those people is a much more subjective (and network-reliant) exercise than, for example, making a list of movement building organizations and inviting representatives from each of them.
Thanks for your feedback on including the eventâs focus as a limitation of the survey. Thatâs something weâll consider if we run a similar survey and decide to publish the data next year.
Some of the organizations you listed had representatives invited who either did not attend or did not fill out the survey. (The survey was emailed to all invitees, and some of those who filled it out didnât attend the event.) If everyone invited had filled it out, I think the list of represented organizations would look more diverse by your criteria.
Thanks Aaron. Glad to hear the invitee list included a broader list of organizations, and that youâll consider a more explicit discussion of potential selection bias effects going forward.
(I was the interim director of CEA during Leaders Forum, and Iâm now the executive director.)
I think that CEA has a history of pushing longtermism in somewhat underhand ways (e.g. I think that I made a mistake when I published an âEA handbookâ without sufficiently consulting non-longtermist researchers, and in a way that probably over-represented AI safety and under-represented material outside of traditional EA cause areas, resulting in a product that appeared to represent EA, without accurately doing so). Given this background, I think itâs reasonable to be suspicious of CEAâs cause prioritisation.
(Iâll be writing more about this in the future, and it feels a bit odd to get into this in a comment when itâs a major-ish update to CEAâs strategy, but I think itâs better to share more rather than less.) In the future, Iâd like CEA to take a more agnostic approach to cause prioritisation, trying to construct non-gameable mechanisms for making decisions about how much we talk about different causes. An example of how this might work is that we might pay for an independent contractor to try to figure out who has spent more than two years full time thinking about cause prioritization, and then surveying those people. Obviously that project would be complicatedâitâs hard to figure out exactly what âcause prioâ means, it would be important to reach out through diverse networks to make sure there arenât network biases etc.
Anyway, given this background of pushing longtermism, I think itâs reasonable to be skeptical of CEAâs approach on this sort of thing.
When I look at the list of organizations that were surveyed, it doesnât look like the list of organizations most involved in movement building and coordination. It looks much more like a specific subset of that type of org: those focused on longtermism or x-risk (especially AI) and based in one of the main hubs (London accounts for ~50% of respondents, and the Bay accounts for ~30%).* Those that prioritize global poverty, and to a lesser extent animal welfare, seem notably missing. Itâs possible the list of organizations that didnât respond or werenât named looks a lot different, but if thatâs the case it seems worth calling attention to and possibly trying to rectify (e.g. did you email the survey to anyone or was it all done in person at the Leaders Forum?)
I think youâre probably right that there are some biases here. How the invite process worked this year was that Amy Labenz, who runs the event, draws up a longlist of potential attendees (asking some external advisors for suggestions about who should be invited). Then Amy, Julia Wise, and I voted yes/âno/âmaybe on all of the individuals on the longlist (often adding comments). Amy made a final call about who to invite, based on those votes. I expect that all of this means that the final invite list is somewhat biased by our networks, and some background assumptions we have about individuals and orgs.
Given this, I think that it would be fair to view the attendees of the event as âsome people who CEA staff think it would be useful to get together for a few daysâ rather than âthe definitive list of EA leadersâ. I think that we were also somewhat loose about what the criteria for inviting people should be, and Iâd like us to be a bit clearer on that in the future (see a couple of paragraphs below). Given this, I think that calling the event âEA Leaders Forumâ is probably a mistake, but others on the team think that changing the name could be confusing and have transition costsâweâre still talking about this, and havenât reached resolution about whether weâll keep the name for next year.
I also think CEA made some mistakes in the way we framed this post (not just the author, since it went through other readers before publication.) I think the post kind of frames this as âEA leaders think Xâ, which I expect would be the sort of thing that lots of EAs should update on. (Even though I think it does try to explicitly disavow this interpretation (see the section on âWhat this data does and does not representâ, I think the title suggests something thatâs more like âEA leaders think these are the prioritiesâprobably you should update towards these being the prioritiesâ). I think that the reality is more like âsome people that CEA staff think itâs useful to get together for an event think Xâ, which is something that people should update on less.
Weâre currently at a team retreat where weâre talking more about what the goals of the event should be in the future. I think that itâs possible that the event looks pretty different in future years, and weâre not yet sure how. But I think that whatever we decide, we should think more carefully about the criteria for attendees, and that will include thinking carefully about the approach to cause prioritization.
Thank you for taking the time to respond, Max. I appreciate your engagement, your explanation of how the invitation process worked this year, and your willingness to acknowledge that CEA may have historically been too aggressive in how it has pushed longtermism and how it has framed the results of past surveys.
In the future, Iâd like CEA to take a more agnostic approach to cause prioritisation, trying to construct non-gameable mechanisms for making decisions about how much we talk about different causes.
Very glad to hear this. As you note, implementing this sort of thing in practice can be tricky. As CEA starts designing new mechanisms, Iâd love to see you gather input (as early possible) from people who have expressed concern about CEAâs representativeness in the past (Iâd be happy to offer opinions if youâd like). These also might be good people to serve as âexternal advisorsâ who generate suggestions for the invite list.
Good luck with the retreat! I look forward to seeing your strategy update once thatâs written up.
Some of the organizations you listed had representatives invited who either did not attend or did not fill out the survey. [...] If everyone invited had filled it out, I think the list of represented organizations would look more diverse by your criteria.
Depends a bit on how much you mean to stretch the word âsomeâ⌠This is false as far as I can tell.. at best I would describe your comment as highly misleading.
Iâm not sure what you mean, Peter, but Iâll try to be more clear. Of the seven organizations listed in the comment to which I replied, three of them had people invited, according to the list of people who were recorded as having been sent the invite email.
How did you interpret the word âsomeâ? Is there another sense in which you saw the comment as misleading?
Iâm sorry. I was reading uncharitably and wrote too quickly. Your latest response sounds clear and fair to me. Thanks for providing the numbers and Iâm sorry for misjudging the situation.
A small team of CEA staffers (I was not one of them) selected an initial invite list (58 people). At present, we see Leaders Forum as an event focused on movement building and coordination. We focus on inviting people who play a role in trying to shape the overall direction of the EA movement (whatever cause area they focus on), rather than people who mostly focus on direct research within a particular cause area. As youâd imagine, this distinction can be somewhat fuzzy, but thatâs the mindset with which CEA approaches invites (though other factors can play a role).
To give a specific example, while the Against Malaria Foundation is an important charity for people in EA who want to support global health, Iâm not aware of any AMF staffers who have both a strong interest in the EA movement as a whole and some relevant movement-building experience. I donât think that, say, Rob Mather (AMFâs CEO), or a representative from the Gates Foundation, would get much value from the vast majority of conversations/âsessions at the event.
I should also note that the event has gotten a bit smaller over time. The first Leaders Forum (2016) had ~100 invitees and 62 attendees and wasnât as focused on any particular topic. The next year, we shifted to a stronger focus on movement-building in particular (including community health, movement strategy, and risks to EA), which naturally led to a smaller, more focused invite list.
As with any other CEA program, Leaders Forum may continue to change over time; we arenât yet sure how many people weâll invite next year.
I mostly agree! For several reasons, I wouldnât put much stock in the cause-area data. Most participants likely arrived at their answers very quickly, and the numbers are of course dependent on the backgrounds of the people who both (a) were invited and (b) took the time to respond. However, because we did conduct the survey, it felt appropriate to share what information came out of it, even if the value of that information is limited.
I do, however, think itâs good to have this information to check whether certain âextremeâ conditions are present â for example, it would have been surprising and notable if wild animal welfare had wound up with a median score of â0â, as that would seem to imply that most attendees think the cause doesnât matter at all.
Given the limited utility of the prioritization data, I donât know how much more helpful a cause-area breakdown would be. (Also, many if not most respondents currently work on more than one of the areas mentioned, but not necessarily with an even split between areas â any number I came up with would be fairly subjective.)
In addition to what I noted above about the types of attendees we aimed for, Iâll note that the list of respondent organizations doesnât perfectly match who we invited; quite a few other organizations had invitees who didnât fill out the survey. However, I will note that Giving What We Can (which is a project of CEA and was represented by CEA staff) did have representatives there.
As for organizations like DeepMind or GFI: While some of the orgs on the list are focused on a single narrow area, the employees we invited often had backgrounds in EA movement building and (in some cases) direct experience in other cause areas. (One invitee has run at least three major EA-aligned projects in three different areas.)
This wasnât necessarily the case for every attendee (as I mentioned, we considered factors other than community-building experience), but itâs an important reason that the org list looks the way it does.
I really wish this had been included in the OP, in the section that discusses the weaknesses of the data. That section seems to frame the data as a more or less random subset of leaders of EA organizations (âThese results shouldnât be taken as an authoritative or consensus view of effective altruism as a whole. They donât represent everyone in EA, or even every leader of an EA organization.â)
When I look at the list of organizations that were surveyed, it doesnât look like the list of organizations most involved in movement building and coordination. It looks much more like a specific subset of that type of org: those focused on longtermism or x-risk (especially AI) and based in one of the main hubs (London accounts for ~50% of respondents, and the Bay accounts for ~30%).* Those that prioritize global poverty, and to a lesser extent animal welfare, seem notably missing. Itâs possible the list of organizations that didnât respond or werenât named looks a lot different, but if thatâs the case it seems worth calling attention to and possibly trying to rectify (e.g. did you email the survey to anyone or was it all done in person at the Leaders Forum?)
Some of the organizations Iâd have expected to see included, even if the focus was movement building/âcoordination: GiveWell (strategy/âgrowth staff, not pure research staff), LEAN, Charity Entrepreneurship, Vegan Outreach, Rethink Priorities, One for the World, Founders Pledge, etc. Most EAs would see these as EA organizations involved to some degree with movement building. But weâre not learning what they think, while we are apparently hearing from at least one org/âperson who âwant to avoid being connected explicitly to the EA movementâfor example, if almost all their work happens in non-EA circles, where EA might have a mixed reputation.â
Iâm worried that people who read this report are likely to misinterpret the data being presented as more broadly representative than it actually is (e.g. the implications of respondents believing ~30% of EA resources should go to AI work over the next 5 years are radically different if those respondents disproportionally omit people who favor other causes). I have the same concerns about this survey was presented as Jacy Reese expressed about how the leaders survey from 2 years ago (which also captured a narrow set of opinions) was presented:
Lastly, Iâll note that thereâs a certain irony in surveying only a narrow set of people, given that even among those respondents: âThe most common theme in these answers [about problems in the EA community] seems to be the desire for EA to be more inclusive and welcoming. Respondents saw a lot of room for improvement on intellectual diversity, humility, and outreach, whether to distinct groups with different views or to the general population.â I suspect if a more diverse set of leaders had been surveyed, this theme would have been expressed even more strongly.
* GFI and Effective Giving both have London offices, but Iâve assumed their respondents were from other locations.
I agree with you that the orgs you mentioned (e.g. One for the World) are more focused on movement building than some of the other orgs that were invited.
I talked with Amy Labenz (who organized the event) in the course of writing my original reply. We want to clarify that when we said: âAt present, we see Leaders Forum as an event focused on movement building and coordination. We focus on inviting people who play a role in trying to shape the overall direction of the EA movement (whatever cause area they focus on)â. We didnât mean to over-emphasize âmovement buildingâ (in the sense of âbringing more people to EAâ) relative to âpeople shaping the overall direction of the EA movement (in the sense of âfiguring out what the movement should prioritize, growth or otherwiseâ).
My use of the term âmovement buildingâ was my slight misinterpretation of an internal document written by Amy. The eventâs purpose was closer to discussing the goals, health, and trajectory of the movement (e.g. âhow should we prioritize growth vs. other things?â) than discussing how to grow/âbuild the movement (e.g. âhow should we introduce EA to new people?â)
Thanks Aaron, thatâs a helpful clarification. Focusing on âpeople shaping the overall direction of the EA movementâ rather than just movement building seems like a sensible decision. But one drawback is that coming up with a list of those people is a much more subjective (and network-reliant) exercise than, for example, making a list of movement building organizations and inviting representatives from each of them.
Thanks for your feedback on including the eventâs focus as a limitation of the survey. Thatâs something weâll consider if we run a similar survey and decide to publish the data next year.
Some of the organizations you listed had representatives invited who either did not attend or did not fill out the survey. (The survey was emailed to all invitees, and some of those who filled it out didnât attend the event.) If everyone invited had filled it out, I think the list of represented organizations would look more diverse by your criteria.
Thanks Aaron. Glad to hear the invitee list included a broader list of organizations, and that youâll consider a more explicit discussion of potential selection bias effects going forward.
(I was the interim director of CEA during Leaders Forum, and Iâm now the executive director.)
I think that CEA has a history of pushing longtermism in somewhat underhand ways (e.g. I think that I made a mistake when I published an âEA handbookâ without sufficiently consulting non-longtermist researchers, and in a way that probably over-represented AI safety and under-represented material outside of traditional EA cause areas, resulting in a product that appeared to represent EA, without accurately doing so). Given this background, I think itâs reasonable to be suspicious of CEAâs cause prioritisation.
(Iâll be writing more about this in the future, and it feels a bit odd to get into this in a comment when itâs a major-ish update to CEAâs strategy, but I think itâs better to share more rather than less.) In the future, Iâd like CEA to take a more agnostic approach to cause prioritisation, trying to construct non-gameable mechanisms for making decisions about how much we talk about different causes. An example of how this might work is that we might pay for an independent contractor to try to figure out who has spent more than two years full time thinking about cause prioritization, and then surveying those people. Obviously that project would be complicatedâitâs hard to figure out exactly what âcause prioâ means, it would be important to reach out through diverse networks to make sure there arenât network biases etc.
Anyway, given this background of pushing longtermism, I think itâs reasonable to be skeptical of CEAâs approach on this sort of thing.
I think youâre probably right that there are some biases here. How the invite process worked this year was that Amy Labenz, who runs the event, draws up a longlist of potential attendees (asking some external advisors for suggestions about who should be invited). Then Amy, Julia Wise, and I voted yes/âno/âmaybe on all of the individuals on the longlist (often adding comments). Amy made a final call about who to invite, based on those votes. I expect that all of this means that the final invite list is somewhat biased by our networks, and some background assumptions we have about individuals and orgs.
Given this, I think that it would be fair to view the attendees of the event as âsome people who CEA staff think it would be useful to get together for a few daysâ rather than âthe definitive list of EA leadersâ. I think that we were also somewhat loose about what the criteria for inviting people should be, and Iâd like us to be a bit clearer on that in the future (see a couple of paragraphs below). Given this, I think that calling the event âEA Leaders Forumâ is probably a mistake, but others on the team think that changing the name could be confusing and have transition costsâweâre still talking about this, and havenât reached resolution about whether weâll keep the name for next year.
I also think CEA made some mistakes in the way we framed this post (not just the author, since it went through other readers before publication.) I think the post kind of frames this as âEA leaders think Xâ, which I expect would be the sort of thing that lots of EAs should update on. (Even though I think it does try to explicitly disavow this interpretation (see the section on âWhat this data does and does not representâ, I think the title suggests something thatâs more like âEA leaders think these are the prioritiesâprobably you should update towards these being the prioritiesâ). I think that the reality is more like âsome people that CEA staff think itâs useful to get together for an event think Xâ, which is something that people should update on less.
Weâre currently at a team retreat where weâre talking more about what the goals of the event should be in the future. I think that itâs possible that the event looks pretty different in future years, and weâre not yet sure how. But I think that whatever we decide, we should think more carefully about the criteria for attendees, and that will include thinking carefully about the approach to cause prioritization.
Thank you for taking the time to respond, Max. I appreciate your engagement, your explanation of how the invitation process worked this year, and your willingness to acknowledge that CEA may have historically been too aggressive in how it has pushed longtermism and how it has framed the results of past surveys.
Very glad to hear this. As you note, implementing this sort of thing in practice can be tricky. As CEA starts designing new mechanisms, Iâd love to see you gather input (as early possible) from people who have expressed concern about CEAâs representativeness in the past (Iâd be happy to offer opinions if youâd like). These also might be good people to serve as âexternal advisorsâ who generate suggestions for the invite list.
Good luck with the retreat! I look forward to seeing your strategy update once thatâs written up.
Depends a bit on how much you mean to stretch the word âsomeâ⌠This is false as far as I can tell.. at best I would describe your comment as highly misleading.
Iâm not sure what you mean, Peter, but Iâll try to be more clear. Of the seven organizations listed in the comment to which I replied, three of them had people invited, according to the list of people who were recorded as having been sent the invite email.
How did you interpret the word âsomeâ? Is there another sense in which you saw the comment as misleading?
Iâm sorry. I was reading uncharitably and wrote too quickly. Your latest response sounds clear and fair to me. Thanks for providing the numbers and Iâm sorry for misjudging the situation.