Thank you so much for the effort that you put into this review. Your work is incredibly thorough, and I think it’s a public service to hold key organizations accountable.
I also appreciate that you make clear that many of these issues occurred before 2019, and that CEA has been on a positive trajectory over the last few years.
We had a chance to fact-check this post before it was published, and I haven’t re-read it since then, so I don’t have any substantive comments to add on what happened.
Instead, I’ll focus on our actions and attitudes towards the suggestions you make. I generally won’t say too much about our future plans (we try to avoid making public commitments about our work, partly for reasons you explain in your post!).
CEA should hire dedicated Metrics, Evaluation, and Learning staff
This was already on a list of potential future hires. We may also shift one of our current staff (who could be a good fit for this area) into this sort of role (this move would be prompted by other factors, not this post in particular).
However, overall I endorse that we focused on filling operational roles on our programs over this hire, because:
I feel like we have enough capacity within each team to reflect on that team’s work and make improvements: for example, the online team do a lot of data analysis and user interviews; and the events team do an in-depth analysis of EA Global survey responses after each event, making recommendations for what to change.
I feel like others (particularly OpenPhilanthropy) have provided broader-scope analysis that suggests that our programs are highly impactful.
I think that we’ve managed to make some exceptionally useful hires to our programs: e.g. events staff have helped us to ~quadruple our capacity at events this year. Given the last two points, I think that the marginal impact of these hires is greater than a Metrics/Evaluation/Learning hire.
CEA should prioritize sharing evaluations publicly
I think that you’re right that doing more of this would help others to learn from our experience, and allow others to more easily provide feedback on our work. These are benefits.
I still feel pretty unsure whether they outweigh the costs of producing public reports, especially because I think much of our work relies on data that it’s hard to communicate about publicly. I discuss a couple of specific examples below. But thanks for this feedback—we’ll bear it in mind for the future.
CEA should publish a post describing the process and benefits of its expansion and professionalization
Thanks for this suggestion and the implicit positive feedback behind it—I think that you’re right that this could be helpful for some people, and I might write more on our experiences here in the future (though I’m also aware that we’re just one case study, and we don’t have all the answers).
In the meantime, if people feel like they would benefit from hearing about CEA’s experience here, I would probably be happy to chat—feel free to message me on the Forum.
CEA should clearly and explicitly communicate its strategy
As you noted, I have a draft post on this which I may share at some point. For reference, our current strategy page is here.
CEA should publish what it has learned about group support work and invest in structured evaluation
Again, sharing more information publicly would be useful, but I’m not convinced that sharing things publicly vs. more privately with people actively working in the space is the right call.
On quasi-experiments: my feeling is that overall these wouldn’t be cruxy for deciding whether this sort of work is worth doing at all (because I think that we have strong enough evidence, e.g. from OP’s survey, that this is likely the case).
So then they’d be focused on finding the best way to achieve our goals. As background, we’re operating in quite a complex and judgement-driven domain (there aren’t nice outcome measures like there would be for studies of e.g. malaria). For this sort of setup, I think that we’re better off using a looser form of iteration, feedback, and case studies, and user interviews. (This is analogous to the pre-product-market-fit stage of a company where you’re not doing tonnes of A/B testing or profit maximization, but are instead trying to get a richer sense of what products would be useful via user interviews etc.) I think that experiments/quasi-experiments are much more useful for situations where there are clear outcome measures and the overall complexity of the environment is somewhat lower.
Additionally, we have contracted someone (contract confirmed, however they are yet to start) who will be centralising and sharing tools that will be useful—potentially including an intro training for new group organisers.
CEA should have a meaningful mistakes page, or no mistakes page
As you mentioned, we tried to flag that the page was not going to be complete.
We have updated our mistakes page to include many of the mistakes that you mention, particularly historic ones that we weren’t previously aware of. We also plan to link to this post from our mistakes page.
I expect that you will think that this new page misses or underplays some important mistakes, and indeed there were some issues that you list that we deliberately decided not to include.
On that page we say “We don’t list all the ways our projects were inefficient or suboptimal.” I think that we need to do this to limit the scope of the page and make it easier to decide what to include. I think that you may categorize some things as “mistakes” where I would say they’re “suboptimal”.
Re the Funds issues: We think that these issues were resolved around the time that this project spun out of CEA, and then were reintroduced when we no longer controlled the project. We therefore view them as out of scope of the mistakes page, and we have tried to make this clearer in the introduction text.
Overall, I believe that it’s better to have this page, which is at least relatively complete, than nothing at all. I’d be interested in others’ views here.
CEA should consider creating a public dashboard of its commitments to others
As mentioned to you, we maintained an internal dashboard of commitments in 2019, to address these issues. My impression (including from your report) is that these issues were mostly (but not totally) resolved by this and other processes.
It’s not currently a priority to maintain a public or private dashboard of such commitments, partly because we generally try to avoid making public commitments about our plans.
CEA should publish its internal evaluations of EA Grants
I think that sharing these internal evaluations would not be particularly informative for other grantmakers or the community: the program clearly had serious issues, and I don’t think that it was sufficiently close to optimal that other grantmakers would learn much from it. In any case, the board and key grantmakers were informed of the status of the program and had the chance to ask more questions. Grantmakers have since been conducting better-run experiments of this sort (e.g. EA Funds and FTX’s recent experiments).
Writing this up for the public would take valuable staff time, so I don’t think that this is worth the cost. The basic version would be easier to produce but seems even lower value, particularly given that the program has disbursed nothing in the last 3 years or so.
The EA community should seriously engage with governance questions
This is in the “EA community” section, but seems to be mostly directed at CEA, so I’ll respond to it too.
CEA currently receives oversight and scrutiny from its board and its funders. They regularly (~weekly) share critical feedback with us, and annually appraise my performance. I’ve also recently been spending more time talking to community members (from different sections of the community) to ask for their input and feedback on our work. We have recently asked for (anonymous) feedback from community members. Additionally, we regularly receive public feedback (positive and negative) via surveys and public Forum posts/comments. I don’t think that this is a setup with few opportunities for giving feedback, or little accountability.
I think it’s pretty unclear whether there should be more, fewer or different people providing that accountability. I think that the current balance is OK all things considered. That being said, as we go through the operations spinoff and since the organization is growing the board is planning to think more about the right governance setup for CEA (and other organizations in the legal entity), and they may make some changes here. I expect these conversations to be mostly between CEA staff and the board, though they will probably consult other community members too.
Implicitly, you’re suggesting that the appropriate place for discussion of CEA’s governance is in public. In contrast, I think CEA’s board holds that responsibility. Overall, I think that discussion is going to be more productive with a few highly-skilled and high-context individuals rather than a broad discussion. While I agree that getting broad community input into our work is important, I also think that it’s critical that we are held accountable for doing impactful work, which will not always be the same as what pleases community members.
Re: CEA should consider creating a public dashboard of its commitments to others
It’s not currently a priority to maintain a public or private dashboard of such commitments, partly because we generally try to avoid making public commitments about our plans.
As I demonstrated several places in my analysis, the main problem with CEA missing public commitments is that it makes it difficult for other community members to make good plans. CEA avoiding making public commitments doesn’t really solve this problem, and could make it worse. Similarly, it doesn’t help much if CEA says “we hope to do X by Y date, but don’t consider this a commitment” because people are still likely to use that info in their plans for lack of a better option.
A far better outcome would be for CEA to make more accurate public commitments (by adding conservatism and/or providing wide ranges around dates/deliverables to incorporate uncertainty) and then providing timely updates when not on track to meet those commitments. CEA is too important an organization for other EAs not to be able to plan around.
I personally don’t think we can expect orgs to “make accurate predictions”, it’s just too hard.
I’d instead aim to have the org share their best guess often, including “here is our uncertainty” (not as a number, but as something they know better, like “we don’t know if our previous product, X, will be adopted quickly or need a lot of changes”).
Or some other method that as a manager you’d want to use with an employee.
+1 to “not giving any estimates doesn’t solve the problem”, just like it wouldn’t if you’d be a manager and your employee would stop giving estimates
Maybe I was a bit casual saying that “we try not to announce plans publicly”.
We’ve definitely updated in this direction since 2019, but I think that our current communications probably allow people to coordinate relatively well with us.
Let’s look program-by-program:
We plan and announce events well ahead of time, at the point where we confirm venues (arguably we could give even more notice, this is something that we’re working on).
The online team plans major goals on a monthly cycle and then does weekly sprints towards those goals, so there would be at most a 1 month delay between a particular plan being made and it being public.
The groups team is mostly doing repeatable work (basic groups funding, monthly VP rounds, etc). We iteratively make small improvements to those programs, so again there shouldn’t be big gaps between changes being planned and being public.
In terms of less “routine” groups work:
For UGAP, as with events, we announce rounds ahead of time.
The CBG program has mostly been operating via hiring rounds recently, which again are announced/publicised to the appropriate people once we have firm plans. We work with local stakeholders on these rounds.
The Community health team does some “routine” work, which we maintain and iteratively improve (for instance our work on interpersonal harm). For non-routine work that we can discuss publicly, I think that we’ve tended to also announce it publicly.
If we were to stop or majorly deviate from routine work, we’d let people know about that.
So I think that when you look at the program-by-program specifics, I think that people would at least know about our plans shortly after we’ve made them. I think that the key thing that we’ve stopped doing is to commit to timescales for specific improvements to our programs, but I don’t think that this is likely to be causing significant negative externalities (let me know if it is).
I also want to say that if people are worried about stepping on our toes:
Competition can be good! Feel free to just go ahead, and if we end up trying a similar project, then may the best one (or both) succeed.
Please feel free to reach out to us (see my profile for various ways you can do this) to ask whether we have any half-implemented work here, and to see if we can share any advice. (Again, ignoring staff time, ideally people wouldn’t have to ask, but I hope that this setup is reasonably accessible and more time efficient.)
I think that our current communications probably allow people to coordinate relatively well with us.
Yeah, I think you’re generally doing an improved job in this area and that people can currently coordinate fairly well, particularly around the “routine” work you describe. I guess I see part of the benefit of a public dashboard as making sure that routine commitments continue to be met (e.g. timely announcement of event dates and timely delivery of grant money). I’d also expect it to be helpful for monitoring how things are going with new projects that come up (the EA Librarian is a relatively recent example of a new project where commitments weren’t met, albeit one with pretty minor knock-on implications for the community).
I also want to say that if people are worried about stepping on our toes:
Competition can be good! Feel free to just go ahead, and if we end up trying a similar project, then may the best one (or both) succeed.
Please feel free to reach out to us (see my profile for various ways you can do this) to ask whether we have any half-implemented work here, and to see if we can share any advice. (Again, ignoring staff time, ideally people wouldn’t have to ask, but I hope that this setup is reasonably accessible and more time efficient.)
I think it’s great you’re open to people reaching out (though I’m somewhat concerned people will be reluctant to for fear of wasting your time). I also think it was a very positive step for CEA to publish a list of areas where you’re not focusing.
However, I get the sense (especially from your first bullet point) that you’re significantly underestimating how much people will want to avoid competing with CEA. It’s a huge hurdle to compete against a better funded, better connected, and better known organization. I’d guess that if someone inquired about CEA’s plans in an area and were told “we’re not currently working on that but might want to do something in a couple of years” that would still constitute a major deterrent.
I also think there’s an important historical context here, which Peter Wildeford described in late 2019:
I think CEA has frequently tried to “acquire” core activities from other organizations, sometimes using fairly overt pressure. In many cases this has turned out well, but in many cases this has pushed out another group that may have done a good job only for the newly acquired activity to end up “under delivered” by CEA.
While CEA has improved in a lot of areas since 2019, I’m not sure how much progress has been made in this area (which, quite understandably, people are generally reluctant to discuss publicly). I can think of at least one post-2019 instance where, while not exactly matching the pattern Peter describes, I think CEA did much more gate-keeping of an area than was warranted.
Coming back to this, I’m not sure that I have tonnes to add here: I think you’re right that saying that would probably deter people. I think generally in such cases we’d drop the second clause (just say “we’re not currently working on that”, without the “but we might in the future”), to decrease this effect.
I am also aware of some post-2019 instances where we put off people from working in an area. I think that this was mostly inadvertent, but still a significant mistake. If you’re open to DMing me about the instance you’re thinking of, I’d be interested in that. One of our core values is alliance mentality—we want to work with others to improve the world rather than trying to grab territory. So I think we’re trying to do this well. If we’re ever deterring people from doing work, I’m keen to hear this (including anonymously), and I’ll try to make sure that we get out of the way as much as possible.
I strongly encourage people to compete with CEA and ask us about our plans.
As we go through the operations spinoff and since the organization is growing the board is planning to think more about the right governance setup for CEA (and other organizations in the legal entity), and they may make some changes here. I expect these conversations to be mostly between CEA staff and the board, though they will probably consult other community members too.
Glad to hear these conversations are taking place. Even if most of the conversations take place between CEA and the board, I think there’d be value in publicly soliciting thoughts on the matter (even if that didn’t involve a deep public discussion); people outside your direct networks may have some good ideas. FWIW, I’m deeply skeptical that a board anywhere near the size and composition of CEA’s current board could provide sufficient oversight for the numerous organizations that will be part of the larger legal entity. To the extent some or all of those organizations minimize the use of public program evaluations, that raises my skepticism considerably, as that model requires much more board time and attention.
Implicitly, you’re suggesting that the appropriate place for discussion of CEA’s governance is in public. In contrast, I think CEA’s board holds that responsibility. Overall, I think that discussion is going to be more productive with a few highly-skilled and high-context individuals rather than a broad discussion. While I agree that getting broad community input into our work is important, I also think that it’s critical that we are held accountable for doing impactful work, which will not always be the same as what pleases community members.
To clarify my position, I think there should be public meta-level discussion about CEA’s governance, at least as it relates to work CEA is doing on behalf of the community. My sense is there’s very little clarity about a) what are the areas where CEA is managing community resources? (e.g. as I raised in our private correspondence, the @effect_altruism twitter account seems to have been framed as a community account but operated more like CEA’s) b) what are CEA’s responsibilities in those areas? c) what accountability mechanisms should be in place for those areas?
Once there is some clarity on those issues (which I think requires some public discussion, though CEA laying out a proposal would be a reasonable way to kick that off), object-level governance discussions needn’t involve much public discussion. One plausible model (which I’m not endorsing as the “right” governance model) would be to have the community elect a sort of ombudsperson who would serve on CEA’s board with a role of monitoring (and reporting on?) CEA’s responsibilities to the community. In that model, CEA would still be mainly accountable to the board (and could have more efficient private discussions with the board), but there would be a better mechanism for ensuring accountability to the community.
I also want to point out that in areas where CEA’s board has different beliefs than the broader community, the board is a poor accountability mechanism for ensuring that CEA manages community resources in a way that reflects community values. To state an obvious example, CEA’s board favors longtermism more than the community at large. CEA has inappropriately favored longtermism in community resources for many years (and this problem is ongoing). I struggle to see why relying on accountability to the board would be expected to resolve that.
Thanks! I think that a lot of this is an area for the board more than for me (I’ll flag this thread to them for input, but obviously they might not reply). I and the board are tracking how we can best scale governance (and aware that it might be hard to do this just with the current board), and we’ve also considered the ombudsman model (and not yet rejected it, though I think that many versions of it might not really change things too much—I think the board do care about CEA following through on its responsibilites to the community).
Re the EA twitter account: CEA does operate that account, and I think that we inappropriately used it for sharing CEA job ads. We’re changing this now. Thanks for pointing it out. I think that we run some other EA social media accounts, but I’m not aware of any other projects that we do where it’s not clear that CEA runs them.
I think the board do care about CEA following through on its responsibilites to the community
I’m glad this is something the board cares about. That said, I think the board will have difficulty keeping CEA accountable for those responsibilities without 1) a specific board member being explicitly assigned this and 2) an explicit list of what those responsibilities so that CEA, its board, and the community all have the same understanding (and so non-obvious things, like the Twitter account, don’t get missed).
Related to CEA’s board: does CEA have any policies around term-limits for board members? This is a fairly common practice for nonprofits and I’m curious about how CEA thinks about the pros and cons.
My sense is that the board is likely to remain fairly stable, and fairly consistently interested in this.
I also don’t really see why democracy is better on the front of “checking that an org consistently follows through on what it says it’s going to do”: all of your arguments about board members would also seem like they could apply to any electorate. There might be other benefts of a democracy, of course (though I personally think that community democracy would be the wrong governance structure for CEA, for reasons stated elsewhere).
My sense is that the board is likely to remain fairly stable, and fairly consistently interested in this.
Would you trust a governing body on the basis of someone you don’t even personally know saying that their sense is that it’s alright?
all of your arguments about board members would also seem like they could apply to any electorate.
Only for a limited time period—elected officials have to stand for re-election, and separation and balance of powers help keep them in check in the meantime. Changes in the community are also reflected by new elections.
I personally think that community democracy would be the wrong governance structure for CEA, for reasons stated elsewhere
Could you please point to that ‘elsewhere’? I don’t think I’ve encountered your views on the matter.
Would you trust a governing body on the basis of someone you don’t even personally know saying that their sense is that it’s alright?
Probably not—I understand if this doesn’t update you much. I would suggest that you look at public records on what our board members do/have done, and see if you think that suggests that they would hold us accountable for this sort of thing. I admit that’s a costly thing to do. I would also suggest that you look at what CEA has done, especially during the most recent (most relevant) periods—this post highlights most of our key mistakes, and this sequence might give you a sense of positive things we achieved. You could also look at comments/posts I’ve written in order to get a sense of whether you can trust me.
I hope that helps a bit!
Only for a limited time period—elected officials have to stand for re-election, and separation and balance of powers help keep them in check in the meantime. Changes in the community are also reflected by new elections.
My point is that the electorate (not the elected representatives) can leave/new people can join the community. Also their opinions can change. So I don’t think it’s a very robust mechanism for the specific thing of making sure an organization follows through on things it said it would do. I think you’re right that your third point does apply though.
Could you please point to that ‘elsewhere’? I don’t think I’ve encountered your views on the matter.
I don’t literally argue for that position, but I think that the last section of this comment touches on my views.
Ok, I now get what you mean about the electorate. But I think (it’s been some time) my point was about responsibilities to the community rather than on following through.
Regarding the last point, I’m a bit confused because in parallel to this thread we’re discussing another one where I quoted this specific bit exactly, and you replied that it’s not about who should govern CEA, but one meta-level up from that (who decides on the governance structure).
Ah cool, yeah agree that democracy is pretty strongly designed around responsibilities to the community, so it’s probably better than an unelected board on that dimension.
The final paragraph in the comment I just linked to is about one-meta-level-up. The penultimate and antipenultimate paragraphs are just about the ideal governance structure. Sorry, that’s maybe a bit unclear.
Re: CEA should prioritize sharing evaluations publicly
I think that you’re right that doing more of this would help others to learn from our experience, and allow others to more easily provide feedback on our work. These are benefits.
I still feel pretty unsure whether they outweigh the costs of producing public reports, especially because I think much of our work relies on data that it’s hard to communicate about publicly. I discuss a couple of specific examples below. But thanks for this feedback—we’ll bear it in mind for the future.
To clarify, I think CEA itself would also learn a lot from this practice. I’ve raised a number of points that CEA was unaware of, including areas where CEA had attempted to examine the program and including occasions under current management. If one person using public data can produce helpful information, I’d expect the EA hive mind with access to data that’s currently private to produce many more valuable lessons.
I’d also like to emphasize that one big reason I think the benefits of public evaluations are worth the cost is for the signal they send to both outside parties and other EA organizations. As I wrote:
If CEA deprioritizes public evaluations, this behavior could become embedded in EA culture. That would remove valuable feedback loops from the community and raise concerns of hypocrisy since EAs encourage evaluations of other nonprofits.
I’m curious if you have a ballpark estimate of what percentage of EA organizations should publish evaluations. Some of the objections to public evaluations you raise are relevant to most EA orgs, some are specific to CEA, and I’d like to get a better sense of how you think this should play out community-wide.
Thanks—I think you’re right that the EA hive mind would also find some interesting things!
Re the % that should produce public evaluations: I feel pretty unsure. I think it’s important that organizations that are 1) trying to demonstrate with a lot of rigor that they’re extremely cost-effective, and 2) asking for lots of public donations should probably do public evaluations. Maybe my best guess is that most other orgs shouldn’t do this, but should have other governance and feedback mechanisms? And then maybe the first type of organizations are like 20% of total EA orgs, and ~50% of current donations (numbers totally made up).
FWIW, I think about this quite differently. My mental model is more along the lines of “EAs should hold EA charities to the same or higher standards of public evaluation (in terms of frequency and quality) as comparable (in terms of size and type of work) charities outside of EA.” I think the effective altruism homepage does a pretty good job of encapsulating those standards (“We should evaluate the work that charities do, and value transparency and good evidence”). The fact that this statement links to GiveWell (along with lots of other EA discourse) implies that we generally think that evaluation should be public.
Re: CEA should publish what it has learned about group support work and invest in structured evaluation
On quasi-experiments: my feeling is that overall these wouldn’t be cruxy for deciding whether this sort of work is worth doing at all (because I think that we have strong enough evidence, e.g. from OP’s survey, that this is likely the case).
I think it’s fair to say OP’s survey indicates that groups are valuable (at least for longtermism, which is where the survey focused). I think it provides very little information as to why some groups are more valuable than others (groups at top universities seem particularly valuable, but we don’t know if that’s because of their prestige, the age of the groups, paid organizers, or other factors) or which programs from CEA (or others) have the biggest (or smallest) impact on group success. So even if we assume that groups are valuable, and that CEA does group support work, I don’t think those assumptions imply that CEA’s group support work is valuable. My best guess is that CEA’s group support is valuable, but that we don’t know much about which work (e.g. paid organizers vs. online resources) has the most impact on the outcomes we care about. I find it quite plausible that some of the work could actually be counterproductive (e.g. this discussion).
Greater (and more rigorous) experimentation would help sort these details out, especially if it were built into new programs at the outset.
For this sort of setup, I think that we’re better off using a looser form of iteration, feedback, and case studies, and user interviews. (This is analogous to the pre-product-market-fit stage of a company where you’re not doing tonnes of A/B testing or profit maximization, but are instead trying to get a richer sense of what products would be useful via user interviews etc.) I think that experiments/quasi-experiments are much more useful for situations where there are clear outcome measures and the overall complexity of the environment is somewhat lower.
I feel like this has been going on for many years, without a lot of concrete lessons to show for it. Years ago, and also more recently, CEA has discussed feedback loops being too long to learn much, and capacity being too tight to experiment as much as desired.
I agree that we care about multiple outcomes and that this adds some complexity. But we can still do our best to measure those different outcomes and go from there. Six years (more if you count early GWWC groups or EA Build) into CEA’s group support work, we should be well beyond the point of trying to establish product-market-fit.
EA movement building needs more measurement. I’m not privy to all the details of how EA movement building works but it comes across to me as more of a “spray and pray” strategy than I’d like. While we have done some work I think we’ve still really underinvested in market research to test how our movement appeals to the public before running the movement out into the wild big-time. I also think we should do more to track how our current outreach efforts are working, measuring conversion rates, etc. It’s weird that EA has a reputation of being so evidence-based but doesn’t really take much of an evidence-based orientation to its own growth as far as I can tell.
Also worth noting: Peter is a manager of the EAIF, the main funding option for national/city based groups. Max has mentioned that one of the reasons why he thinks public and/or (quasi) experimental evaluation of group work is relatively low priority is because CEA is already sharing information with other funders and key stakeholders (including, I assume, EAIF). Peter’s comment suggests that he doesn’t view whatever information he’s received as constituting a firm base of evidence to guide future decision making.
Max’s comments from our private correspondence (which he’s given me permission to share):
I think that we’ve shared this [i.e. learnings re: group support] with people who are actively trying to do similar things, and we’re happy to continue to do this. I’m not sure I see doing a full public writeup being competitive with other things we could focus on… it’s not that we have a great writeup that we could share but are hoarding: it would take a lot of staff time to communicate it all publicly, and we also couldn’t say some of the most important things. It’s easier to have conversations with people who are interested (where you can focus on the most relevant bits, say things that are hard to say publicly).”
Hey, thanks for this. I work on CEA’s groups team. When you say “we don’t know much about which work … has the most impact on the outcomes we care about”—I think I would rather say
a) We have a reasonable, yet incomplete, view on how many people different groups cause to engage in EA, and some measure on what is the depth of that engagement
b) We are unsure how many of those people would have become engaged in EA anyway
c) We do not have a good mapping from “people engaging with EA” to the things that we actually want in the world
I think we should be sharing more of the data we have on what types of community building have, so far, seemed to generate more engagement. To this end we have a contractor who will be providing a centralized service for some community building tasks, to help spread what is working. I also think groups that seem to be performing well should be running experiments where other groups adopt their model. I have proposed this to several groups, and will continue to do so.
However trying to predict the mapping from engagement to good things happening in the world is (a) sufficiently difficult that I don’t think anyone can do it reliably (b) deeply unpleasant to a lot of communities. In trying to measure this we could decrease the amount of good that is happening in the world—and also probably wouldn’t succeed in taking the measurement accurately.
I think we should be sharing more of the data we have on what types of community building have, so far, seemed to generate more engagement. To this end we have a contractor who will be providing a centralized service for some community building tasks, to help spread what is working.
I’d love to see more sharing of data and what types of community building seem most effective. But I guess I’m confused as to how you’re assessing the latter. To what extent does this assessment incorporate control groups, even if imperfect (e.g. by comparing the number of engaged EAs a group generates before and after getting a paid organizer, or by comparing the trajectory of EAs generated by groups with paid organizers to that of groups without them?)
trying to predict the mapping from engagement to good things happening in the world is (a) sufficiently difficult that I don’t think anyone can do it reliably (b) deeply unpleasant to a lot of communities.
Yes, totally agree that trying to map from engagement to final outcomes is overkill. Thanks for clarifying this point. FWIW, the difficulty issue is the key factor for me, I was surprised by your “unpleasant to a lot of communities” comment. By that, are you referring to the dynamic where if you have to place value on outcomes, some people/orgs will be disappointed with the value you place on their work?
I also think groups that seem to be performing well should be running experiments where other groups adopt their model. I have proposed this to several groups, and will continue to do so.
This seems like another area where control groups would be helpful in making the exercise an actual experiment. Seems like a fairly easy place to introduce at least some randomization into, i.e. designate a pool of groups that could potentially benefit from adopting another group’s practices, and randomly select which of those groups actually do so. Presumably there would be some selection biases since some groups in the “adopt another group’s model” condition may decline to do so, but still potentially a step forward in measuring causality.
I was surprised by your “unpleasant to a lot of communities” comment. By that, are you referring to the dynamic where if you have to place value on outcomes, some people/orgs will be disappointed with the value you place on their work?
Not really. I was more referring that any attempt to quantify the likely impact someone will have is (a) inaccurate (b) likely to create some sort of hierarchy and unhealthy community dynamics.
This seems like another area where control groups would be helpful in making the exercise an actual experiment. Seems like a fairly easy place to introduce at least some randomization into
I agree with this, I like the idea of successful groups joining existing mentorship programs such that there is a natural control group of “average of all the other mentors.” (There are many ways this experiment would be imperfect, as I’m sure you can imagine) - I think the main implementation challenge here so far has been “getting groups to actually want to do this.” We are very careful to preserve the groups’ autonomy, I think this acts as a check on our behaviour. If groups engage on programs with us voluntarily, and we don’t make that engagement a condition of funding, it demonstrates that our programs are at least delivering value in the eyes of the organizers. If we started trying to claim more autonomy and started designating groups into experiments, we’d lose one of our few feedback measures. On balance I think I would prefer to have the feedback mechanism rather than the experiment. (The previous paragraph does contain some simplifications, it would certainly be possible to find examples of where we haven’t optimised purely for group autonomy)
Thanks for clarifying these points Rob. Agree that group autonomy is an important feedback loop, and that this feedback is more important than the experiment I suggested. But to the extent its possible to do experimentation on a voluntary basis, I do think that’d be valuable.
Thank you so much for the effort that you put into this review. Your work is incredibly thorough, and I think it’s a public service to hold key organizations accountable.
I also appreciate that you make clear that many of these issues occurred before 2019, and that CEA has been on a positive trajectory over the last few years.
We had a chance to fact-check this post before it was published, and I haven’t re-read it since then, so I don’t have any substantive comments to add on what happened.
Instead, I’ll focus on our actions and attitudes towards the suggestions you make. I generally won’t say too much about our future plans (we try to avoid making public commitments about our work, partly for reasons you explain in your post!).
CEA should hire dedicated Metrics, Evaluation, and Learning staff
This was already on a list of potential future hires. We may also shift one of our current staff (who could be a good fit for this area) into this sort of role (this move would be prompted by other factors, not this post in particular).
However, overall I endorse that we focused on filling operational roles on our programs over this hire, because:
I feel like we have enough capacity within each team to reflect on that team’s work and make improvements: for example, the online team do a lot of data analysis and user interviews; and the events team do an in-depth analysis of EA Global survey responses after each event, making recommendations for what to change.
I feel like others (particularly OpenPhilanthropy) have provided broader-scope analysis that suggests that our programs are highly impactful.
I think that we’ve managed to make some exceptionally useful hires to our programs: e.g. events staff have helped us to ~quadruple our capacity at events this year. Given the last two points, I think that the marginal impact of these hires is greater than a Metrics/Evaluation/Learning hire.
CEA should prioritize sharing evaluations publicly
I think that you’re right that doing more of this would help others to learn from our experience, and allow others to more easily provide feedback on our work. These are benefits.
I still feel pretty unsure whether they outweigh the costs of producing public reports, especially because I think much of our work relies on data that it’s hard to communicate about publicly. I discuss a couple of specific examples below. But thanks for this feedback—we’ll bear it in mind for the future.
CEA should publish a post describing the process and benefits of its expansion and professionalization
Thanks for this suggestion and the implicit positive feedback behind it—I think that you’re right that this could be helpful for some people, and I might write more on our experiences here in the future (though I’m also aware that we’re just one case study, and we don’t have all the answers).
In the meantime, if people feel like they would benefit from hearing about CEA’s experience here, I would probably be happy to chat—feel free to message me on the Forum.
CEA should clearly and explicitly communicate its strategy
As you noted, I have a draft post on this which I may share at some point. For reference, our current strategy page is here.
CEA should publish what it has learned about group support work and invest in structured evaluation
As you acknowledge we have shared some of this publicly (and, for instance, Jack later edited his post to clarify that he had missed a recent public update where we share some reflections).
Again, sharing more information publicly would be useful, but I’m not convinced that sharing things publicly vs. more privately with people actively working in the space is the right call.
On quasi-experiments: my feeling is that overall these wouldn’t be cruxy for deciding whether this sort of work is worth doing at all (because I think that we have strong enough evidence, e.g. from OP’s survey, that this is likely the case).
So then they’d be focused on finding the best way to achieve our goals. As background, we’re operating in quite a complex and judgement-driven domain (there aren’t nice outcome measures like there would be for studies of e.g. malaria). For this sort of setup, I think that we’re better off using a looser form of iteration, feedback, and case studies, and user interviews. (This is analogous to the pre-product-market-fit stage of a company where you’re not doing tonnes of A/B testing or profit maximization, but are instead trying to get a richer sense of what products would be useful via user interviews etc.) I think that experiments/quasi-experiments are much more useful for situations where there are clear outcome measures and the overall complexity of the environment is somewhat lower.
Additionally, we have contracted someone (contract confirmed, however they are yet to start) who will be centralising and sharing tools that will be useful—potentially including an intro training for new group organisers.
CEA should have a meaningful mistakes page, or no mistakes page
As you mentioned, we tried to flag that the page was not going to be complete.
We have updated our mistakes page to include many of the mistakes that you mention, particularly historic ones that we weren’t previously aware of. We also plan to link to this post from our mistakes page.
I expect that you will think that this new page misses or underplays some important mistakes, and indeed there were some issues that you list that we deliberately decided not to include.
On that page we say “We don’t list all the ways our projects were inefficient or suboptimal.” I think that we need to do this to limit the scope of the page and make it easier to decide what to include. I think that you may categorize some things as “mistakes” where I would say they’re “suboptimal”.
Re the Funds issues: We think that these issues were resolved around the time that this project spun out of CEA, and then were reintroduced when we no longer controlled the project. We therefore view them as out of scope of the mistakes page, and we have tried to make this clearer in the introduction text.
Overall, I believe that it’s better to have this page, which is at least relatively complete, than nothing at all. I’d be interested in others’ views here.
CEA should consider creating a public dashboard of its commitments to others
As mentioned to you, we maintained an internal dashboard of commitments in 2019, to address these issues. My impression (including from your report) is that these issues were mostly (but not totally) resolved by this and other processes.
It’s not currently a priority to maintain a public or private dashboard of such commitments, partly because we generally try to avoid making public commitments about our plans.
CEA should consider using targeted pilot programs
This is something that we have done recently, for instance with the UGAP program, the biosecurity advisors pilot or the EA Librarian project.
CEA should publish its internal evaluations of EA Grants
I think that sharing these internal evaluations would not be particularly informative for other grantmakers or the community: the program clearly had serious issues, and I don’t think that it was sufficiently close to optimal that other grantmakers would learn much from it. In any case, the board and key grantmakers were informed of the status of the program and had the chance to ask more questions. Grantmakers have since been conducting better-run experiments of this sort (e.g. EA Funds and FTX’s recent experiments).
Writing this up for the public would take valuable staff time, so I don’t think that this is worth the cost. The basic version would be easier to produce but seems even lower value, particularly given that the program has disbursed nothing in the last 3 years or so.
The EA community should seriously engage with governance questions
This is in the “EA community” section, but seems to be mostly directed at CEA, so I’ll respond to it too.
CEA currently receives oversight and scrutiny from its board and its funders. They regularly (~weekly) share critical feedback with us, and annually appraise my performance. I’ve also recently been spending more time talking to community members (from different sections of the community) to ask for their input and feedback on our work. We have recently asked for (anonymous) feedback from community members. Additionally, we regularly receive public feedback (positive and negative) via surveys and public Forum posts/comments. I don’t think that this is a setup with few opportunities for giving feedback, or little accountability.
I think it’s pretty unclear whether there should be more, fewer or different people providing that accountability. I think that the current balance is OK all things considered. That being said, as we go through the operations spinoff and since the organization is growing the board is planning to think more about the right governance setup for CEA (and other organizations in the legal entity), and they may make some changes here. I expect these conversations to be mostly between CEA staff and the board, though they will probably consult other community members too.
Implicitly, you’re suggesting that the appropriate place for discussion of CEA’s governance is in public. In contrast, I think CEA’s board holds that responsibility. Overall, I think that discussion is going to be more productive with a few highly-skilled and high-context individuals rather than a broad discussion. While I agree that getting broad community input into our work is important, I also think that it’s critical that we are held accountable for doing impactful work, which will not always be the same as what pleases community members.
Re: CEA should consider creating a public dashboard of its commitments to others
As I demonstrated several places in my analysis, the main problem with CEA missing public commitments is that it makes it difficult for other community members to make good plans. CEA avoiding making public commitments doesn’t really solve this problem, and could make it worse. Similarly, it doesn’t help much if CEA says “we hope to do X by Y date, but don’t consider this a commitment” because people are still likely to use that info in their plans for lack of a better option.
A far better outcome would be for CEA to make more accurate public commitments (by adding conservatism and/or providing wide ranges around dates/deliverables to incorporate uncertainty) and then providing timely updates when not on track to meet those commitments. CEA is too important an organization for other EAs not to be able to plan around.
I personally don’t think we can expect orgs to “make accurate predictions”, it’s just too hard.
I’d instead aim to have the org share their best guess often, including “here is our uncertainty” (not as a number, but as something they know better, like “we don’t know if our previous product, X, will be adopted quickly or need a lot of changes”).
Or some other method that as a manager you’d want to use with an employee.
+1 to “not giving any estimates doesn’t solve the problem”, just like it wouldn’t if you’d be a manager and your employee would stop giving estimates
Maybe I was a bit casual saying that “we try not to announce plans publicly”.
We’ve definitely updated in this direction since 2019, but I think that our current communications probably allow people to coordinate relatively well with us.
Let’s look program-by-program:
We plan and announce events well ahead of time, at the point where we confirm venues (arguably we could give even more notice, this is something that we’re working on).
The online team plans major goals on a monthly cycle and then does weekly sprints towards those goals, so there would be at most a 1 month delay between a particular plan being made and it being public.
The groups team is mostly doing repeatable work (basic groups funding, monthly VP rounds, etc). We iteratively make small improvements to those programs, so again there shouldn’t be big gaps between changes being planned and being public.
In terms of less “routine” groups work:
For UGAP, as with events, we announce rounds ahead of time.
The CBG program has mostly been operating via hiring rounds recently, which again are announced/publicised to the appropriate people once we have firm plans. We work with local stakeholders on these rounds.
The Community health team does some “routine” work, which we maintain and iteratively improve (for instance our work on interpersonal harm). For non-routine work that we can discuss publicly, I think that we’ve tended to also announce it publicly.
If we were to stop or majorly deviate from routine work, we’d let people know about that.
So I think that when you look at the program-by-program specifics, I think that people would at least know about our plans shortly after we’ve made them. I think that the key thing that we’ve stopped doing is to commit to timescales for specific improvements to our programs, but I don’t think that this is likely to be causing significant negative externalities (let me know if it is).
I also want to say that if people are worried about stepping on our toes:
Competition can be good! Feel free to just go ahead, and if we end up trying a similar project, then may the best one (or both) succeed.
Please feel free to reach out to us (see my profile for various ways you can do this) to ask whether we have any half-implemented work here, and to see if we can share any advice. (Again, ignoring staff time, ideally people wouldn’t have to ask, but I hope that this setup is reasonably accessible and more time efficient.)
Yeah, I think you’re generally doing an improved job in this area and that people can currently coordinate fairly well, particularly around the “routine” work you describe. I guess I see part of the benefit of a public dashboard as making sure that routine commitments continue to be met (e.g. timely announcement of event dates and timely delivery of grant money). I’d also expect it to be helpful for monitoring how things are going with new projects that come up (the EA Librarian is a relatively recent example of a new project where commitments weren’t met, albeit one with pretty minor knock-on implications for the community).
I think it’s great you’re open to people reaching out (though I’m somewhat concerned people will be reluctant to for fear of wasting your time). I also think it was a very positive step for CEA to publish a list of areas where you’re not focusing.
However, I get the sense (especially from your first bullet point) that you’re significantly underestimating how much people will want to avoid competing with CEA. It’s a huge hurdle to compete against a better funded, better connected, and better known organization. I’d guess that if someone inquired about CEA’s plans in an area and were told “we’re not currently working on that but might want to do something in a couple of years” that would still constitute a major deterrent.
I also think there’s an important historical context here, which Peter Wildeford described in late 2019:
While CEA has improved in a lot of areas since 2019, I’m not sure how much progress has been made in this area (which, quite understandably, people are generally reluctant to discuss publicly). I can think of at least one post-2019 instance where, while not exactly matching the pattern Peter describes, I think CEA did much more gate-keeping of an area than was warranted.
Oh I should have said, I’m on holiday for the next week, so I won’t be responding to replies in these threads for that period, hope that’s ok!
No problem, have a great holiday :)
Coming back to this, I’m not sure that I have tonnes to add here: I think you’re right that saying that would probably deter people. I think generally in such cases we’d drop the second clause (just say “we’re not currently working on that”, without the “but we might in the future”), to decrease this effect.
I am also aware of some post-2019 instances where we put off people from working in an area. I think that this was mostly inadvertent, but still a significant mistake. If you’re open to DMing me about the instance you’re thinking of, I’d be interested in that. One of our core values is alliance mentality—we want to work with others to improve the world rather than trying to grab territory. So I think we’re trying to do this well. If we’re ever deterring people from doing work, I’m keen to hear this (including anonymously), and I’ll try to make sure that we get out of the way as much as possible.
I strongly encourage people to compete with CEA and ask us about our plans.
Re: Governance…
Glad to hear these conversations are taking place. Even if most of the conversations take place between CEA and the board, I think there’d be value in publicly soliciting thoughts on the matter (even if that didn’t involve a deep public discussion); people outside your direct networks may have some good ideas. FWIW, I’m deeply skeptical that a board anywhere near the size and composition of CEA’s current board could provide sufficient oversight for the numerous organizations that will be part of the larger legal entity. To the extent some or all of those organizations minimize the use of public program evaluations, that raises my skepticism considerably, as that model requires much more board time and attention.
To clarify my position, I think there should be public meta-level discussion about CEA’s governance, at least as it relates to work CEA is doing on behalf of the community. My sense is there’s very little clarity about a) what are the areas where CEA is managing community resources? (e.g. as I raised in our private correspondence, the @effect_altruism twitter account seems to have been framed as a community account but operated more like CEA’s) b) what are CEA’s responsibilities in those areas? c) what accountability mechanisms should be in place for those areas?
Once there is some clarity on those issues (which I think requires some public discussion, though CEA laying out a proposal would be a reasonable way to kick that off), object-level governance discussions needn’t involve much public discussion. One plausible model (which I’m not endorsing as the “right” governance model) would be to have the community elect a sort of ombudsperson who would serve on CEA’s board with a role of monitoring (and reporting on?) CEA’s responsibilities to the community. In that model, CEA would still be mainly accountable to the board (and could have more efficient private discussions with the board), but there would be a better mechanism for ensuring accountability to the community.
I also want to point out that in areas where CEA’s board has different beliefs than the broader community, the board is a poor accountability mechanism for ensuring that CEA manages community resources in a way that reflects community values. To state an obvious example, CEA’s board favors longtermism more than the community at large. CEA has inappropriately favored longtermism in community resources for many years (and this problem is ongoing). I struggle to see why relying on accountability to the board would be expected to resolve that.
Thanks! I think that a lot of this is an area for the board more than for me (I’ll flag this thread to them for input, but obviously they might not reply). I and the board are tracking how we can best scale governance (and aware that it might be hard to do this just with the current board), and we’ve also considered the ombudsman model (and not yet rejected it, though I think that many versions of it might not really change things too much—I think the board do care about CEA following through on its responsibilites to the community).
Re the EA twitter account: CEA does operate that account, and I think that we inappropriately used it for sharing CEA job ads. We’re changing this now. Thanks for pointing it out. I think that we run some other EA social media accounts, but I’m not aware of any other projects that we do where it’s not clear that CEA runs them.
I’m glad this is something the board cares about. That said, I think the board will have difficulty keeping CEA accountable for those responsibilities without 1) a specific board member being explicitly assigned this and 2) an explicit list of what those responsibilities so that CEA, its board, and the community all have the same understanding (and so non-obvious things, like the Twitter account, don’t get missed).
Related to CEA’s board: does CEA have any policies around term-limits for board members? This is a fairly common practice for nonprofits and I’m curious about how CEA thinks about the pros and cons.
On 1), there is a specific board member assigned to assessing CEA’s performance (which would include this). I agree that 2) is somewhat missing.
I’m not aware of a policy on term limits for the Effective Ventures board, and can’t speak for them.
Re: 1, can you share which board member is responsible for this?
Re: 2, is this something CEA plans to work on in say the next 3 months? If not, would it help if a volunteer did an initial draft?
Sure, it’s currently Claire Zabel, but it was Nick Beckstead until July.
We don’t plan to do this in the next 3 months. If a volunteer did a good initial draft, I think there’s an 80% chance that we use that in some way.
I hope that’s true, but there are at least two problems with that:
It’s impossible for the community to verify
It can very easily change as:
Board members leave and new ones join
Board members’ opinions on this change
Most importantly, the community itself changes in ways not reflected by the board
As far as I can see, only democratic mechanisms guarantee accountability that stays stable over time.
My sense is that the board is likely to remain fairly stable, and fairly consistently interested in this.
I also don’t really see why democracy is better on the front of “checking that an org consistently follows through on what it says it’s going to do”: all of your arguments about board members would also seem like they could apply to any electorate. There might be other benefts of a democracy, of course (though I personally think that community democracy would be the wrong governance structure for CEA, for reasons stated elsewhere).
I’m not sure I follow.
Would you trust a governing body on the basis of someone you don’t even personally know saying that their sense is that it’s alright?
Only for a limited time period—elected officials have to stand for re-election, and separation and balance of powers help keep them in check in the meantime. Changes in the community are also reflected by new elections.
Could you please point to that ‘elsewhere’? I don’t think I’ve encountered your views on the matter.
Probably not—I understand if this doesn’t update you much. I would suggest that you look at public records on what our board members do/have done, and see if you think that suggests that they would hold us accountable for this sort of thing. I admit that’s a costly thing to do. I would also suggest that you look at what CEA has done, especially during the most recent (most relevant) periods—this post highlights most of our key mistakes, and this sequence might give you a sense of positive things we achieved. You could also look at comments/posts I’ve written in order to get a sense of whether you can trust me.
I hope that helps a bit!
My point is that the electorate (not the elected representatives) can leave/new people can join the community. Also their opinions can change. So I don’t think it’s a very robust mechanism for the specific thing of making sure an organization follows through on things it said it would do. I think you’re right that your third point does apply though.
I don’t literally argue for that position, but I think that the last section of this comment touches on my views.
Ok, I now get what you mean about the electorate. But I think (it’s been some time) my point was about responsibilities to the community rather than on following through.
Regarding the last point, I’m a bit confused because in parallel to this thread we’re discussing another one where I quoted this specific bit exactly, and you replied that it’s not about who should govern CEA, but one meta-level up from that (who decides on the governance structure).
Ah cool, yeah agree that democracy is pretty strongly designed around responsibilities to the community, so it’s probably better than an unelected board on that dimension.
The final paragraph in the comment I just linked to is about one-meta-level-up. The penultimate and antipenultimate paragraphs are just about the ideal governance structure. Sorry, that’s maybe a bit unclear.
Re: CEA should prioritize sharing evaluations publicly
To clarify, I think CEA itself would also learn a lot from this practice. I’ve raised a number of points that CEA was unaware of, including areas where CEA had attempted to examine the program and including occasions under current management. If one person using public data can produce helpful information, I’d expect the EA hive mind with access to data that’s currently private to produce many more valuable lessons.
I’d also like to emphasize that one big reason I think the benefits of public evaluations are worth the cost is for the signal they send to both outside parties and other EA organizations. As I wrote:
I’m curious if you have a ballpark estimate of what percentage of EA organizations should publish evaluations. Some of the objections to public evaluations you raise are relevant to most EA orgs, some are specific to CEA, and I’d like to get a better sense of how you think this should play out community-wide.
Thanks—I think you’re right that the EA hive mind would also find some interesting things!
Re the % that should produce public evaluations: I feel pretty unsure. I think it’s important that organizations that are 1) trying to demonstrate with a lot of rigor that they’re extremely cost-effective, and 2) asking for lots of public donations should probably do public evaluations. Maybe my best guess is that most other orgs shouldn’t do this, but should have other governance and feedback mechanisms? And then maybe the first type of organizations are like 20% of total EA orgs, and ~50% of current donations (numbers totally made up).
Thanks for sharing your thinking on this.
FWIW, I think about this quite differently. My mental model is more along the lines of “EAs should hold EA charities to the same or higher standards of public evaluation (in terms of frequency and quality) as comparable (in terms of size and type of work) charities outside of EA.” I think the effective altruism homepage does a pretty good job of encapsulating those standards (“We should evaluate the work that charities do, and value transparency and good evidence”). The fact that this statement links to GiveWell (along with lots of other EA discourse) implies that we generally think that evaluation should be public.
Re: CEA should publish what it has learned about group support work and invest in structured evaluation
I think it’s fair to say OP’s survey indicates that groups are valuable (at least for longtermism, which is where the survey focused). I think it provides very little information as to why some groups are more valuable than others (groups at top universities seem particularly valuable, but we don’t know if that’s because of their prestige, the age of the groups, paid organizers, or other factors) or which programs from CEA (or others) have the biggest (or smallest) impact on group success. So even if we assume that groups are valuable, and that CEA does group support work, I don’t think those assumptions imply that CEA’s group support work is valuable. My best guess is that CEA’s group support is valuable, but that we don’t know much about which work (e.g. paid organizers vs. online resources) has the most impact on the outcomes we care about. I find it quite plausible that some of the work could actually be counterproductive (e.g. this discussion).
Greater (and more rigorous) experimentation would help sort these details out, especially if it were built into new programs at the outset.
I feel like this has been going on for many years, without a lot of concrete lessons to show for it. Years ago, and also more recently, CEA has discussed feedback loops being too long to learn much, and capacity being too tight to experiment as much as desired.
I agree that we care about multiple outcomes and that this adds some complexity. But we can still do our best to measure those different outcomes and go from there. Six years (more if you count early GWWC groups or EA Build) into CEA’s group support work, we should be well beyond the point of trying to establish product-market-fit.
This comment from Peter Wildeford’s recently published criticisms of EA seems relevant to this topic:
Also worth noting: Peter is a manager of the EAIF, the main funding option for national/city based groups. Max has mentioned that one of the reasons why he thinks public and/or (quasi) experimental evaluation of group work is relatively low priority is because CEA is already sharing information with other funders and key stakeholders (including, I assume, EAIF). Peter’s comment suggests that he doesn’t view whatever information he’s received as constituting a firm base of evidence to guide future decision making.
Max’s comments from our private correspondence (which he’s given me permission to share):
Hey, thanks for this. I work on CEA’s groups team. When you say “we don’t know much about which work … has the most impact on the outcomes we care about”—I think I would rather say
a) We have a reasonable, yet incomplete, view on how many people different groups cause to engage in EA, and some measure on what is the depth of that engagement
b) We are unsure how many of those people would have become engaged in EA anyway
c) We do not have a good mapping from “people engaging with EA” to the things that we actually want in the world
I think we should be sharing more of the data we have on what types of community building have, so far, seemed to generate more engagement. To this end we have a contractor who will be providing a centralized service for some community building tasks, to help spread what is working. I also think groups that seem to be performing well should be running experiments where other groups adopt their model. I have proposed this to several groups, and will continue to do so.
However trying to predict the mapping from engagement to good things happening in the world is (a) sufficiently difficult that I don’t think anyone can do it reliably (b) deeply unpleasant to a lot of communities. In trying to measure this we could decrease the amount of good that is happening in the world—and also probably wouldn’t succeed in taking the measurement accurately.
Thanks Rob, this is helpful!
I’d love to see more sharing of data and what types of community building seem most effective. But I guess I’m confused as to how you’re assessing the latter. To what extent does this assessment incorporate control groups, even if imperfect (e.g. by comparing the number of engaged EAs a group generates before and after getting a paid organizer, or by comparing the trajectory of EAs generated by groups with paid organizers to that of groups without them?)
Yes, totally agree that trying to map from engagement to final outcomes is overkill. Thanks for clarifying this point. FWIW, the difficulty issue is the key factor for me, I was surprised by your “unpleasant to a lot of communities” comment. By that, are you referring to the dynamic where if you have to place value on outcomes, some people/orgs will be disappointed with the value you place on their work?
This seems like another area where control groups would be helpful in making the exercise an actual experiment. Seems like a fairly easy place to introduce at least some randomization into, i.e. designate a pool of groups that could potentially benefit from adopting another group’s practices, and randomly select which of those groups actually do so. Presumably there would be some selection biases since some groups in the “adopt another group’s model” condition may decline to do so, but still potentially a step forward in measuring causality.
Not really. I was more referring that any attempt to quantify the likely impact someone will have is (a) inaccurate (b) likely to create some sort of hierarchy and unhealthy community dynamics.
I agree with this, I like the idea of successful groups joining existing mentorship programs such that there is a natural control group of “average of all the other mentors.” (There are many ways this experiment would be imperfect, as I’m sure you can imagine) - I think the main implementation challenge here so far has been “getting groups to actually want to do this.” We are very careful to preserve the groups’ autonomy, I think this acts as a check on our behaviour. If groups engage on programs with us voluntarily, and we don’t make that engagement a condition of funding, it demonstrates that our programs are at least delivering value in the eyes of the organizers. If we started trying to claim more autonomy and started designating groups into experiments, we’d lose one of our few feedback measures. On balance I think I would prefer to have the feedback mechanism rather than the experiment. (The previous paragraph does contain some simplifications, it would certainly be possible to find examples of where we haven’t optimised purely for group autonomy)
Thanks for clarifying these points Rob. Agree that group autonomy is an important feedback loop, and that this feedback is more important than the experiment I suggested. But to the extent its possible to do experimentation on a voluntary basis, I do think that’d be valuable.
I agree with this statement entirely.
Go team!