Re: CEA should prioritize sharing evaluations publicly
I think that you’re right that doing more of this would help others to learn from our experience, and allow others to more easily provide feedback on our work. These are benefits.
I still feel pretty unsure whether they outweigh the costs of producing public reports, especially because I think much of our work relies on data that it’s hard to communicate about publicly. I discuss a couple of specific examples below. But thanks for this feedback—we’ll bear it in mind for the future.
To clarify, I think CEA itself would also learn a lot from this practice. I’ve raised a number of points that CEA was unaware of, including areas where CEA had attempted to examine the program and including occasions under current management. If one person using public data can produce helpful information, I’d expect the EA hive mind with access to data that’s currently private to produce many more valuable lessons.
I’d also like to emphasize that one big reason I think the benefits of public evaluations are worth the cost is for the signal they send to both outside parties and other EA organizations. As I wrote:
If CEA deprioritizes public evaluations, this behavior could become embedded in EA culture. That would remove valuable feedback loops from the community and raise concerns of hypocrisy since EAs encourage evaluations of other nonprofits.
I’m curious if you have a ballpark estimate of what percentage of EA organizations should publish evaluations. Some of the objections to public evaluations you raise are relevant to most EA orgs, some are specific to CEA, and I’d like to get a better sense of how you think this should play out community-wide.
Thanks—I think you’re right that the EA hive mind would also find some interesting things!
Re the % that should produce public evaluations: I feel pretty unsure. I think it’s important that organizations that are 1) trying to demonstrate with a lot of rigor that they’re extremely cost-effective, and 2) asking for lots of public donations should probably do public evaluations. Maybe my best guess is that most other orgs shouldn’t do this, but should have other governance and feedback mechanisms? And then maybe the first type of organizations are like 20% of total EA orgs, and ~50% of current donations (numbers totally made up).
FWIW, I think about this quite differently. My mental model is more along the lines of “EAs should hold EA charities to the same or higher standards of public evaluation (in terms of frequency and quality) as comparable (in terms of size and type of work) charities outside of EA.” I think the effective altruism homepage does a pretty good job of encapsulating those standards (“We should evaluate the work that charities do, and value transparency and good evidence”). The fact that this statement links to GiveWell (along with lots of other EA discourse) implies that we generally think that evaluation should be public.
Re: CEA should prioritize sharing evaluations publicly
To clarify, I think CEA itself would also learn a lot from this practice. I’ve raised a number of points that CEA was unaware of, including areas where CEA had attempted to examine the program and including occasions under current management. If one person using public data can produce helpful information, I’d expect the EA hive mind with access to data that’s currently private to produce many more valuable lessons.
I’d also like to emphasize that one big reason I think the benefits of public evaluations are worth the cost is for the signal they send to both outside parties and other EA organizations. As I wrote:
I’m curious if you have a ballpark estimate of what percentage of EA organizations should publish evaluations. Some of the objections to public evaluations you raise are relevant to most EA orgs, some are specific to CEA, and I’d like to get a better sense of how you think this should play out community-wide.
Thanks—I think you’re right that the EA hive mind would also find some interesting things!
Re the % that should produce public evaluations: I feel pretty unsure. I think it’s important that organizations that are 1) trying to demonstrate with a lot of rigor that they’re extremely cost-effective, and 2) asking for lots of public donations should probably do public evaluations. Maybe my best guess is that most other orgs shouldn’t do this, but should have other governance and feedback mechanisms? And then maybe the first type of organizations are like 20% of total EA orgs, and ~50% of current donations (numbers totally made up).
Thanks for sharing your thinking on this.
FWIW, I think about this quite differently. My mental model is more along the lines of “EAs should hold EA charities to the same or higher standards of public evaluation (in terms of frequency and quality) as comparable (in terms of size and type of work) charities outside of EA.” I think the effective altruism homepage does a pretty good job of encapsulating those standards (“We should evaluate the work that charities do, and value transparency and good evidence”). The fact that this statement links to GiveWell (along with lots of other EA discourse) implies that we generally think that evaluation should be public.