I agree that more public evaluations of things like CEA programs and programs that they fund, would be really valuable. I’m sure people at CEA would agree too. In my experience at QURI, funders are pretty positive about this sort of work.
One of the biggest challenges is in finding strong people to do it. Generally the people qualified to do strong evaluation work are also qualified to do grant funding directly, so just go and do that. It’s hard to do evaluation well, and public writeups present a bunch of extra challenges, many of which aren’t very fun.
If people here have thoughts on how we can scale public evaluation, I’d be very curious.
I agree that more public evaluations of things like CEA programs and programs that they fund, would be really valuable. I’m sure people at CEA would agree too.
I don’t think CEA staff, or at least CEA leadership, actually agrees that public evaluations are “really valuable”. CEA has repeatedly deprioritized public evaluations, even after public commitments to conduct them (e.g. CBG, EA Grants, Pareto Fellowship). And Max has been pretty clear that he prioritizes accountability to board members and funders for CEA rather than to the public or EA community, and that he thinks public evaluations generally aren’t worth the cost to produce them (especially because CEA is hesitant to publicly criticize people/orgs and that is often where the most the most useful information can be gleaned.) So my sense is that CEA thinks public evaluations would be valuable in some abstract sense where there aren’t any costs to conducting them, but not in a practical sense that incorporates the tradeoffs that always exist in the real world.
We might be quibbling a bit over what “really valuable” means. I agree that CEA definitely could have prioritized these higher, and likely would have if they cared about it much more.
I think they would be happy to have evaluations done if they were very inexpensive or free, for whatever that’s worth. This is much better than with many orgs, who would try to oppose evaluations even if they are free; but perhaps it is suboptimal.
I think only doing something if it’s free/inexpensive is almost the opposite of thinking something is ‘really valuable’, so that’s far from a quibble (almost the opposite because, as you point out, actively being against something is the actual opposite).
Ozzie, I think you and I agree on CEA’s stance about public evaluations (not actively opposed to them, but mainly interested only if they are free or very inexpensive to execute). My interpretation of that position is largely in line with Rebecca’s though.
One of the biggest challenges is in finding strong people to do it. Generally the people qualified to do strong evaluation work are also qualified to do grant funding directly, so just go and do that. It’s hard to do evaluation well, and public writeups present a bunch of extra challenges, many of which aren’t very fun.
If people here have thoughts on how we can scale public evaluation, I’d be very curious.
Some miscellaneous thoughts:
EA jobs are (still) very hard to get. So if evaluation jobs were available, I’d expect them to attract very talented applicants.
If evaluation work was seen as a stepping stone to grantmaking work, that would make evaluation jobs even more desirable.
It isn’t obvious to me where these jobs should live. It could live with the organizations running the programs, with grantmakers, or in a separate organization (a la GiveWell). I have some concerns about organizations evaluating their own programs, as the incentives aren’t great (lots of reasons to say “we evaluated our work and it looks great!”) and it’s very hard to have common standards/methodologies across orgs. Grantmakers may be hesitant to do public evaluations as it could undermine grantee relationships. I’d lean toward a dedicated evaluation organization, though that has its own problems (need a way to fund it, orgs would need to provide it with program data, etc.)
I don’t know a ton about impact certificates, but I wonder if they could be a useful funding mechanism where funders and/or orgs being evaluated would pay for evaluations they find useful after the fact.
It’s definitely easier to come up with an evaluation model when the org/project involved wants to be evaluated (if not, there are a lot of added complications). I wonder if it would be worthwhile for an org that wants to be evaluated to contract with a university or local group (which I think are often looking for tangible ways to get involved) to execute an evaluation, and see if that evaluation proved valuable.
My sense is that evaluating individual orgs/projects would be a lot easier if we had better high level community metrics in place (e.g. better understanding of EA growth rate, value of a new EA, value of a university group, etc.)
Very quick note:
I agree that more public evaluations of things like CEA programs and programs that they fund, would be really valuable. I’m sure people at CEA would agree too. In my experience at QURI, funders are pretty positive about this sort of work.
One of the biggest challenges is in finding strong people to do it. Generally the people qualified to do strong evaluation work are also qualified to do grant funding directly, so just go and do that. It’s hard to do evaluation well, and public writeups present a bunch of extra challenges, many of which aren’t very fun.
If people here have thoughts on how we can scale public evaluation, I’d be very curious.
I don’t think CEA staff, or at least CEA leadership, actually agrees that public evaluations are “really valuable”. CEA has repeatedly deprioritized public evaluations, even after public commitments to conduct them (e.g. CBG, EA Grants, Pareto Fellowship). And Max has been pretty clear that he prioritizes accountability to board members and funders for CEA rather than to the public or EA community, and that he thinks public evaluations generally aren’t worth the cost to produce them (especially because CEA is hesitant to publicly criticize people/orgs and that is often where the most the most useful information can be gleaned.) So my sense is that CEA thinks public evaluations would be valuable in some abstract sense where there aren’t any costs to conducting them, but not in a practical sense that incorporates the tradeoffs that always exist in the real world.
We might be quibbling a bit over what “really valuable” means. I agree that CEA definitely could have prioritized these higher, and likely would have if they cared about it much more.
I think they would be happy to have evaluations done if they were very inexpensive or free, for whatever that’s worth. This is much better than with many orgs, who would try to oppose evaluations even if they are free; but perhaps it is suboptimal.
I think only doing something if it’s free/inexpensive is almost the opposite of thinking something is ‘really valuable’, so that’s far from a quibble (almost the opposite because, as you point out, actively being against something is the actual opposite).
Ozzie, I think you and I agree on CEA’s stance about public evaluations (not actively opposed to them, but mainly interested only if they are free or very inexpensive to execute). My interpretation of that position is largely in line with Rebecca’s though.
I might be interested in working on this, maybe to run a short POC. Started to write some thoughts now on a public doc :)
Thanks for starting that doc! I added some comments from my response to Ozzie.
Some miscellaneous thoughts:
EA jobs are (still) very hard to get. So if evaluation jobs were available, I’d expect them to attract very talented applicants.
If evaluation work was seen as a stepping stone to grantmaking work, that would make evaluation jobs even more desirable.
It isn’t obvious to me where these jobs should live. It could live with the organizations running the programs, with grantmakers, or in a separate organization (a la GiveWell). I have some concerns about organizations evaluating their own programs, as the incentives aren’t great (lots of reasons to say “we evaluated our work and it looks great!”) and it’s very hard to have common standards/methodologies across orgs. Grantmakers may be hesitant to do public evaluations as it could undermine grantee relationships. I’d lean toward a dedicated evaluation organization, though that has its own problems (need a way to fund it, orgs would need to provide it with program data, etc.)
I don’t know a ton about impact certificates, but I wonder if they could be a useful funding mechanism where funders and/or orgs being evaluated would pay for evaluations they find useful after the fact.
It’s definitely easier to come up with an evaluation model when the org/project involved wants to be evaluated (if not, there are a lot of added complications). I wonder if it would be worthwhile for an org that wants to be evaluated to contract with a university or local group (which I think are often looking for tangible ways to get involved) to execute an evaluation, and see if that evaluation proved valuable.
My sense is that evaluating individual orgs/projects would be a lot easier if we had better high level community metrics in place (e.g. better understanding of EA growth rate, value of a new EA, value of a university group, etc.)