One of the biggest challenges is in finding strong people to do it. Generally the people qualified to do strong evaluation work are also qualified to do grant funding directly, so just go and do that. It’s hard to do evaluation well, and public writeups present a bunch of extra challenges, many of which aren’t very fun.
If people here have thoughts on how we can scale public evaluation, I’d be very curious.
Some miscellaneous thoughts:
EA jobs are (still) very hard to get. So if evaluation jobs were available, I’d expect them to attract very talented applicants.
If evaluation work was seen as a stepping stone to grantmaking work, that would make evaluation jobs even more desirable.
It isn’t obvious to me where these jobs should live. It could live with the organizations running the programs, with grantmakers, or in a separate organization (a la GiveWell). I have some concerns about organizations evaluating their own programs, as the incentives aren’t great (lots of reasons to say “we evaluated our work and it looks great!”) and it’s very hard to have common standards/methodologies across orgs. Grantmakers may be hesitant to do public evaluations as it could undermine grantee relationships. I’d lean toward a dedicated evaluation organization, though that has its own problems (need a way to fund it, orgs would need to provide it with program data, etc.)
I don’t know a ton about impact certificates, but I wonder if they could be a useful funding mechanism where funders and/or orgs being evaluated would pay for evaluations they find useful after the fact.
It’s definitely easier to come up with an evaluation model when the org/project involved wants to be evaluated (if not, there are a lot of added complications). I wonder if it would be worthwhile for an org that wants to be evaluated to contract with a university or local group (which I think are often looking for tangible ways to get involved) to execute an evaluation, and see if that evaluation proved valuable.
My sense is that evaluating individual orgs/projects would be a lot easier if we had better high level community metrics in place (e.g. better understanding of EA growth rate, value of a new EA, value of a university group, etc.)
Some miscellaneous thoughts:
EA jobs are (still) very hard to get. So if evaluation jobs were available, I’d expect them to attract very talented applicants.
If evaluation work was seen as a stepping stone to grantmaking work, that would make evaluation jobs even more desirable.
It isn’t obvious to me where these jobs should live. It could live with the organizations running the programs, with grantmakers, or in a separate organization (a la GiveWell). I have some concerns about organizations evaluating their own programs, as the incentives aren’t great (lots of reasons to say “we evaluated our work and it looks great!”) and it’s very hard to have common standards/methodologies across orgs. Grantmakers may be hesitant to do public evaluations as it could undermine grantee relationships. I’d lean toward a dedicated evaluation organization, though that has its own problems (need a way to fund it, orgs would need to provide it with program data, etc.)
I don’t know a ton about impact certificates, but I wonder if they could be a useful funding mechanism where funders and/or orgs being evaluated would pay for evaluations they find useful after the fact.
It’s definitely easier to come up with an evaluation model when the org/project involved wants to be evaluated (if not, there are a lot of added complications). I wonder if it would be worthwhile for an org that wants to be evaluated to contract with a university or local group (which I think are often looking for tangible ways to get involved) to execute an evaluation, and see if that evaluation proved valuable.
My sense is that evaluating individual orgs/projects would be a lot easier if we had better high level community metrics in place (e.g. better understanding of EA growth rate, value of a new EA, value of a university group, etc.)