FWIW I think that GiveWell selects organisations are based on close to the best evidence base we have, kind of the opposite of “p-hacking”. It doesn’t make sense to me that anyone can be sure their local charity is doing “any good at all” without knowing the counterfactual.
My classic example to illustrate this is the original microloans in Bangladesh. Everyone could “see” how much they were helping as most of the women loaned money were growing successful businesses. Until they looked at cohorts of women who didn’t get loans and maybe of them were also running successful businesses as well. The loans were helping counterfactual but only in a minor way—most of the women would have done great anyway without the loan.
I think with situations like ACE charities that ended up not being useful it’s less of a p-hacking problem and more of an uncertainty problem. They just don’t have the same rigo to of us evidence base for efficacy as global health interventions so there are likely to be more failures and that’s hard to avoid.
FWIW I think that GiveWell selects organisations are based on close to the best evidence base we have, kind of the opposite of “p-hacking”. It doesn’t make sense to me that anyone can be sure their local charity is doing “any good at all” without knowing the counterfactual.
My classic example to illustrate this is the original microloans in Bangladesh. Everyone could “see” how much they were helping as most of the women loaned money were growing successful businesses. Until they looked at cohorts of women who didn’t get loans and maybe of them were also running successful businesses as well. The loans were helping counterfactual but only in a minor way—most of the women would have done great anyway without the loan.
I think with situations like ACE charities that ended up not being useful it’s less of a p-hacking problem and more of an uncertainty problem. They just don’t have the same rigo to of us evidence base for efficacy as global health interventions so there are likely to be more failures and that’s hard to avoid.