How GiveWell’s Research is Evolving

Link post

To date, most of GiveWell’s re­search ca­pac­ity has fo­cused on find­ing the most im­pact­ful pro­grams among those whose re­sults can be rigor­ously mea­sured. This work has led us to recom­mend, and di­rect sev­eral hun­dred mil­lion dol­lars to, char­i­ties im­prov­ing health, sav­ing lives, and in­creas­ing in­come in low-in­come coun­tries.

One of the most im­por­tant rea­sons we have fo­cused on pro­grams where ro­bust mea­sure­ment is pos­si­ble is be­cause this ap­proach largely does not rely on sub­ject-mat­ter ex­per­tise. When Holden and I started GiveWell, nei­ther of us had any ex­pe­rience in philan­thropy, so we looked for char­i­ties that we could eval­u­ate through data and ev­i­dence that we could an­a­lyze, to make recom­men­da­tions that we could fully ex­plain. This led us to fo­cus on or­ga­ni­za­tions that had im­pacts that were rel­a­tively easy to mea­sure.

The out­put of this pro­cess is re­flected in our cur­rent top char­i­ties and the pro­grams they run, which are an­a­lyzed in our in­ter­ven­tion re­ports.

GiveWell has now been do­ing re­search to find the best giv­ing op­por­tu­ni­ties in global health and de­vel­op­ment for 11 years, and we plan to in­crease the scope of giv­ing op­por­tu­ni­ties we con­sider. We plan to ex­pand our re­search team and scope in or­der to de­ter­mine whether there are giv­ing op­por­tu­ni­ties in global health and de­vel­op­ment that are more cost-effec­tive than those we have iden­ti­fied to date.

We ex­pect this ex­pan­sion of our work to take us in a num­ber of new di­rec­tions, some of which we have be­gun to ex­plore over the past few years. We have con­sid­ered, in a few cases, the im­pact our top and stand­out char­i­ties have through pro­vid­ing tech­ni­cal as­sis­tance (for ex­am­ple, De­worm the World and Pro­ject Healthy Chil­dren), sup­ported work to change gov­ern­ment poli­cies through our In­cu­ba­tion Grants pro­gram (for ex­am­ple, grants to the Cen­tre for Pes­ti­cide Suicide Preven­tion and In­no­va­tion in Govern­ment Ini­ti­a­tive), and be­gun to ex­plore ar­eas like to­bacco policy and lead paint elimi­na­tion.

Over the next sev­eral years, we plan to con­sider ev­ery­thing that we be­lieve could be among the most cost-effec­tive (broadly defined) giv­ing op­por­tu­ni­ties in global health and de­vel­op­ment. This in­cludes more com­pre­hen­sively re­view­ing di­rect in­ter­ven­tions in sec­tors where im­pacts are more difficult to mea­sure, in­ves­ti­gat­ing op­por­tu­ni­ties to in­fluence gov­ern­ment policy, as well as other ar­eas.

Mak­ing progress in ar­eas where it is harder to de­ter­mine causal­ity will be challeng­ing. In my opinion, we are ex­cel­lent eval­u­a­tors of em­piri­cal re­search, but we have yet to demon­strate the abil­ity to make good judg­ments about giv­ing op­por­tu­ni­ties when less em­piri­cal in­for­ma­tion is available. Our val­ues, in­tel­lec­tual frame­work, cul­ture, and the qual­ity of our staff make me op­ti­mistic about our chances, but all of us at GiveWell rec­og­nize the difficulty of the pro­ject we are em­bark­ing on.

Our staff does not cur­rently have the ca­pac­ity or the ca­pa­bil­ities to make enough progress in this di­rec­tion, so we are plan­ning to sig­nifi­cantly in­crease the size of our staff. We have a re­search team of ten peo­ple, and we are plan­ning to more than dou­ble in size over the next three years. We are plan­ning to add some ju­nior staff but are pri­mar­ily aiming to hire peo­ple with rele­vant ex­pe­rience who can con­tribute as re­searchers and/​or man­agers on our team.

GiveWell’s top char­i­ties list is not go­ing to change dra­mat­i­cally in the near fu­ture, and it may always in­clude the char­i­ties we recom­mend to­day. Our top char­i­ties achieve out­stand­ing, cost-effec­tive re­sults, and we be­lieve they are some of the best giv­ing op­por­tu­ni­ties in global health and de­vel­op­ment. We ex­pect to con­clude that many of the op­por­tu­ni­ties we con­sider in ar­eas that are new for us are less cost-effec­tive than those we cur­rently recom­mend, but we also think it is pos­si­ble that we will iden­tify some op­por­tu­ni­ties that are much more cost-effec­tive. We be­lieve it is worth a ma­jor effort to find out.

What ar­eas will we look into?

As with any ex­plo­ra­tion into a new area, we ex­pect the speci­fics of the work we will un­der­take to shift as we learn more. Below we dis­cuss two ma­jor ar­eas of work we are em­bark­ing on and build­ing our team for cur­rently. In the long term, we are open to con­sid­er­ing mak­ing grants or recom­men­da­tions in all ar­eas of global health and de­vel­op­ment. We have not yet com­pre­hen­sively con­sid­ered what those ar­eas might be, but they could in­clude (for ex­am­ple) re­search and de­vel­op­ment, or so­cial en­trepreneur­ship.

Us­ing rea­soned judg­ment and less ro­bust ev­i­dence to come to con­clu­sions about ad­di­tional di­rect-de­liv­ery interventions

In the past, we have of­ten asked, “does this in­ter­ven­tion meet our crite­ria?” rather than “what is our best guess about how promis­ing this in­ter­ven­tion is rel­a­tive to our top char­i­ties?” Our in­ter­ven­tion re­port on ed­u­ca­tion is a good ex­am­ple of ask­ing the ques­tion, “does this meet our crite­ria?” It re­views all ran­dom­ized con­trol­led tri­als of ed­u­ca­tion pro­grams that mea­sure long-term out­comes, but it does not at­tempt to reach a bot­tom line about how cost-effec­tive ed­u­ca­tion in de­vel­op­ing coun­tries is.

We plan to more deeply ex­plore how we can reach con­clu­sions about how ar­eas such as nu­tri­tion, agri­cul­ture, ed­u­ca­tion, re­pro­duc­tive health, sur­gi­cal in­ter­ven­tions, men­tal health, and non-com­mu­ni­ca­ble dis­eases com­pare to our cur­rent top char­i­ties.

In­ves­ti­gat­ing op­por­tu­ni­ties to im­prove gov­ern­ment spend­ing and in­fluence gov­ern­ment policy

Note from Aaron: This post was cross-posted from GiveWell’s web­site. To see their full table of ex­am­ples in its origi­nal for­mat­ting, please see the origi­nal post.

Some of the ar­eas we will con­sider ex­plor­ing to lev­er­age gov­ern­ment re­sources and af­fect gov­ern­ment policy are:

  • Public health regulation

  • Im­prov­ing gov­ern­ment pro­gram selection

  • Im­prov­ing gov­ern­ment implementation

  • Im­prov­ing non-pro­gram­matic gov­ern­ment capabilities

  • Im­proved or in­creased aid spending

  • Ad­vo­cat­ing for in­creased spend­ing on highly cost-effec­tive, di­rect-de­liv­ery programs

  • In­creas­ing eco­nomic growth and re­dis­tri­bu­tion

  • Nega­tive ex­ter­nal­ities of high-in­come coun­try policies

  • Im­prov­ing governance

  • Re­duc­ing the cost of health commodities

  • Im­prov­ing data collection

  • One-off big bets

How will our anal­y­sis change? How will it be the same?

Writ­ing up and pub­lish­ing the de­tails of the rea­son­ing be­hind the recom­men­da­tions we make is a core part of GiveWell. We will re­main fully trans­par­ent about our re­search.

Judg­ment calls that are not eas­ily grounded in em­piri­cal data have long been a part of GiveWell’s re­search. For ex­am­ple, we make difficult, de­ci­sion-rele­vant judg­ment calls about moral weights, in­ter­pret­ing con­flict­ing ev­i­dence about de­worm­ing, and es­ti­mat­ing the crowd­ing-out and crowd­ing-in effects of our dona­tions on other ac­tors (what we call lev­er­age and fung­ing).

As we move into ar­eas where mea­sur­ing out­comes and at­tribut­ing causal im­pact is more difficult, we ex­pect sub­jec­tive judg­ments to play a larger role in our de­ci­sion mak­ing. For ex­am­ples of the ap­proach we have taken to date, see our writeup of our re­cent recom­men­da­tion for a grant to the In­no­va­tion in Govern­ment Ini­ti­a­tive, a grant­mak­ing en­tity within the Ab­dul Latif Jameel Poverty Ac­tion Lab (J-PAL) or our page eval­u­at­ing phase I of our 2016 grant to Re­sults for Devel­op­ment (R4D). While writ­ing about such judg­ments will be a challenge of this work, we are fully com­mit­ted to shar­ing what has led us to our de­ci­sions, with only limited ex­cep­tions due to con­fi­den­tial or sen­si­tive in­for­ma­tion.

What does this mean for staffing and or­ga­ni­za­tional growth?

We need to grow our team to achieve our goals. Re­peat­edly this past year, we had to make the difficult choice to not take on a re­search pro­ject or in­ves­ti­gate a grant op­por­tu­nity that seemed promis­ing be­cause we did not have the ca­pac­ity.

We are plan­ning to roughly dou­ble our re­search team over the next few years, pri­mar­ily by adding re­searchers who have ex­pe­rience and/​or an aca­demic back­ground in global health and de­vel­op­ment. We are look­ing to add both in­di­vi­d­ual con­trib­u­tors and re­search man­agers to the team. We ex­pect that the peo­ple we hire in the next few years will play a crit­i­cal role in shap­ing GiveWell’s fu­ture re­search agenda and will be some of the lead­ers of GiveWell in the fu­ture.

For more in­for­ma­tion about the re­search roles we’re hiring for, see our jobs.