Hi! Writing as a Research Advisor at Founders Pledge and formerly staff at J-PAL. I really enjoyed reading this report and was very impressed by the detail, clarity, and brevity (30 pages isn’t short, of course, but this could’ve been much wordier—it’s very to-the-point). Some specific things I found particularly exciting:
(1) The purpose of this report and the goals/values of the donors were made very clear at the beginning and referenced throughout the document. This is surprisingly easy to leave out, especially when writing for a familiar audience (like Effective Altruists) whom we may expect to have identical worldviews to our own.
(2) It was great to see your rationale for selecting your top interventions, as well as the reasons you chose not to select others from Table 1. It helped provide a complete picture of how and why you arrived at your conclusions.
(3) It was interesting (with no positive nor negative connotation—just truly interesting!) to see the GiveWell-evaluated health interventions included in the category of early childhood development. I’ve never seen them framed in that way, but it made sense to me while reading.
Other organizations you could consider looking into:
(1a) BRAC is, of course, the creator of the approach and a very impressive and evidence-focused organization. I think it would be worth getting in touch with BRAC to see if donors of this size could earmark support to the Ultra-Poor Graduation Program in Uganda.
(1b) BOMA Project implements a version of the Graduation approach in a few countries, with most of its work in Kenya. I don’t know them well, but they’re often speakers at conferences with sessions on the Graduation approach and related social protection programs.
(1c) Trickle Up also implements the Graduation approach on the African continent and elsewhere. I have a positive impression of them from a few meetings with their M&E team, but don’t have deep knowledge about their programs so can’t recommend for or against.
(2) BRAC’s Empowerment and Livelihoods for Adolescents program is also something that might be of interest. It’s been studied with RCTs in Uganda and Sierra Leone (the latter study is really interesting as it took place during the Ebola outbreak) and found to have positive results on income and education levels. I’ve never seen a cost-effectiveness analysis, but the SL RCT notes that the program “has proved to be scalable and cost-effective across countries” (p. 11).
(3) J-PAL’s Innovation in Government Initiative (IGI). I’d guess this was left out because it fits into the “evidence-based policy” category, but the competitive nature of the fund means that the projects it supports are more like TaRL Africa’s work (specific, discrete technical assistance to meet policymaker demand) than generic advocacy. IGI is the recommended charity from Founders Pledge’s report on evidence-based policy, and fits the “transformative” criterion in a slightly different way—enabling the transformation of entire government programs that are designed to improve lives.
A few notes and questions I had while reading:
(1) Did you assess try to assess organizations’ track record/strength when making these recommendations? From my experience, both TaRL Africa and Village Enterprise are exceptionally strong, evidence-based organizations with strong leadership and implementation on the ground. It could strengthen your recommendation to include a note about their commitment to evidence use and generation, local partnerships, etc.
(2) For Village Enterprise in particular, it would be good to mention the Sedlmayr, Shah, and Sulaiman (2018) study that directly compared VE’s program with a cost-equivalent cash transfer in Uganda. This can help bolster your reasoning for selecting them over GiveDirectly (although the cash only results were really noisy and hard to interpret—I hope an updated paper will be released sometime soon).
(3) I was confused about your cost-effectiveness estimates for TaRL until I looked at the linked spreadsheet. The intervention can vary along a number of axes, and both the costs and impacts will vary based on the local context and population served, so I wasn’t sure what a cost-effectiveness estimate for the TaRL approach meant. I see from the spreadsheet that your cost estimates came from the studies with Pratham in India—I think it would help to clarify that in the document itself.
Regarding the other organizations that implement the graduation approach in Africa, I did indeed look into them (see the report on cash transfers for details). BRAC might have been a good choice, but they run a very wide range of programs, and it was alot harder to figure out what would be the net impact of even a targeted donation to their graduation approach program (see e.g. this GiveWell post on BRAC). I was also influenced by discussions with Founders Pledge who also looked into these different charities and ended up recommending Village Enterprise.
As for J-PAL’s Innovation in Government Initiative, I personally find it very interesting. I don’t know if it would be a good fit with the donors’ preferences. They will be going over the comments here, though, so they’ll get a chance to see some of the other options that have been proposed.
Responses to the questions you posted:
1)Yes, I did try to assess organizations’ strength and track record, though admittedly that was harder to do than going over RCTs and literature. It was part of the motivation for recommending TaRL Africa and Village Enterprise. I’ll look over the text again and see if I can make that clearer. Mind you, all the other organizations that made it to Table 1 also seem to have a strong track record. I partly relied on the fact that many of these had been vetted by GiveWell and Founders Pledge.
2) and 3) I’ll do that, thanks for the suggestion :)
samcart, just a quick note that TaRL Africa is (or was) part of J-PAL’s Innovation in Government Initiative (or its predecessor, the Government Partnership Initiative). See these links:
Hi! Writing as a Research Advisor at Founders Pledge and formerly staff at J-PAL. I really enjoyed reading this report and was very impressed by the detail, clarity, and brevity (30 pages isn’t short, of course, but this could’ve been much wordier—it’s very to-the-point). Some specific things I found particularly exciting:
(1) The purpose of this report and the goals/values of the donors were made very clear at the beginning and referenced throughout the document. This is surprisingly easy to leave out, especially when writing for a familiar audience (like Effective Altruists) whom we may expect to have identical worldviews to our own.
(2) It was great to see your rationale for selecting your top interventions, as well as the reasons you chose not to select others from Table 1. It helped provide a complete picture of how and why you arrived at your conclusions.
(3) It was interesting (with no positive nor negative connotation—just truly interesting!) to see the GiveWell-evaluated health interventions included in the category of early childhood development. I’ve never seen them framed in that way, but it made sense to me while reading.
Other organizations you could consider looking into:
(1) BOMA Project, Trickle Up, and BRAC all implement the Graduation approach on the African continent.
(1a) BRAC is, of course, the creator of the approach and a very impressive and evidence-focused organization. I think it would be worth getting in touch with BRAC to see if donors of this size could earmark support to the Ultra-Poor Graduation Program in Uganda.
(1b) BOMA Project implements a version of the Graduation approach in a few countries, with most of its work in Kenya. I don’t know them well, but they’re often speakers at conferences with sessions on the Graduation approach and related social protection programs.
(1c) Trickle Up also implements the Graduation approach on the African continent and elsewhere. I have a positive impression of them from a few meetings with their M&E team, but don’t have deep knowledge about their programs so can’t recommend for or against.
(2) BRAC’s Empowerment and Livelihoods for Adolescents program is also something that might be of interest. It’s been studied with RCTs in Uganda and Sierra Leone (the latter study is really interesting as it took place during the Ebola outbreak) and found to have positive results on income and education levels. I’ve never seen a cost-effectiveness analysis, but the SL RCT notes that the program “has proved to be scalable and cost-effective across countries” (p. 11).
(3) J-PAL’s Innovation in Government Initiative (IGI). I’d guess this was left out because it fits into the “evidence-based policy” category, but the competitive nature of the fund means that the projects it supports are more like TaRL Africa’s work (specific, discrete technical assistance to meet policymaker demand) than generic advocacy. IGI is the recommended charity from Founders Pledge’s report on evidence-based policy, and fits the “transformative” criterion in a slightly different way—enabling the transformation of entire government programs that are designed to improve lives.
A few notes and questions I had while reading:
(1) Did you assess try to assess organizations’ track record/strength when making these recommendations? From my experience, both TaRL Africa and Village Enterprise are exceptionally strong, evidence-based organizations with strong leadership and implementation on the ground. It could strengthen your recommendation to include a note about their commitment to evidence use and generation, local partnerships, etc.
(2) For Village Enterprise in particular, it would be good to mention the Sedlmayr, Shah, and Sulaiman (2018) study that directly compared VE’s program with a cost-equivalent cash transfer in Uganda. This can help bolster your reasoning for selecting them over GiveDirectly (although the cash only results were really noisy and hard to interpret—I hope an updated paper will be released sometime soon).
(3) I was confused about your cost-effectiveness estimates for TaRL until I looked at the linked spreadsheet. The intervention can vary along a number of axes, and both the costs and impacts will vary based on the local context and population served, so I wasn’t sure what a cost-effectiveness estimate for the TaRL approach meant. I see from the spreadsheet that your cost estimates came from the studies with Pratham in India—I think it would help to clarify that in the document itself.
Hi samcart,
I’m glad you liked the report :) Thanks for the detailed and constructive feedback! Note: One reason that it isn’t too long is that other reports preceded it (on education, early childhood development, small and medium enterprise support, rural electrification and long-term effects of cash transfers). This final report tries to summarize things.
Regarding the other organizations that implement the graduation approach in Africa, I did indeed look into them (see the report on cash transfers for details). BRAC might have been a good choice, but they run a very wide range of programs, and it was alot harder to figure out what would be the net impact of even a targeted donation to their graduation approach program (see e.g. this GiveWell post on BRAC). I was also influenced by discussions with Founders Pledge who also looked into these different charities and ended up recommending Village Enterprise.
As for J-PAL’s Innovation in Government Initiative, I personally find it very interesting. I don’t know if it would be a good fit with the donors’ preferences. They will be going over the comments here, though, so they’ll get a chance to see some of the other options that have been proposed.
Responses to the questions you posted:
1)Yes, I did try to assess organizations’ strength and track record, though admittedly that was harder to do than going over RCTs and literature. It was part of the motivation for recommending TaRL Africa and Village Enterprise. I’ll look over the text again and see if I can make that clearer. Mind you, all the other organizations that made it to Table 1 also seem to have a strong track record. I partly relied on the fact that many of these had been vetted by GiveWell and Founders Pledge.
2) and 3) I’ll do that, thanks for the suggestion :)
samcart, just a quick note that TaRL Africa is (or was) part of J-PAL’s Innovation in Government Initiative (or its predecessor, the Government Partnership Initiative). See these links:
https://www.povertyactionlab.org/blog/1-16-18/increasing-use-data-and-evidence-real-world-policy-stories-j-pals-government
https://www.povertyactionlab.org/blog/8-8-17/zambia-scale-teaching-right-level-program-1800-schools