Local charity evaluation: Lessons from the “Maximum Impact” Program in Israel
TL;DR
Maximum Impact program, led by Effective Altruism Israel, aims to significantly increase the impact of Israeli philanthropy by conducting nonprofit evaluations, identifying high-impact local nonprofits (even if their recipients are abroad), and promoting effective giving. Additionally, we aim to strengthen the community internally by developing a replicable model that could achieve these goals in other local communities and by building the Israeli EA community through action-oriented projects.
During the last year, we joined forces with promising non-profits and conducted and published 21 cost-effectiveness analyses. Together with an acclaimed team of expert judges, we identified 3 nonprofits as relatively cost-effective, including Smoke-Free Israel, a tobacco taxation nonprofit, which might be competitive with top charities. We learnt a lot from the pilot and have many ways to improve the program, but overall the pilot was successful: we identified high-impact local nonprofits and have engaged Israeli philanthropies interested in cost-effectiveness and in the results of the program. Finally, having a large-scale program on nonprofit effectiveness was a huge boost for EA Israel’s community building, and many people heard about EA and became more involved because of the program. We’re looking forward to continuing to maximize the impact of Israelis and to share the knowledge and experience we gained with the EA community.
Motivation
We identified that global effective giving platforms currently don’t “meet donors where they are” and only offer recommendations on global nonprofits without being able to engage donors. Donors who are initially interested in local donations but can later be introduced and nudged to give to global high-impact nonprofits are currently not targeted by global effective giving initiatives.
Israeli and Jewish donors abroad (~17.2bn per annum) currently donate almost entirely without evidence and cost-effectiveness estimations, as the Israeli non-profit ecosystem doesn’t yet emphasize these as prerequisites for donations.
With almost no research currently available on local effectiveness and a strong bias of donors towards donating locally, there are not enough impact-based donations in the local nonprofit sector.
Based on discussions with EAs worldwide, we assume this problem is not unique to Israel and would like to find a cost-effective way to solve it.
What have we done so far?
Our research program
In our 10-month program, we chose 25 promising nonprofits from more than 160 applications from Israeli-based NGOs, 21 of which successfully published their first-ever cost-effectiveness reports. In the pilot program, we provided guidance and assistance to support non-profits to execute a cost-effectiveness analysis. After the research period, we sent the reports to a team of expert judges for review, and chose the most promising non-profits based on the expert review. We offered monetary incentives to encourage nonprofits to participate: 15 nonprofits received preliminary research grants of $6k, the 3 top outstanding nonprofits received $24k prizes, and $26k was allocated to a donation matching campaign to all nonprofits who publish a report. A private donor and an Infrastructure Fund grant funded the pilot.
Guiding nonprofits on evaluation and improvement
We implemented various tools and resources to help incentivize and enable non-profits to conduct robust effectiveness analyses, including:
An extensive set of templates and how-tos.
Research grants for 10 young academic researchers (MA and PHDs) from Israel’s top universities to conduct research with the organizations and join the EA community during the process.
Personal guidance from skilled community members to plan and conduct the reports, aligned with the EA view and principles. The community members gained hands-on evaluation experience while improving the research quality.
Monthly 1-1 meetings with our Head of Research were highly valuable, helping researchers and organizations resolve specific research issues and ensure deadlines were met.
Judges and advisory board
We are grateful for the guidance and expertise that went into developing our model. We received support from esteemed local economics experts, as well as known researchers from the EA community, in developing our program and assessing the results. Our panel of judges included Claire Walsh from J-PAL, Dan Stein from IDinsight, and Karolina Sarek from Charity Entrepreneurship. Additionally, we consulted with members of SoGive, GiveWell, and GWWC about research questions and program design. Within Israel, we received counsel from leaders such as Assaf Kovo, Chief Economist for the Israel Innovation Authority, Analia Schlosser, an acclaimed empirical economics professor at Tel Aviv University, and Ahinoam Zeigel of Social Finance Israel. We truly appreciate the time and insights this distinguished group contributed to help us develop and refine a robust and successful work model.
Success
This was the program’s pilot year, and there are positive outcomes and things that can be improved.
Top nonprofits found
In terms of output, we published 21 cost-effectiveness analyses on Israeli nonprofits, becoming the first public database on nonprofit effectiveness in Israel. Although there is variance in research quality, all analyses were conducted based on a relatively high standard. We’ve shown we can conduct thorough analyses at a relatively low cost while still using high-quality methods. Our analyses cost an average of just $6,300 each yet over half employed quasi-experimental designs. With adequate resources, we can scale this process and maximize the potential we’ve identified during the pilot.
While further research is required to verify the results, we identified a few promising nonprofits. The most promising organization among those we evaluated (and probably one of the most effective nonprofits in Israel) is Smoke-Free Israel. Our evaluations suggest they may be more effective than GiveWell’s top charities, with an estimated 7.18 QALYs per dollar (This is in line with the CE report, WHO, and multiple resources, suggesting that tobacco taxation may be highly cost-effective). Although they currently have limited room for funding, identifying and quantifying their success is important in helping them continue their work (they’ve already presented the research done in the program to the Israeli Knesset in a debate on tobacco legislation) and to show that highly cost-effective global health nonprofits could potentially be found outside of GiveWell’s top charities and in local contexts in developing countries. We rewarded two other organizations for their relatively high levels of effectiveness. NALA, a WASH intervention in Ethiopia, and Kav LaOved that help migrant workers from developing countries in recovering unpaid wages, an important yet neglected cause. We were also impressed by the Better Plate accelerator operated by MAF, a local accelerator focused on alternative protein startups.
To improve our identification of top nonprofits, we collaborated with the EA group at Tel Aviv University for a hackathon. Over 80 participants worked on developing a technological tool and a robust methodology for identifying promising non-profits using publicly available data. We are currently collaborating with the winning team to build an MVP that we hope will be operational soon. We aim to implement this tool in the upcoming year to enhance our outreach efforts and proactively recruit high-potential non-profits based on the outcomes.
Collaboration with the philanthropic sector in Israel
Our most significant achievement has been cultivating partnerships with leading philanthropic organizations and individuals across Israel. We aim to bridge research and philanthropy advising establish ourselves as a major influencer in the sector, and establish our reputation as providing high-quality effectiveness research.
We’ve had some success since the end of the research program. We launched a website and donation platform published this year’s research and which will soon feature global high-impact nonprofits (GiveWell, GWWC recommended charities) alongside Israeli nonprofits. We had a successful event where we presented the program and discussed the importance of evaluation in nonprofit work. Two major local philanthropies sponsored the event, a few others attended, and we have meetings scheduled with a few more in the coming weeks to explore potential collaborations and talk about the importance of effectiveness based giving. We were featured on Israel’s most popular economics podcast, reaching over 300k monthly listeners, and we’ve been told that a senior official in Israel’s Ministry of Finance heard about EA and became interested through the podcast.
We have leveraged the success of pilot by actively building new connections and reinforcing existing ones. We have established valuable partnerships with numerous Israeli philanthropic entities, including the largest funders network operated by Jewish philanthropists internationally. Additionally, we have forged strong ties with the largest local Donor-Advised Fund (DAF). Furthermore, we have initiated new collaborations with multiple Impact-tech companies, expanding our network and fostering innovative partnerships.
Onboarding young professionals to high-impact career paths
Part of the impact comes from onboarding promising young professionals into impactful research since we see ourselves as coming from and working with the EA community. By connecting young professional researchers, and local nonprofits, our process provides valuable skills to organizations while offering hands-on research experience aligned with EA principles and cost-effectiveness analysis methods to the researchers.
Along the way, we’ve helped nurture emerging careers. One participant from our hackathon is joining the Rethink Priorities “AI Governance & Strategy” team next month. Meanwhile, four researchers continue to collaborate with their non-profits locally and abroad (at the moment, one of them is running RCTs in Zambia), three others have asked to further build expertise by joining our program managing team, one applied to a research role at GiveWell and another one joined the evaluation team at one of the philanthropies we’ve collaborated with, hands-on research experience seems to inspire new career paths. We believe diving into fieldwork with guidance from experts can be a useful complementary product to EA traditional career advising.
Build a replicable model
Part of the Maximum Impact program’s goal is to roll out our model to other countries so that they can run their own version of the program and in order to replicate our success. EA Israel has already done this in the community building space, and our templates are referred to often and serve as the basis for an SOP (Standard operating procedure) for local and national groups being built by CEA.
During the pilot, we created an extensive toolkit that can support the implementation of our program in additional countries and communities. It may also prove useful as easy-to-use resources for monitoring and evaluation planning or teaching purposes. We have received expressions of interest in carrying out our work in multiple locations such as India, Brazil, Switzerland and Turkey.
What was less successful?
Our research design still can be better—To be as rigorous as we can, we’ve most of our research is empirical and quasi-experimental designed (and RCTs when possible) based on the outputs of each nonprofits. However, the final result can be improved to achieve higher internal and external validity, and we have many ideas how we can improve the quality of research in coming rounds,
Our selection process should be improved—some of the non-profits we’ve decided to collaborate with turned out to not be mature enough to carry out an in-depth evaluation. This led to a lower-quality database and limited staff power we could use to support our research. For the next round, we will revamp our screening process and do active outreach to the most promising nonprofits.
Insufficient funds shifted to effective nonprofits - With the research completed and a donation infrastructure in place, we hope to shift funds to high-impact nonprofits. However, we haven’t yet seen the results we’d like, and we have shifted only $10k in the month to the outstanding nonprofits on the websites. It is promising that over 85% have supported the most effective nonprofits in the program, even though it’s possible to donate (and receive matching) to all participating nonprofits. We are now dedicating significant effort to establishing stronger connections with local donors and philanthropies and developing our marketing efforts and brand. We believe that some hard work on this aspect should deliver much more significant outcomes in the near future.
What’s next?
We’re evaluating the program (Keen thanks to the Rethink Priorities survey team, which helps us a lot in the questioning process!), conducting exit interviews with the participating nonprofits, and planning the next round. We have some open questions about the format, but we want to base the next program on the success we’ve gained and the proof of working closely with nonprofits and philanthropy.
The format we’re leaning towards is one where we’d partner with Israeli philanthropies and provide Evaluation-as-a-service to them while independently researching a smaller group of especially promising nonprofits. This would enable us to strengthen our connections with philanthropy and strengthen our sustainability, while creating more public knowledge on the best giving opportunities in Israel.
We aim to add global effective giving nonprofits to our website and fundraise for them, while “nudging” Israeli donors who are interested in effectiveness to donate globally.
Funding permitting, we want to finish our template toolkit and implement the program in another country, ideally one that has a bias to local giving similar to Israel and potential for high-impact nonprofits (India is a promising option).
How you can get involved
If you know funders interested in spreading nonprofit evaluation worldwide or in an innovative approach to action-based community building, please contact us!
If you know Israeli nonprofit funders interested in maximizing their portfolios’ impact, let them know about the program.
If you want to implement a nonprofit evaluation program in your country, please reach out!
If you’re interested in reviewing the cost-effectiveness analysis of the top nonprofit we identified and how it compares to global top nonprofits—we’d love to see that done!
Great initiative! It’s good to see that charities outside of the anglosphere are also being analysed and in some cases, even turn out to be very effective.
Thanks! I guess GW’s top charities will remain my go-to, but happily, we found some promising options for local donations
Fantastic work! It’s awesome to see a national EA chapter taking on such an ambitious project and having the follow through to make it happen.
I just want to clarify what you mean when you say “most of our research is empirical and quasi-experimental designed (and RCTs when possible) based on the outputs of each nonprofits”: I assume this means that your research uses existing empirical (preferably quasi-experimental or better yet RCT) evidence . You don’t mean you’re actually conducting or funding any primary research, right? I ask because that would be insanely cheap and fast
Good point, thanks for bringing this up.
As you’ve mentioned, with the time constraints of the program, it was unreasonable for all data collection to occur within the program timeframe. So in most cases, we relied on high-quality evidence from prior research conducted by the nonprofits. So yes, the cost per nonprofit I mentioned doesn’t include data collection in most cases (which can be VERY costly, specifically when doing RCTs)
Some assessments did leverage experiments that had started before and continued during the program. And in select cases, we assisted nonprofits with parts of their data collection and experiment implementation through facilitation.
Some questions/comments from a quick reading of this and the link, hope it’s OK.
Does this analysis assume that Smoke Free Israel’s campaign was pivotal in the tax reform? How can that be justified? Did you consider that it might have just led the reform to occur slightly earlier?
How confident are you that there is really room-for-more-funding for the next campaign, or will SFI likely get enough funds to run it even without EA involvement?
Given what (I think) is limited room for more funding, perhaps you should be highlighting and boosting the other highly rated charities a bit more? Consider also that people may not look literally at all your details and may use it as a loose proxy. The ‘Smoke Free Israel is the best’ charity might get interpreted as ‘charities that work on domestic Israeli politics/health issues are among the most effective.’ Whereas I suspect that in general, health interventions in poor countries, and perhaps also the alternative proteins work, may be more effective as a general recommendation.
Hi!
From what I understand from conversations with SmokeFree Israel’s staff (which admittedly might be biased) is that they were the only body pushing the legislation forward, and they had to work AGAINST the existing legislation. SFI wokred to fix problematic loopholes in the update to the tobacco taxation policy that had recently been passed, and petitioned to external legal bodies to help force the government to put the policy back on the agenda. They also provided the data and expert opinions that were pivotal in the discussions within the legislature once the issue had returned to the agenda.
Regarding room for funding—that point is entirely valid. We don’t think that SFI replaces AMF or MC as a top charity that everyone should donate to, but is evidence that more highly cost-effective opportunities exist if you look for them.
On SFI’s room for more funding—they’ve said they’re not looking to expand internationally right now. So with their current scope, I agree their funding capacity seems limited, even maintaining effectiveness. We’ve noted that for potential donors.
The tax reform question - @EdoArad was the internal reviewer for SFI’s report. He’d be the best person to speak to that.
And I totally agree with encouraging local donors to give across each of the top nonprofits we identified, without focusing on SFI.
Within this forum, we highlighted SFI since they were the only one we found already competitive internationally. But absolutely, spreading support across all 3-5 effective charities we identified is ideal for local donors.
Regarding the first question, I just briefly looked again at the report and indeed I don’t see that explicitly taken into account. I only vaguely remember thinking about that, and I’m not sure how that was resolved.
I think the main causal pathway they used in their report is QALY gains from people quitting smoking, so that sounds like it wouldn’t change drastically if the intervention was delayed by, say, a couple of years. So I agree that this is a good question to look into further, and I expect that could indeed reduce the cost effectiveness by 3x-10x. Great catch David!
Impressive work you have done and clearly written report! Looking forward to follow how you scale it up/duplicate your effort to other countries :)
This is incredible ! I’m very happy about this initiative and hope other national orgs will draw inspiration from it. I think this is also a wonderful way of making EA known. Courage for the rest !
Happy to hear that, Thanks!
Hello! Want to chat with you about your project. It could help EA Armenia that has quite a similar goals. Please replay in dm or share you contacts)