How the Giving Games Project Tracks Its Impact

(Posted on behalf of Kathryn Mecrow-Flynn. Original here.)

In early May, Chris O’Bleness and Paige Ely facilitated a Giving Game at the Pacific Northwest University of Health Sciences’ Family Medicine Interest Group (PNWU). This blog will use the Giving Game results as a case study to consider how we measure the impact from Giving Games and what we hope to achieve.

Chris is a Doctor of Pharmacy and Paige is a Student Osteopathic Doctor, both working in Washington state. This is the second Giving Game they have hosted at PNWU. They are particularly interested in promoting impactful giving from the perspective of healthcare providers through future Giving Games and their blog Practice Of Giving.

This Giving Game featured three of The Life You Can Save’s (TLYCS) recommended nonprofits; the Against Malaria Foundation (AMF), who distribute long-lasting insecticide-treated mosquito nets to prevent the spread of malaria, GiveDirectly, who provide unconditional cash transfers using cell phone technology, and Innovations for Poverty Action (IPA) who identify, evaluate, and scale solutions to social and development problems. A fourth nonprofit that has been proven to have unintended negative consequences on the program’s beneficiaries was selected as a case study to highlight the benefits of looking past the initial impression of nonprofits to the evidence for their impact.

The Giving Game was designed to consider and compare the effectiveness of nonprofits primarily focused on global health and international development. Built around the impactful, informed, and intentional giving framework, the content was tailored to resonate with an audience particularly interested in global health, public policy, and international development.

Measuring the impact of Giving Games

In all Giving Games, facilitators aim to inspire a thought-provoking discussion and the application of tools employed in high impact philanthropy for real-life donation scenarios. In this case, we additionally wished to explore with participants how considerations such as the analysis of evidence for a nonprofit’s effectiveness can be used in the design, execution, and assessment of public policy.

Our new approach to measuring the impact of Giving Games primarily looks at three core criteria:

Changes in participant attitudes toward prioritizing those factors associated with impactful, informed, intentional giving
Changes in participant plans to donate resources effectively going forward or to engage with an organisation that encourages high impact philanthropy
Money donated to effective nonprofits

We heavily rely on the administration of surveys and facilitator reports to enable us to identify strengths and weaknesses in our approach and evidence our impact. Since the beginning of this year, we have administered surveys immediately before and after the Giving Game. We additionally request email addresses to contact participants one month later with a follow-up survey.

During each of these three surveys, we ask participants to rank in order of importance six potential considerations they may have when donating. The merits and challenges of each of these considerations are then discussed during the Giving Game.

The six considerations are listed below and are phrased in the same way they appear on the survey. We believe that the “Impactful Factors” are important to high impact philanthropy and that while the “Alternative Factors” may be commonly held, they have little bearing on effectiveness. Accordingly, we hope that participants shift their views over the course of the Giving Game to give higher priority to “Impactful Factors” while decreasing their sense of the importance and prioritization of the “Alternative Factors.”

Impactful Factors:

Supported by evidence on the effectiveness of its work (Informed: people donate based on the existing evidence of a nonprofit’s impact.)
Having a high positive impact per dollar donated (Impactful: people donate based on the nonprofit’s impact per dollar donated.)
Tackling one of the world’s most critical problems. (Intentional: people donate based on the severity and scale of the issue the nonprofit is working to address.)

Alternative Factors:

Spending the majority of its funds on the cause, not its own running costs (The “overhead myth”: a nonprofit’s impact can be accurately assessed by the percentage of expenses devoted to administrative and fundraising costs versus what is spent on program costs.)
Helping people in my community (Local and international giving: people donate to nonprofits working within their own communities.)
Working on an issue I have a personal connection to (Salient: people donate if they are more knowledgeable or emotionally connected with an issue).

Tracking changes in the relative weight given to these different factors is one of the ways we measure the impact of a Giving Game. We additionally:

Collect information from participants on relevant plan changes.
Monitor the results of the preliminary vote for charities and compare this to the final vote, looking for increased voting for impactful charities.
Track subscription rates to the newsletters of The Life You Can Save and the Giving Games Project.
Employ back-end tracking with all of our recommended charities through the inclusion of Giving Games as an option when they ask “How did you hear about [name of charity]?”
Track giving by participants in Giving Games run by partners such as One for the World (OFTW).

We will also soon start asking participants if they wish to donate at the end of the Game.

Results from the PNWU Giving Game

We expect during the pre-Giving Game Survey that respondents will report that alternative factors or a mix of impactful and alternative factors are most important to their decision-making process when donating. Following a Giving Game, we hope to see impactful factors increase in importance to the participant and alternative factors decrease.

This table shows the effects of a Giving Game in changing the views of participants in the relative importance of “Impactful Factors” versus “Alternative Factors.” The data comes from the Giving Game described above. “Pre” and “Post” in the table refer to the surveys given to participants before the Giving Game and immediately after. “Top 1 Impact” refers to the number of participants who rated an impactful factor as their most important consideration when making decisions about charitable donations, while “Top 1 Alternative” refers to the participants who considered an alternative factor as their most important consideration. “Top 2 Impact” refers to those who rated impactful factors as the first and second most important considerations, while “Top 2 Alternative” is for those who put alternative factors as their first and second most important considerations. The “Top 3” for each is for those who rated either all three impactful or all three alternative factors as their top three selections. As such, these numbers are not additive. By the end of the game, ten of nineteen—over half—placed the three impactful factors as their top three factors. This was an increase of 400% from only two respondents before the Giving Game. We additionally see the number of people who answered that a charity’s overhead was the most important consideration to them drop from seven to three participants.

Looking at some of the other factors that we track, we find that Against Malaria Foundation (AMF) was awarded $375, Give Directly $500 (due to a $100 bonus for receiving the most overall votes)” and Innovations for Poverty Action, ($75). During the Game, seven participants subscribed to The Life You Can Save newsletter. Key plan changes reported include:

Resolutions to conduct research before donating. For example, “It made me realize that I should be asking more questions and conducting more research before donating.”
Awareness of the type of considerations when donating to charity. For example, “I will be more aware of possible reasons to choose or not choose an organization, and will be more self-aware when making decisions.”
Awareness of the potential impact of donations. For example, “Opened my eyes to see how I can make a better positive impact!”
Utilization of the resources available to guide donation decisions. For example, “I will definitely be utilizing the resources available to make the most impact with my charity donations.”
Awareness of cost-effectiveness as a factor in donating to nonprofits. For example, “I am glad to have a good resource to check how much of an impact my dollar.”
Resolution not to rely on the overhead myth; “I will not give as much weight to the running costs.”

Next Steps

Due to our preliminary results in this and other Giving Games (a total sample size of 240), we are currently experimenting with the hypothesis that we can have the most impact if we focus on introductory audiences. To this end, we will be directing much of our efforts on sponsoring and supporting Giving Games that can demonstrate i) An audience with relatively little exposure to our core ideas and ii) Scalable models for Giving Games that can reach many people and redirect them to more detailed materials or groups, for example One For the World (OFTW) and The Life You Can Save (TLYCS).

Additionally, we will soon be launching a Giving Game Platform to administer our surveys and voting process, and an updated website which reflects these changes. If you have any questions, please contact Kathryn at kathryn.mecrow@thelifeyoucansave.org.