Book Giveaway Impact Analysis (Doing Good Better in NZ)
~11min read
Introduction
This post provides an overview and analysis of the Doing Good Better book giveaway through Effective Altruism New Zealand (EANZ). The analysis covers data collected from survey responses between 05-Jan-17 and 17-Dec-19, for which there were a total of 298 responses, with appreciable variance in the amount of the survey which was completed. This analysis was initially completed around Jan 2020 so any reference to “to date” refers to then.
All monetary values are in NZD.
Tables with specific data are reported in the appendices at the end of the post.
Epistemic Status: I am not highly confident about specific values, for reasons of self-reporting on behalf of the respondents, and some assumptions on the analysis side.
Overview
The book giveaway is accessed through an online form on the EANZ website. The online form did not ask specific questions on donations—so part of the analysis had to be done by comparing survey results with donation data from the EANZ Charitable Trust (which forwards tax-deductible donations to a select number of GiveWell top charities).
Data was collected in two ways:
Tracking book recipients who started donating to the EANZCT after receiving the book.
Book recipients were emailed an anonymous survey, 3 months after being sent a book.
Key Outcomes
Book recipients on average donated $28 to the EANZCT in the first year after they received the book. Deducting the costs of the book & delivery ($15), this accounts for a net increase of $13 in the first year, and $38 in the first two years. This is likely to be an underestimation as people may be donating through other channels or to other charities, and it doesn’t track donations from people who received the book from their friends.
Survey respondents reported they would give $232 per year more to effective charities as a result of reading the book. This equates to $56 per book, per year. However, these numbers have a high uncertainty as: a) non-respondents may also still donate, b) social-desirability bias may be inflating numbers, and c) people may not actually donate what they said they will.
29% of respondents reported that reading the book made them reconsider their career path, and 30% their dietary choices.
The main suggestion to increase the accuracy of the data and of a future analysis is to have a more thorough pre-giveaway survey that includes similar questions as the follow-up survey. This would decrease the need to compare different sources of data and reduces reliance on donors who donate to effective charities via the EANZCT. Surveying first-time EANZCT donors may also provide valuable data.
Survey Questions & Results
The follow-up survey was sent via email to giveaway participants approximately 3 months after being sent a book. Bracketed numbers on graphs indicate total responses to the question.
Section 1 - Pre-book Data
The first section of the survey assessed the knowledge, beliefs and donations of the participants before they were sent the book.
Prior EA Knowledge
As we would expect, the majority of people who received the book had heard little to nothing of EA before applying for the giveaway.
EA Affiliation
As expected, the majority of people didn’t consider themselves an EA prior to the book. 13% (38) of respondents said they considered themselves to be an effective altruist. 67% (196) responded no, and 20% (57) responded maybe.
Surprisingly, there were 8 people who didn’t know what effective altruism was, yet still considered themselves to be EA’s. The table below shows the responses of those who answered “Yes” to considering themselves an effective altruist before the book. It appears there isn’t a strong correlation between how much someone knew about EA prior to the book and whether they consider themselves an effective altruist. Some respondents chose both “I didn’t know what effective altruism was” and “I had been to effective altruism meetups”, which is particularly peculiar.
| # Yes EA | Response |
| 8 | I didn’t know what effective altruism was |
| 9 | I had heard of effective altruism, but didn’t know much |
| 11 | I had read an article or watched a video or two about effective altruism, or had an in depth conversation with someone about effective altruism |
| 10 | I had read several articles or videos, or read a book about effective altruism |
| 8 | I had been to effective altruism meetups (You can check this box as well as one above) |
However, for those who said they “maybe” considered themselves an EA before the book (57), they mostly either didn’t know what it was or had heard of it but didn’t know much.
| # Maybe EA | Response |
| 19 | I didn’t know what effective altruism was |
| 18 | I had heard of effective altruism, but didn’t know much |
| 12 | I had read an article or watched a video or two about effective altruism, or had an in depth conversation with someone about effective altruism |
| 8 | I had read several articles or videos, or read a book about effective altruism |
| 4 | I had been to effective altruism meetups (You can check this box as well as one above) |
Prior Donations
The largest number of respondents donated between $100 and $999 prior to reading the book. The overall total was $342,000, an average of $1,200 and a median annual donation of $100 (both including the $0 responses).
The donations according to how the respondents identified with EA are shown in the table below. On average, the more someone identified with EA, the higher their annual donations to charity, which is nice to see.
| Total | Mean | |
| Yes EA (38) | $94,100 | $2,480 |
| Maybe EA (57) | $90,100 | $1,610 |
| Not EA (196) | $157,600 | $820 |
Another question asked in this section was: “If you donated to charities in the past, what charities did you support ? (Give up to four charities)”. This data was used to inform the later analysis on increased funds directed to effective charities. (See Appendix B for calculations)
On average, 10% of the charities which respondents donated to were effective charities, prior to reading the book.
Section 2 - Post-book data
The second section of the survey asks respondents if they have read the book, and assesses behaviour/belief changes of those which have read at least part of it.
Reading completion
About half of respondents had read the book, with 84% (247) having read at least part of it. These respondents were asked follow up questions about behaviour change. Those who said they had not read the book were not asked follow up questions.
Donation influences
63% (149) of respondents said reading the book influenced their charity choices, and 48% (114) said they would donate more. Respondents were asked how much more or less they would donate, depending on their response. Of the 114 respondents who said they would donate more, 94 said they would donate more than $0. Overall the result was a 26% increase in planned annual donations due to reading the book. Three respondents stated they would donate less, one of which ($2000 less) was due to “increasing volunteering hours in their local community and to develop start-up social enterprises”, and another due to donating more time.
Note: This is general charitable donations, not specifically donations to effective charities.
On average, 34% of the charities which respondents said they donate to now (post-book), or intend to donate to in the future, were effective charities.
EA Affiliation
31% (70) respondents now consider themselves part of the EA community as a result of reading the book, with 45% (101) unsure and only 9% (20) responding no.
Career & diet changes
Respondents were asked whether or not they were reconsidering their career path, or diet, as a result of reading the book. 29% of respondents said that reading the book made them reconsider their career path, and 30% would reconsider their dietary choices. 14% (32) respondents said that reading the book made them reconsider both their career path and dietary choices.
Section 3 - EA Influence
In the final section, respondents were asked about whether they had introduced others to EA ideas, and if so, how they responded to these.
Of 245 respondents, the total number of people they had lent/given the book to was 227, meaning that on average each person passed the book on to (just less than) one person. However, as a result of reading the book, 222 respondents reported that they had introduced a total of 1363 people to EA ideas, with a mean of 6 people and a median of 3 people. The majority (139, 68%) of 205 respondents who had introduced people to EA ideas said that the response to the ideas was positive. One respondent reported that there was a negative response.
Key Takeaways
Expected/reported changes to effective donations
The respondents who already donated to effective charities, and would continue to do so, said they would donate an overall $23,700 more to effective charities (per year), with an average of $1,820 and a median of $250 (per respondent per year). Those who donated to ineffective charities and switched to effective charities would reportedly donate a total of $45,200 more to effective charities, with an average of $900 and a median of $440. This brings an expected total of $68,900 more going to effective charities, which accounts for 78.75% of the total increase in donations.
With 297 respondents to the survey, this results in an average $232 increase in expected funds going to effective charities per respondent.
With 1244 books sent, this equates to an average $56 increase in expected funds going to effective charities per book.
Notably, one individual reported they would donate an estimated $660 less to effective charites due to “reduc[ing] my working hours to do more volunteer work in my local community and work to develop new start-up social enterprises”.
Donors to EANZ Charitable Trust
In order to calculate the ‘actual’ cost-effectiveness of each book given out, the amount donated to the EANZCT by donors who made their first donation after receiving the book was investigated. It was found that within one year of receiving the book, the total amount donated was approx. $35,000, equating to $28 per book sent. The donations were also summed to date and this was found to equate to $53 per book given away. This provides a strong case for the effectiveness of the giveaway of the Doing Good Better books.
| Total number of books sent | 1244 |
| Donations within one year of receiving the book | $ 34,756 |
| Donations to date | $ 66,393 |
| Donations within a year of receiving book per book sent | $ 28 |
| Donations to date per book sent | $ 53 |
| Cost per book incl. freight | $ 20 |
| Net donation per book sent (per year) | $ 8 |
| Net donation per book sent (to date) | $ 33 |
Considering the cost per book including postage is $20, it was found that the net donation to effective charities per book per year was $8, and $33 to date! Evidently this isn’t highly aligned with the expected increase in donations based on the survey, however it is possible donors made donations through channels other than the EANZCT. Perhaps the survey can be used as a proxy for actual donations.
Noteworthy free-responses
Respondents were given the opportunity to make additional comments at the end of the survey. Here are some noteworthy ones (they mainly focus on the book itself, so feel free to skip):
- “The book was well worth reading, glad I picked it up, it will change the way I go about giving in the future, in effort to make a better difference”
- “Book giveaway is a good idea, in my opinion. Many people I have spoken to cite the book as the thing that got them initially involved in EA—this survey may say the same but from personal experience, many of those who are most involved got involved at least partially due to the book.”
- “I read the book straight away and found it very informative. I usually get all my books from the library, so wouldn’t have bought it if it wasn’t free. You have reminded me to get the book back from the person I loaned it to and give it to someone else.”
- “I think giving the book away was a good idea. Based on it’s cover / title, I would not have purchased the book.”
- “It was very good so far but my brain started to hurt and I had to take a break, have lent to a friend and will get it back later. Perhaps a “for dummies” version could be good?”
One other point of note is that several people mentioned that they wish there was more coverage on the effectiveness of NZ charities. This seems to be a recurring theme when introducing people to EA here in NZ (and I am uncertain what should be done about it, if anything at all).
Key Improvements
Include a question for first-time EANZCT donors to ask if their donation was a result the book giveaway
Expand the pre-giveaway data collection to include questions about donations, and specifically ask in both the pre- and post- giveaway survey about current donations rather than planned/expected ones
Include a dropdown list of the major charities, with an “other” option (would save a LOT of time on analysis)
At the end of the survey, ask for contact details of respondents and whether they would be ok with us following up with them regarding their responses.
Some individuals mentioned they would be interested in participating in events / learning more—however we had no means to reach out to them.
Other ideas
Develop a system to include book identifier numbers that go with the book e.g. sticker. This will support better tracking and follow-up systems for when the book is passed on—essentially increasing the impact of each book (or at minimum allowing for better data capture).
Potentially include questions around demographic data, though I’m uncertain how valuable this would be
Final Thoughts
I think the book giveaway is a worthwhile endeavor—especially as word of EA grows worldwide. It provides people with a deeper dive into EA topics and introduces the core concepts. Furthermore, the data indicates the giveaway directs more funds to effective charities than the costs to run the giveaway, which, coupled with the other effects (awareness of EA, potential career trajectory changes, etc) suggests the giveaway is effective. With some minor adjustments and improvements to the system/process—I am interested to see the results in a year or two’s time.
If anyone would like to start a book giveaway, has suggestions on how ours could be improved, or has other thoughts—I’d love to hear about it!
Appendices (Google Doc)
Appendix A: Specific response data per question
Appendix B: Calculation of effective donations
Edit: Amended book + delivery cost to $20, from $15.
As a NZer who recieved Doing Good Better through the book giveaway process, I can definitely say that it has had a significant impact on my decision to be more involved in EA and certainly triggered deeper thoughts surrounding career opportunitites, donating and eating habits. Furthermore I have also passed the book on to friends which I hope allows them to get as much benefit as I did. I am currently donating around 5% of income and plan on taking the 10% pledge next year as I enter my second year working full time, so from that perspective the book has certainly had a measurable impact.
Nice work Ivan! We’ve been considering doing this at Giving What We Can. Quick one: how did you get the cost of the book + delivery down to $15?
Thanks Luke! Looking forward to getting some better data after we improve the system. I think it would be a good project for GWWC to implement. Perhaps with the larger scale you might be able to automate a few things (e.g. various follow ups) and get more out of each book.
I’ve just followed up about the cost. Some wires must have got crossed somewhere down the line, as it seems that the cost of book + delivery is actually $20 (so that would make the book itself ~$15). Woops. I’ll update the post with this.
Thanks!
I think that this would be amenable to a randomised controlled trial. (Whether by you or someone else.) This would address some of the concerns from the methodology you used here. E.g. how much would people donate without receiving the book? How much support would they express for EA?
You could send 50% of people who asked for it a book that wasn’t actually focused on EA. E.g. a book about productivity or something else, where you could frame it in a way that it wouldn’t be misleading/disappointing to receive this book rather than DGB.
Or you could put people on a wait list control group, where they still get sent DGB, it just comes ~3 months later, and all participants (whether they’ve received the book or not yet) fill out a second survey ~3 months after they first request it.
I’m going to be writing some forum posts soon about the possibilities for this sort of experiment in EA movement building, following on from two such experiments we have run at Animal Advocacy Careers. A lot of the methodology could be similar!
Thanks for doing this. Have you considered suggesting the EA infrastructure fund funds much wider book giveaways?
That’s a good suggestion, thank you. I hadn’t considered that. I would feel more confident in making a suggestion to the EA infrastructure fund after we implement the improvements and do a follow up analysis. However, the EAIF wouldn’t administer the books themselves—it would probably need to be done via some organization (e.g. CEA).
CEA already gives away many books through the virtual programs, so perhaps getting people onboarded onto those is a good strategy. Alternatively, groups can also apply for funding for book giveaways through CEA.
I could see a book giveaway being offered alongside the virtual programs being potentially valuable—but the question would be if it would have a net value increase, or if it would detract from the value of the programs (due to some percentage of people choosing the book over committing to the program, and then not actually reading the book).
I have a feeling it is probably net positive, and could potentially be a precursor to the virtual program, as it provides more flexibility and less commitment. But again, wider distribution seems to be more of an organization-level implementation than a particularly ‘funding-constrained’ issue.
Thanks Ivan, this is really excellent and interesting. It makes me update towards believing that book giveaways are impactful. With that said, you probably have some sample bias (i.e., the people who responded to the survey are probably different from those who didn’t in meaningful ways). That probably means that the impacts are somewhat smaller than the survey suggests.
If you are still running the survey, it would be interesting to give people who ask for the free book the option to complete a presurvey so you have some pre and post measures. Also, I think that you might have forgotten to link to Appendix B.
Thanks Peter.
I agree about the sample bias. Which is part of the reason why I think the data from actual donations is relevant—rather than relying on respondents self-reported figures.
Implementing a presurvey is one of the top improvements to action—as I do expect that to help a lot. If you have any particular suggestions for useful pre/post measures, I’d love to hear them.
Appendix B is contained within the Appendices (Google Doc).
Also ‘outside the quote text bar’ to see if that does anything different.
Sorry I’m not sure what you mean here?
Thanks, updated!
I have a feeling that if the argument of EA can motivate a people has a lot to do with a person’s natural disposition and circumstance. It would be hard for many people to give away the notation that “we should take care only of ours”. And it’s also difficult for someone with a lot of personal issues to think about others. People who have the time and energy to actually read a book of EA perhaps are already disposed to accept its ideas.