First, thanks very much for publishing this! I really appreciate the transparency and I think the entire EA community can learn from the efforts here.
Second, it’s awesome that we’re aiming to get a higher profile to EA and introduce it to more people. Introducing things through the media does seem to be quite high value to me, and I think we should keep on doing it.
Third, I think the “experimental” approach is right-on and your initial skepticism of the absolute impact of media and differential relative impacts of different media are smart. And it’s definitely true that it’s exceptionally hard to get good data here. Media causality is a really tough challenge. Hence the old quote “I know half of my advertisements are worthless, I just don’t know which half”.
And fourth, wow for getting such a widely shared article!
-
From their [GiveWell] 2013 metrics data, $3mn was moved via small donors (less than $10k) in 2013. On average, 14% of donors came via online referral; assuming that this is representative for small donors that would give $420,000 via online referral. There were 151,000 visitors via referral links, giving a money moved per visitor figure of $2.78. Using this number, the Quartz article would have moved $27,800 (not including regular giving beyond a year). This was quickly estimated, and there may well be errors in this assessment, and still seems somewhat too high to me. We could therefore give cautious bounds of between $5,000 and $50,000 moved as a result of the article.*
First, I find it hard to believe that $2.78 is really generated via online referral; I imagine some data integrity problems or sampling problems are leading to a large skew. I expect the marginal value of an additional unique visit to GiveWell is much less than $2.78.
Additionally, relevant to writing articles, I expect there to be large differences in the relative values of different kinds of online referrals. For example, it seems quite plausible to me that someone coming to GiveWell from a piece about how awesome GiveWell is versus someone coming to GiveWell from a piece about how much the ice bucket challenge sucks will become donors at much different rates and donate different amounts.
-
Soon after the article, we received 3 pledges and 8 try out givers who cited ‘on-line article’. We’ll follow up with these people, but it’s likely that not all were influenced by the Quartz article, rather than some other online article.
That’s awesome—would be great to hear how the follow up goes and if they were truly inspired by the article.
-
If we assume 3 pledges only, the article raised approximately $1 million in pledged donations.
I expect successful, long-term pledges are made by people who have multiple points of contact, of which the article is just one. Additionally, I also suspect that people coming to GWWC and pledging based mostly on an article are going to be lower-quality pledges in terms of the amount that is actually donated. Following these pledges (and all pledges) looking for differences in pledge quality would be very neat, but would probably require 3+ years of data, which would take time.
-
In the interview with Tim Harford, Elie and Niel discussed SCI, and Tim Hartford decided to donate there at the end of the show. After the appearances, SCI contacted us to report that they had received several £1000s of donations as a result of our media. The exact amount SCI received as a result of this media attention was difficult for them to estimate relative to the variable background rate, but they suggested it may have been as much as £10,000.
That’s pretty awesome. How does SCI estimate that? It does seem pretty difficult to me.
-
However, my guess is that the main benefit of media attention is longer-term: leading to further media opportunities in the future; improving my position as a public advocate of effective altruism; increasing the real-world credibility of the effective altruism movement; generating connections with people who we wouldn’t otherwise have met; and, most importantly, learning via practical experience of engaging in the media.
This seems quite plausible to me as well.
-
Moreover, I have a distinct worry that what I call poor quality was a positive factor, in terms of the article’s popularity.
I have this worry as well. But it’s also possible that while higher-quality articles get lower viewership, they may get higher-quality viewership in terms of increased response and conversion rates from the viewers that are attracted.
-
This could be investigated further by, for example, looking at popular writers, and seeing how the quality of their most popular writing compares to the quality of their less popular writing.
That would be cool!
-
It’s very difficult to assess how valuable the article was, and whether it was positive or negative in value.
This is very true. Part of the inherent difficulties of doing cost-effectiveness for outreach; especially media outreach.
-
In the coming months I will experiment by with writing longer and comparatively more rigorous articles on carefully chosen topics.
I look forward to seeing them!
-
Given that the money moved per visitor figure may be as high as $2.78, and may be even higher, it seems worthwhile to experiment with paying for ads to GiveWell’s site, to see if one can achieve a better than 1:1 return on expenditure.
Doesn’t GiveWell already run a lot of Google Adwords? How well do those do?
Thanks for these comments Peter. I think I agree with most of them. To respond specifically to the one I have additional information about:
In the interview with Tim Harford, Elie and Niel discussed SCI, and Tim Hartford decided to donate there at the end of the show. After the appearances, SCI contacted us to report that they had received several £1000s of donations as a result of our media. The exact amount SCI received as a result of this media attention was difficult for them to estimate relative to the variable background rate, but they suggested it may have been as much as £10,000.
That’s pretty awesome. How does SCI estimate that? It does seem pretty difficult to me.
If I understood Alix at SCI correctly the rate of online donations in the few days after the show and associated article was many times higher than usual (perhaps even more than an order of magnitude higher—I can’t remember exactly), and so they were estimating the difference between the increased rate and the background rate. This assumes that the spike and the additional donations were due to the media attention, which may well be a false assumption, but given the immediacy of the spike in donations, the scale of the spike, the prominence of the media attention, and the prominence of SCI in the media attention, I am inclined to think that most of the spike was probably down to the media attention. One other thing to note is that if I understand correctly this figure only includes donations direct to SCI, and does not include any donations made to SCI via GiveWell, who were also featured prominently in the media attention. Nonetheless I agree that it is difficult to estimate exactly how much additional donations went to SCI.
First, I find it hard to believe that $2.78 is really generated via online referral; I imagine some >data integrity problems or sampling problems are leading to a large skew. I expect the >marginal value of an additional unique visit to GiveWell is much less than $2.78.
That sounds plausible to me too. Insofar as I’m ultimately concluding that this sort of media courting isn’t justified by the short-run impact, then using an optimistic figure for impact-per-clickthrough is the conservative assumption. Would still be useful to get into this more if someone wants to run an ‘on-line advertising to GiveWell’ project (which is something I’d love to see come out of .impact or EA Ventures—it’s an idea that’s been floated for quite a while).
Additionally, I also suspect that people coming to GWWC and pledging based mostly on an >article are going to be lower-quality pledges in terms of the amount that is actually donated.
Yes, my anecdotal view from previous media rounds was that people who saw GWWC online and immediately joined were a) giving a lot more anyway; b) giving to what are probably less effective organisations. HOWEVER, there are two big aspects of GWWC-related impact from media that are neglected by focusing on immediate member growth: 1) Media attention reaches out to important people who we wouldn’t otherwise be able to contact. GWWC’s largest single donor influence (who’s donated >$8mn (mainly to a DAF) and who attributes 90% of this to GWWC) was via media influence; our contacts in UK government were through media attention; 2) Member increase is normally fuelled by multiple exposures to GWWC, and the media can be a good first exposure. Because of this, quantitative impact assessment is very hard; hence a lot of refection and attempts to be common-sensey.
I have this worry as well. But it’s also possible that while higher-quality articles get lower >viewership, they may get higher-quality viewership in terms of increased response and >conversion rates from the viewers that are attracted.
Yeah, the ‘shit spreads’ explanation would also make sense of some oddities about who the most most high-profile influencers are—I think that Dawkins’ crazy behaviour online probably contributes to his popularity-as-measured-by-twitter-followers.
Additional reason against controversiality is that it keeps option value—you can always be more controversial later on. However, I’m also starting to notice that it’s very difficult to control this aspect—things that don’t seem controversial to me can get regarded as highly controversial, often through crazy misreading. Example: if you actually read ‘The Skeptical Environmentalist’ by Bjorn Lomborg he says this like ‘obviously climate change is happening and it’s bizarre that people would even question it’ (not an actual quote) yet in the media he was painted as a climate denier.
Doesn’t GiveWell already run a lot of Google Adwords? How well do those do?
They do, because they get them for free. I don’t know how well they do.
Thanks, I look forwards to reading that. I’ve heard that GiveWell have said that they haven’t found AdWords very worthwhile however—does that fit with your impression?
After the appearances, SCI contacted us to report that they had received several £1000s of donations as a result of our media. The exact amount SCI received as a result of this media attention was difficult for them to estimate relative to the variable background rate, but they suggested it may have been as much as £10,000.
I’ve heard that SCI are consistently over-generous in attributing donations to particular organisations’ influence, because they want their support and recommendations—e.g. attributing all donations in one of Australia and the UK to GWWC.
_In which I do my best attempt at supportive skepticism:_
First, thanks very much for publishing this! I really appreciate the transparency and I think the entire EA community can learn from the efforts here.
Second, it’s awesome that we’re aiming to get a higher profile to EA and introduce it to more people. Introducing things through the media does seem to be quite high value to me, and I think we should keep on doing it.
Third, I think the “experimental” approach is right-on and your initial skepticism of the absolute impact of media and differential relative impacts of different media are smart. And it’s definitely true that it’s exceptionally hard to get good data here. Media causality is a really tough challenge. Hence the old quote “I know half of my advertisements are worthless, I just don’t know which half”.
And fourth, wow for getting such a widely shared article!
-
First, I find it hard to believe that $2.78 is really generated via online referral; I imagine some data integrity problems or sampling problems are leading to a large skew. I expect the marginal value of an additional unique visit to GiveWell is much less than $2.78.
Additionally, relevant to writing articles, I expect there to be large differences in the relative values of different kinds of online referrals. For example, it seems quite plausible to me that someone coming to GiveWell from a piece about how awesome GiveWell is versus someone coming to GiveWell from a piece about how much the ice bucket challenge sucks will become donors at much different rates and donate different amounts.
-
That’s awesome—would be great to hear how the follow up goes and if they were truly inspired by the article.
-
I expect successful, long-term pledges are made by people who have multiple points of contact, of which the article is just one. Additionally, I also suspect that people coming to GWWC and pledging based mostly on an article are going to be lower-quality pledges in terms of the amount that is actually donated. Following these pledges (and all pledges) looking for differences in pledge quality would be very neat, but would probably require 3+ years of data, which would take time.
-
That’s pretty awesome. How does SCI estimate that? It does seem pretty difficult to me.
-
This seems quite plausible to me as well.
-
I have this worry as well. But it’s also possible that while higher-quality articles get lower viewership, they may get higher-quality viewership in terms of increased response and conversion rates from the viewers that are attracted.
-
That would be cool!
-
This is very true. Part of the inherent difficulties of doing cost-effectiveness for outreach; especially media outreach.
-
I look forward to seeing them!
-
Doesn’t GiveWell already run a lot of Google Adwords? How well do those do?
Thanks for these comments Peter. I think I agree with most of them. To respond specifically to the one I have additional information about:
If I understood Alix at SCI correctly the rate of online donations in the few days after the show and associated article was many times higher than usual (perhaps even more than an order of magnitude higher—I can’t remember exactly), and so they were estimating the difference between the increased rate and the background rate. This assumes that the spike and the additional donations were due to the media attention, which may well be a false assumption, but given the immediacy of the spike in donations, the scale of the spike, the prominence of the media attention, and the prominence of SCI in the media attention, I am inclined to think that most of the spike was probably down to the media attention. One other thing to note is that if I understand correctly this figure only includes donations direct to SCI, and does not include any donations made to SCI via GiveWell, who were also featured prominently in the media attention. Nonetheless I agree that it is difficult to estimate exactly how much additional donations went to SCI.
Hi Peter, thanks for the great comments!
That sounds plausible to me too. Insofar as I’m ultimately concluding that this sort of media courting isn’t justified by the short-run impact, then using an optimistic figure for impact-per-clickthrough is the conservative assumption. Would still be useful to get into this more if someone wants to run an ‘on-line advertising to GiveWell’ project (which is something I’d love to see come out of .impact or EA Ventures—it’s an idea that’s been floated for quite a while).
Yes, my anecdotal view from previous media rounds was that people who saw GWWC online and immediately joined were a) giving a lot more anyway; b) giving to what are probably less effective organisations. HOWEVER, there are two big aspects of GWWC-related impact from media that are neglected by focusing on immediate member growth: 1) Media attention reaches out to important people who we wouldn’t otherwise be able to contact. GWWC’s largest single donor influence (who’s donated >$8mn (mainly to a DAF) and who attributes 90% of this to GWWC) was via media influence; our contacts in UK government were through media attention; 2) Member increase is normally fuelled by multiple exposures to GWWC, and the media can be a good first exposure. Because of this, quantitative impact assessment is very hard; hence a lot of refection and attempts to be common-sensey.
Yeah, the ‘shit spreads’ explanation would also make sense of some oddities about who the most most high-profile influencers are—I think that Dawkins’ crazy behaviour online probably contributes to his popularity-as-measured-by-twitter-followers.
Additional reason against controversiality is that it keeps option value—you can always be more controversial later on. However, I’m also starting to notice that it’s very difficult to control this aspect—things that don’t seem controversial to me can get regarded as highly controversial, often through crazy misreading. Example: if you actually read ‘The Skeptical Environmentalist’ by Bjorn Lomborg he says this like ‘obviously climate change is happening and it’s bizarre that people would even question it’ (not an actual quote) yet in the media he was painted as a climate denier.
They do, because they get them for free. I don’t know how well they do.
Valuable to find out. Does anyone here know them well enough to ask? Or is it on their site?
In 2013, 287,822 (28%) of GiveWell’s website visitors came from Adwords http://blog.givewell.org/2014/03/19/givewell-annual-review-for-2013-details-on-givewells-money-moved-and-web-traffic/
Thanks, I look forwards to reading that. I’ve heard that GiveWell have said that they haven’t found AdWords very worthwhile however—does that fit with your impression?
I’ve heard that SCI are consistently over-generous in attributing donations to particular organisations’ influence, because they want their support and recommendations—e.g. attributing all donations in one of Australia and the UK to GWWC.