An Effective Altruist Message Test

I de­cided to run an Effec­tive Altru­ist mes­sage on a full pop­u­la­tion sur­vey I have ac­cess to, use bayesian mes­sage test­ing soft­ware to an­a­lyze the re­sults, and share the re­sults with the EA com­mu­nity on the fo­rum.

I tested sev­eral EA themed mes­sages aimed at in­creas­ing re­spon­dents’ in­ter­est in donat­ing and effec­tive al­tru­ism. For non-con­trol re­spon­dents, I pre­sented them with ei­ther the claim that AMF can save a life for 3,500 dol­lars (Facts), a short ver­sion of Peter Singer’s Pond anal­ogy (Obli­ga­tion) or a short take on Will MacAskill’s op­por­tu­nity fram­ing (Op­por­tu­nity).

I then asked re­spon­dents how much they planned to donate in the next 12 months, their in­ter­est in EA and gave them the op­por­tu­nity to click on a link to donate to the Against Malaria Foun­da­tion (AMF) or po­ten­tially join Giv­ing What We Can (GWWC). I recorded whether they clicked on these links. Ad­di­tion­ally, I asked how much they donated in the last 12 months as a “pre­screen” to con­trol for in my anal­y­sis, dra­mat­i­cally in­creas­ing my pre­ci­sion. The word­ing of each of the ques­tions de­scribed above is in a doc linked to at the bot­tom of this ar­ti­cle.

Over­all, my re­sults in­di­cate that these brief mes­sages do not in­crease re­spon­dents per­spec­tive dona­tions, but there is some ev­i­dence that the facts and obli­ga­tion mes­sage may in­crease in­ter­est in EA among ed­u­cated in­di­vi­d­u­als. Below, I dis­cuss my re­sults in more de­tail.

Methods

This was em­bed­ded in a 1200 per­son on­line sur­vey rep­re­sen­ta­tive of US cit­i­zens. Within this sur­vey, I de­liv­ered 3 treat­ments (Facts, Obli­ga­tion and Op­por­tu­nity) each with a sam­ple size of ~200 re­spon­dents. I com­pared how in­ter­ested re­spon­dents were in learn­ing more about effec­tive al­tru­ism and how much they planned to donate. I ex­am­ined these re­la­tion­ships over­all and among the crit­i­cal sub­group of those with at least a bach­e­lor’s de­gree (this was my only pre-planned com­par­i­son; hav­ing a bach­e­lor’s de­gree serves as a rough proxy for the EA tar­get elite au­di­ence). Un­for­tu­nately, only 10 re­spon­dents took sub­stan­tive ac­tion by click­ing on the AMF dona­tion or GWWC links, so this sam­ple size was not suffi­cient to con­duct ro­bust anal­y­sis on this de­pen­dent vari­able.

To an­a­lyze this re­la­tion­ship, I used Bayesian hi­er­ar­chi­cal mod­el­ing soft­ware based in R and Stan. This mod­el­ing ap­proach al­lows us to cre­ate a prob­a­bil­ity dis­tri­bu­tion of the treat­ment effects, re­duce var­i­ance by con­trol­ling for other vari­ables in the sur­vey and bor­row power from the full sam­ple when es­ti­mat­ing effects among sub­groups.

Fu­ture Dona­tion Plans

To gauge how the EA mes­sages changed peo­ple’s dona­tions, I pre­dicted the prob­a­bil­ity that re­spon­dents plan to donate at least 6% of their in­come to char­ity. Below are treat­ment effects and prob­a­bil­ity dis­tri­bu­tions for the effect of each mes­sage. Er­ror bars are one stan­dard er­ror.

Screen Shot 2017-03-29 at 9.04.26 PM.png

Screen Shot 2017-03-29 at 9.03.55 PM.png

As ev­i­denced by the plots above, none of the treat­ments had a large im­pact. In fact, the im­pact such as it is is nega­tive. The im­pact is minute, with Aver­age Treat­ment Effects (ATEs) well un­der 1%. Even in the most op­ti­mistic case, few are likely to in­crease their ex­pected dona­tion be­hav­iors in re­sponse to the ar­gu­ments pro­vided. This is not sur­pris­ing; it can take a lot for peo­ple to change sub­stan­tial be­hav­iors, or even sug­gest they may change them in a sur­vey. There’s lit­tle ev­i­dence that any ar­gu­ment is more effec­tive, and will re­quire other means to as­sess.

In­ter­est in Effec­tive Altruism

I also ex­am­ined in­ter­est in Effec­tive Altru­ism (whether re­spon­dents are at least some­what in­ter­ested in learn­ing more about EA). Below are treat­ment effects and prob­a­bil­ity dis­tri­bu­tions for the effect of each mes­sage.

Screen Shot 2017-03-29 at 9.10.34 PM.png

Screen Shot 2017-03-29 at 9.09.51 PM.png

Un­like per­spec­tive dona­tions, in­ter­est ex­hibits sub­stan­tial vari­a­tion be­tween treat­ments. While the op­por­tu­nity mes­sage ap­pears to ac­tively put peo­ple off, let­ting peo­ple know they can save lives for $3,500 does ap­pear to get folks in­ter­ested in effec­tive al­tru­ism, and may be a good way to get them in the door. How­ever, the most in­ter­est­ing ATEs ap­pear when break­ing the re­sults out by ed­u­ca­tion level. Below are ATEs, prob­a­bil­ities that each mes­sage is best, and prob­a­bil­ities of back­lash (a nega­tive effect) for each mes­sage among those with at least a bach­e­lor’s de­gree.

Screen Shot 2017-03-29 at 9.12.33 PM.png

There’s a large gap in the im­pact these mes­sages have on those with and with­out a bach­e­lor’s de­gree. For those with a bach­e­lor’s de­gree, ev­ery mes­sage has a pos­i­tive ATE, and the effects of obli­ga­tion and facts are quite sub­stan­tial (they in­crease the chance in­di­vi­d­u­als are at least some­what in­ter­ested in learn­ing more about EA by over 6 per­centage points). How­ever, there’s a sub­stan­tial amount of var­i­ance in these effect sizes. The true effect of these in­ter­ven­tions could cred­ibly range from slightly nega­tive to over 15 per­centage points, but most of the prob­a­bil­ity mass lies in the mid­dle of this range. Over­all, Singer’s pond anal­ogy and the scope of im­pact peo­ple can have with their dona­tions ap­pear to be effec­tive meth­ods of build­ing in­ter­est in EA; there’s around a 90% chance that one of these is the best mes­sage (out of the three and the con­trol) that you can use with in­di­vi­d­u­als hav­ing at least a bach­e­lor’s de­gree.

How­ever, the re­sults are much differ­ent for those with­out a bach­e­lor’s de­gree. Only the fac­tual ar­gu­ment ap­pears to have any pos­i­tive effect, and they ap­pear to be turned off by the op­por­tu­nity and obli­ga­tion fram­ings. This con­firms what we already sus­pected, prospec­tive EAs are likely to be ed­u­cated elites and it makes sense to tar­get them.

Dis­cus­sion and caveats

Over­all, these find­ings are mixed, but not sur­pris­ing. EA mes­sag­ing does work to get peo­ple in­ter­ested in Effec­tive Altru­ism. How­ever, EA mes­sag­ing alone is not enough to get peo­ple to even claim that they will in­crease their dona­tions. This likely takes a much more sub­stan­tial treat­ment. But get­ting ed­u­cated elites in­ter­ested in Effec­tive Altru­ism is the first step. By em­pha­siz­ing the moral rea­son­ing be­hind effec­tive al­tru­ism and the scale of good we can do in the world, we can en­courage peo­ple to learn more about Effec­tive Altru­ism. From there, we can change their be­hav­ior.

Like all re­search, this is limited. For one, the in­di­vi­d­u­als tar­geted and the con­text are not en­tirely typ­i­cal. EA mes­sag­ing will tend to come from friends and ac­quain­tances in per­son or in dis­cus­sions on­line rather than as an anony­mous mes­sage in a web sur­vey. Peo­ple may re­act differ­ently in other situ­a­tions, but this study does provide an im­por­tant piece of ex­per­i­men­tal ev­i­dence that can in­form how we try to en­gage peo­ple. Ad­di­tion­ally, hav­ing a bach­e­lor’s de­gree is not enough to be in EA’s core au­di­ence. EAs as a whole tend to be an­a­lyt­i­cally ori­ented and are of­ten math­e­mat­i­cal in their think­ing. This pop­u­la­tion is not as re­stric­tive as typ­i­cal per­spec­tive EAs. The good news is that both of these differ­ences sug­gest the true effect size may be larger. A more per­sonal con­tact to an even bet­ter tar­get may be very effec­tive at en­courag­ing peo­ple to join EA. In­deed, that may help ex­plain EA’s sub­stan­tial growth. How­ever, in an­other way, the au­di­ence is overly re­stric­tive; only US cit­i­zens were in­cluded. Differ­ent mes­sages may be more effec­tive in other coun­tries.

This study is a step for­ward, it pro­vides some ev­i­dence on what treat­ments work best and what we can ac­com­plish with a con­tact. As always, more re­search (both ob­ser­va­tional and ex­per­i­men­tal) is needed. As our com­mu­nity en­gages in more tri­als to test our mes­sag­ing, we can con­tinue to fine tune it and ex­pand the ap­peal of EA.

Thanks to Kerry Vaughan for ad­vice on mes­sage choice.

Ques­tion Wording