Summary
As part of our ongoing work to study how to best frame EA, we experimentally tested different phrases and sentences that CEA were considering using on effectivealtruism.org.
Doing Good Better taglines
We observed a consistent pattern where taglines that included the phrase ‘do[ing] good better’ received less support from respondents and inspired less interest in learning about EA.
We replicated these results in a second experiment, where we confirmed that taglines referring to “do[ing] good better” performed less well than those referring to “do[ing] the most good”.
Nouns and sentences
Nouns: The effect of using different nouns to refer to EA was small, but referring to EA as a ‘philosophy’ or ‘movement’ inspired the most curiosity compared to options including ‘project’ and ‘research field’.
Sentences: “Find the most effective ways to do good with your time, money, and career” and “Effective altruism asks the question of how we can do the most good through our time, money, and resources,” were rated as the highest in terms of both curiosity and support.
Method
In our first survey, we recruited 1,453 US adults from the Prolific.co platform, weighting responses to match the US general population across sex, age, education, income, race/ethnicity, region, and political party identification. Each participant saw all nouns, taglines, and sentences, but the order of both sections and items was randomised. We also measured respondents’ ‘effectiveness-focus’.
It’s worth noting that these experiments were designed around action-guiding practical tests, rather than seeking general theoretical insights. We tested the particular terms that CEA were actually considering or expressed interest in, to see which performed best, rather than systematically manipulating variables to see what underlying factors was causing different messages to perform better.
Taglines (Study 1)
For taglines, respondents evaluated the extent to which each phrase inspired them to learn more about EA[1] or sounded like something they would support2.
A striking pattern we observed across both measures was that taglines framed in terms of “do[ing] good better” were consistently less effective. It should be noted, however, that this is a pattern we observe ex post. And, as noted above, the items were not systematically selected, so the “doing good better” items are not exact mirrors of the other items, with only that particular phrase manipulated. Nevertheless, the pattern is striking.
1 Imagine seeing one of the following taglines on a website. To what extent does this make you want to learn more about effective altruism? (1) Not at all - (5) Extremely.
2 Imagine seeing one of the following taglines on a website. To what extent does this sound like something you would support? (1) Not at all - (5) Extremely.
Doing Good Better replication (Study 2)
In order to test this effect more systematically, we ran a second survey aiming to replicate these results. We recruited 1006 US adults from the Prolific.co platform. As our interest here was more directly in testing an experimental effect, we did not weigh the results.
In this survey, we tested two pairs of taglines, each designed to mirror the other:
Evidence-based: “An evidence based way to [do the most good / do good better]”
A framework: “A framework for [doing the most good / for doing good better]”
This allowed us to more directly assess the effect of the phrase “do[ing] good better” compared to a control, across two different messages.
The results were strikingly consistent.
We found that respondents reported that the tagline sounded like something they would support to a considerably larger extent when the phrase “do[ing] the most good” was used rather than “do[ing] good better” across both messages.
Similarly, respondents expressed considerably more interest in learning more about EA when the “do[ing] the most good” phrase was used compared to the “do[ing] good better” phrase.
Sentences (Study 1)
When evaluating sentences, respondents again rated the extent to which they inspired learning about EA3 or sounded worthy of their support4. The sentences “Find the most effective ways to do good with your time, money, and career” and “Effective altruism asks the question of how we can do the most good through our time, money, and resources” were rated highest across both measures. Alternatively, the only sentence that did not mention the use of time or money (i.e. “Ways to use your career and donations to maximize your impact”) was ranked the lowest across both questions.
3 Imagine seeing one of the following descriptions of effective altruism on a website. To what extent does this make you want to learn more about effective altruism? (1) Not at all - (5) Extremely.
4 Imagine seeing one of the following descriptions of effective altruism on a website. To what extent does this sound like something you would support? (1) Not at all - (5) Extremely.
Nouns (Study 1)
We investigated whether using certain nouns in the definition of effective altruism was more likely to inspire respondents to learn about EA. Respondents rated different variations of the sentence “Effective altruism is a [noun] dedicated to doing the most good”—with the [noun] replaced by different words (e.g., ‘framework’, ‘philosophy’, ‘practical community’, etc) in each sentence5. Ratings were given on a five-point Likert scale from “Not at all” to “Extremely.” Although differences were small, “philosophy” and “movement” received the highest average ratings.
5 Imagine that you view one of the following sentences on a website about effective altruism. To what extent does this make you want to learn more about effective altruism?
Effectiveness focus
Lastly, there was no significant relationship between respondents’ level of effectiveness focus and their preferences for specific nouns, taglines, or sentences.
Conclusion
The most striking finding from these studies was that the phrase “doing good better” appears to do worse at engaging people’s interest and support, compared to available alternatives. In our replication study, which compared this more directly to the alternatives “doing the most good”, the observed difference was quite considerable, and consistent. We therefore feel relatively confident recommending that using phrases other than “doing good better” could be advisable and that this could make a significant difference to how messages are received.
Although the effects of other differences in framing we found were relatively modest, certain word choices and sentence structures consistently inspired higher curiosity and support. These findings suggest that small linguistic adjustments can influence how EA is perceived and can help maximise interest and support for the movement.
We will continue to explore experiments as our work in this area continues. If there are particular things that people would like tested in this area, please reach out.
Acknowledgments
Authored by Devangana Prasad with contributions from David Moss. Thanks to Agnes Stenlund and CEA for working together with us on these studies.
- ^
We make no strong claims about the specifics of how the book ‘Doing Good Better’ should have been titled and marketed. We think that people should assess particular cases on the basis of all the details relevant to the particular case in question.
😢
(Also, thank you for doing this analysis, it’s great stuff!)
Thanks for this. Post-hoc theorizing:
‘doing good better’ calls to mind comparisons to the reader’s typical ideas about doing good. It implicitly criticizes those examples, which is a negative experience for the reader and could cause defensiveness.
‘Do the most good’ makes the reader attempt to imagine what that could be, which is a positive and interesting question, and doesn’t immediately challenge the reader’s typical ideas about doing good.
It wouldn’t have been obvious to me before the fact whether the above stuff wouldn’t be outweighed by worries about reactions to ‘the most good’ or what have you, so I appreciate you gathering empirical evidence here.
I would be careful with this—it might be an improvement, but are we sure that optimizing short-term messaging success is the right way to promote the ideas as being important long-term conceptual changes to how people approach life and charity?
Lots of other factors matter, and optimizing one dimension, especially using short term approval, implicitly minimizes other important dimensions of the message. Also, as a partial contrast to this point, see “You get about five words.”
Thanks David.
I certainly agree that we should be careful to make sure that we don’t over-optimise short-term appeal at the cost of other things that matter (e.g. long-term engagement, accuracy and fidelity of the message, etc.). I don’t think we’re calling for people to only consider this dimension: we explicitly say that we “think that people should assess particular cases on the basis of all the details relevant to the particular case in question.”
That said, I think that there are many cases where those other dimensions won’t, in fact, be diminished by selecting messages which are more appealing.[1] For example:
In some cases, like this one, we’re selecting within taglines that had already been selected as suitable candidates based on other factors (such as accuracy). We’re then additionally considering data on how people actually respond.
Relatedly, we might have messages available which seem equally good on the other key dimensions which we care about, but which we know to have higher appeal. For example, in this case, I think “the most good you can do” is at least as accurate and likely to encourage long-term engagement as “doing good better”. So, if this phrasing performs better in terms of initially appealing to people, this is a pro tanto consideration in its favour.
In many contexts, we are only considering the question of which ~5 word tagline to include on a website. So the common tradeoff between shorter, more initially appealing messages and longer but higher-fidelity ones may not apply.
In some cases, like short website taglines, most of the effect of the messages may be whether the person continues reading at all (in which case they read the rest of the website and learn a lot more content) or whether they are instantly turned off. We might not expect the short taglines to have a long-term effect on people’s understanding of EA (dominating all the later content they read) themselves.
While initial appeal and long-term engagement could diverge, in many cases, there’s no particular reason to think that they do. This framing suggests a tradeoff between what is merely superficially appealing vs what promotes long-term engagement. But, often, one message might just appeal less to people simpliciter, e.g. because people find it confusing or off-putting, without promoting any longterm benefits.
More generally, I think that we can often think of plausible ways that a message might appear more appealing, but actually be sub-optimal, when taking into account long-term second order effects or divergent effects across different subgroups and so on. But in such cases I think we typically need more investigation of how people respond, not less.
All that said, I certainly think that we should be careful not to over-optimise any single dimension, but instead carefully weigh all the relevant factors.
Though note that these are not arguments that we should assume that other considerations don’t matter. We should still assess each case on its merits and weigh all the considerations directly.
Understood, and reasonable. The problem is that I’m uncomfortable with “the most good” as the goal anyways, as I explained a few years ago; https://forum.effectivealtruism.org/posts/f9NpDx65zY6Qk9ofe/doing-good-best-isn-t-the-ea-ideal
So moving from ‘doing good better’ to ‘do the most good’ seems explicitly worse on dimensions I care about, even if it performs better on approval.
Experientially, I agree: “doing good better” just isn’t as inspiring a message as “the best ways to help others”.
With that said:
Wasn’t the reason for “doing good better” partially about tamping down maximising messages within the EA community, on account of committed EAs sometimes tending to overmaximise at the cost of their health or their other ethics, then ultimately burn out and quit?
Is it not supposed to provide a more sustainable framework, rather than a more initially personally inspiring one?
So I wouldn’t strike it off the whole record just yet.
Yes!
It is, I think, the difference between outreach-oriented messaging and within-community messaging.
I would hope that EA continue to keep a fairly tight lid on the grabbiness of within-community messaging.
Thanks for the interesting post, David.
It would be interesting to know whether the results vary by demographic. If yes, I wonder whether a sample representative of the US general population is the most adequate. At least historically, the people getting involved with EA have been relatively younger, and had a higher income and level of education. CEA may want to be more broadly appealing, but arguably not to the whole US general population.
Thanks Vasco.
I think there are two distinct questions here: the sample and the analysis.
When resources allow, I think it is often better to draw a representative sample of the full population, and then analyse the effect of different traits / the effects within different sub-populations. Even if the people getting involved in EA have tended to be younger, more educated etc., I think there are still reasons to be concerned about the views of the broader population. Of course, if you are interested in a very niche population then this approach will either not be possible or will be very resources inefficient and you might have to sample more narrowly.
When looking for any interactions with demographics for these results, I found no significant demographic interactions, which is not uncommon. I focus here on the replication study in order to have a cleaner manipulation of the “doing good better” effect, though it’s a smaller sample, and we gathered fewer demographics than in the representative sample in the first study.
For age, we see a fairly consistent effect (the trends are fairly linear even if I allow them to be non-linear, fwiw).
For student status, we see no interaction effect and the same main effect.
For sex, we see no effect and a consistent main effect.[1]
If there’s a lot of interest this we could potentially look at education and income in the first survey.
People sometimes ask why we use sex rather than gender in public surveys and it’s usually to match the census so that we can weight.
Thanks for clarifying, David!
Super important to test our assumptions, including on branding, especially since what sounds good to those already in the EA community might be totally different to what convinces “outsiders”. Great work and thanks for writing up a post to get the word out!
Great stuff, forwarded this directly to our marketing team, thanks for sharing!