To Grow, Hit the Brakes (Why Conversion Rate Matters So Much and Means EA is Not Doing Anywhere Close to as Well as We Think)
Have you ever seen an ad that was just so clearly not meant for you? Perhaps you’re a 20-something single male, and an ad for a Cadillac minivan, touting how many kids it could fit, came on the air. Or maybe you’re a vegetarian, health-focused mother and as you read a magazine you saw an ad showing just how much meat there is in Wendy’s burgers, since they ‘don’t cut corners.’
Think back to how you felt after seeing that ad. Think about how you felt about the company being advertised. It’s likely that you didn’t have much of an opinion about them before, or at least didn’t spend much time thinking about them or forming your opinion… we usually don’t.
But afterward?
Chances are after seeing the ad, you felt worse about the company. Maybe if you’d ever been in the market for a luxury car (ok, we’re EAs, bad example, but whatever) you’d consider a sleek Cadillac, but then that ad would automatically come to mind, and you’d think maybe you ought to go for a luxury brand that doesn’t also make a minivan. Perhaps if you were the health-focused, vegetarian mother, and your kids had said they wanted a Wendy’s McFlurry, you would have occasionally acquiesced as a treat. Now though, there’s an increased likelihood of you having a recoil reaction at the sound of Wendy’s, and recommending Dairy Queen instead.
These examples are meant to demonstrate a single point, the awareness of which seems to be all too often lacking in the EA community: People exposed to a pitch, but not interested or compelled by it, are most often subsequently turned off.
As you’d expect, someone who’s turned off is much more difficult to convince than someone who’s never heard of you. They have developed the heuristic of your company/idea = not interested, so they most often don’t take the time/attention to reevaluate it when further exposed.
For this reason, well-strategized new ventures, products, pitches, ideas and more follow the game plan of avoiding too significant of attention until they’ve put significant, well-strategized effort behind maximizing their conversion rate.
Conversion rate is simple. It is the number of people who take your desired action divided by the total number of people exposed to your message.
The highly simplified, informed approach to growing a company, spreading an idea, or whatever is:
1. Gather and keep gathering enough attention so that you have meaningful sample sizes.
2. Test your idea/product/message, measure the results. Make a change to try to improve, measure if it works, repeat. Go through tons of these iterations. Continue until anything major you think of doesn’t seem to affect your conversion rate in a highly significant positive way.
3. Then, and only then, get mass attention. Throw fuel on that fire. Run the conversion machine to spit out people taking the actions you want! (Note: You’ve exhausted all the big changes you can think of, within reason, but you continue to do step 2 with less major differences).
I claim, that unfortunately, EA has done little to maximize conversion rates, and much to grab attention. We went to step 3 without nearly enough time on step 2.
Think about the enormous amount of press, YouTube views, book sales, etc. that we’ve had. Then think about the low numbers of those who have taken the Giving What We Can Pledge, joined the Facebook group, donated to a top charity, or applied to attend EA Global.
Think of all the VIPs whom we’ve emailed, who have read our books, attended meetings with us, been to our events, or who are within our personal networks that aren’t donating their fortune, or in most instances even one cent. Think of all of the friends within our networks, and think about how often we fail to grab their attention. Think of the ridiculous high level of conformity within the core of the movement, and our continued failure to attract those who would truly represent diversity.
Every exposure that isn’t a win is indeed a loss.
I believe EA organizations and individuals should immediately & drastically stop being so attention seeking, realize the danger of what we’re doing, and focus much more on being experimental, measuring our results, and optimizing for conversion in every single form of outreach that we do. Only once we’re measuring and unable to improve on our conversion rates should we go back to our awareness-building, attention-seeking strategies.
This is a pretty dramatic tone given the level of evidence backing up the claims:
The post never actually figures out a plausible number for a conversion rate that could be compared to anyone else’s. It’s not actually at all clear to me that our conversion rates are unusually low. Conversion rates for everything are low.
This goes double if you’re asking for relatively extreme actions like EA pitches do, and counting fairly tiny exposures like watching a YouTube video in the denominator. It seems pretty unreasonable to expect a high conversion rate from that.
The article doesn’t present any evidence (just hypothetical anecdotes) that non-conversions become unlikely to convert in the future. What’s the magnitude of this effect? How much of it is actually causal (vs. that person just having a very low likelihood of converting in the first place)?
For instance, in a world where the quality of the message doesn’t matter at all, but half the population has a conversion rate of 1 and the other half has a rate of 0, you’ll see this effect (because the people who don’t convert the first time will have a conversion rate of 0 as well) but optimizing your message doesn’t matter at all.
Indeed! I’d love to be able to use evidence. It’d be incredible to demonstrate the number of impressions we’ve garnered, how many people have taken the first subsequent action, and how many people have eventually converted to the movement.
Unfortunately, the measurement and metrics around EA are quite weak and have not yet been brought together. We do not, for example, have numbers around how many people are EAs (no matter what definitions we use), any understanding of total exposure, etc. My advocating for considerations of conversion rate I believe is very much in line with this feedback, as it would require robust measurement and evaluation.
While it is to me self-evident, I do not believe that we have to believe our conversion rate is low for my argument to be compelling. That we haven’t done structured conversion optimization in and of itself strongly implies that our efforts are likely to be highly suboptimal.
Bullet 3 should have been addressed in my post with strong evidence, and I do believe there is likely to be plenty of it out there (this effect is discussed frequently in psychology and marketing). In the absence of it, the rational argument I find compelling.
Membership in most social movements seems leaky and ill-defined. Before it was receiving national news attenton every week, how many Americans were part of the civil rights movement? How many members of there are the Black Lives Matter movement today? Exactly how many vegans and/or vegetarians are there, and what proportion of them count as animal liberationists, rather than just individuals who changed their own diets and nothing more? What are the absolute numbers and percentaged breakdowns of progressives, conservatives, and libertarians in different politcal movements?
These aren’t totally rhetorical questions. I assume historians and social scientists have numbers for each of these questions varying in the degrees of their precision. However, I expect thy’d be rougher estimates than we’d be hoping for. Also, the numbers for this communities, I expect, were only able to be collected once the community had reached a sufficient size someone from outside bothered asking the question, and was able to find and notice where everyone was. Effective altruism is more unique among social movements in that it’s interested in gauging hard metrics of and for itself from the beginning. Unfortunately, this doesn’t mean we’re going to figure out how ot get them as soon as we want them.
I’m not claiming it’s an undoable task. I just think it’s more difficult than we’d like to admit. Effective altruism sometimes generates metrics which didn’t exist before, so we’ll not stop trying now. I think it’s still best we use metrics like money moved not to infer the number of persons invovled in effective altruism, but to estimate its influence. Even if most of the new money was coming from billionaires, it might be much harder to influence a billionaire to donate money to a cause than the typical person, because the bilionaire could be conscious we’re hoping they donate much more money than who we ususally approach.
Another metric, which is informative but is rougher, is the amount of awareness and media coverage on the issue. Obviously, there has been a great uptick in the coverage of A.I. risk in the last year. In the wake of the Syrian refugee crisis, I’ve seen more friends on Facebook sharing articles about the issue of open borders, from publications such as Vox, The Atlantic, and The Guardian. Of course, these articles I noticed were being shared by fellow effective altruists, so of course I have an inflated impression of how much real coverage the issue of open borders is receiving. However, one year ago, I doubt there would have been articles on the topic of open borders from major publications in the first place, with advocates being limited to posts from “openborders.info″, or their own blogs.
A more objective measure of increased coverage or awareness could be to check the number of Google searches, items with searches, and hits for webpages, e.g., including the phrase “open borders”. Any question of this would need to figure out the amount of positive, neutral, and negative news coverage of the issue in the total amount of coverage. That seems like it too would be a more subjective way of measuring impact, but I expect it would also be valuable.
We could improve this by getting as many people as possible to take at least a first page of questions or two on the annual EA census. Among other things that helps establish a lower bound number of people who are EA’s by some definition and their basic characteristics. I remember that around the time the last one was done there were estimates that the total number of EAs was in the low 1000′s. I have no sense of how its grown since then but I have no sense that its exploded so you seem right to that extent.
There must be anecdotes that go the other way as well—For example our group recently had a friend of a member attend a meeting. This person was not sold on EA when he first heard the idea, but over the year as he saw more things on Facebook, heard our member talking about his EA views etc he decided he wanted to know more. More personally my Aunty was first exposed to EA simply through my Facebook posts. The posts weren’t pitched to her more heart felt nature but this didn’t mean she was permanently turned off. Eventually after a year of me posting about my pledge10 donations we had a conversation about why I was doing what I was doing, and now a year later her business donates 10% of profits. Furthermore—I think that the enormous amount of press that has occurred—Peter Singer, Will MacAskill etc has all been pretty broad EA ideas, ideas that are not too pushy, therefore not likely to turn people off unless they are people that we are unlikely to reach anyway, even with a pitch directly tailored to them. There was recently an article posted about how measuring impact isn’t always the most efficient thing to do, and considering how difficult it would be to measure the results of EA outreach it seems to me that to put outreach on hold until we have perfected it seems like it could just be delaying what could potential be huge impact.
I agree that there is a failure to reach a broader audience, but I disagree that every exposure that isn’t a win is a loss. Its just not a win yet. Or it may never be but not because they were ‘put off’ EA but simply because we can’t expect every brain in the world to work the way ours do.
http://ssir.org/articles/entry/measuring_impact_isnt_for_everyone?utm_content=buffer199b8&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer
As Stefan points out, this is hard to do in a broad decentralised movement. However if anyone is interested in this kind of approach and experimentation, whether interested in the results or in helping with the experiments themselves, we are planning on doing roughly this for the EA London community. Feel free to get in contact.
What’s a ‘meaningful sample size’ in practice? I would have thought that we are only just now reaching a scale where we might have decent sizes for some things, in which case we are still in step 1 where seeking attention is still correct.
Separately, I didn’t find the anecdote at all compelling. I can vaguely remember such mis-targeted adverts, but I can’t recall details nor can I remember the companies in question. That’s sort of what I would expect; if some non-intrusive advertising isn’t ‘for you’, you ignore it and forget about it. You can’t form an impression of that which you don’t recall.
In his talk at the 2015 Effective Altruism Global conference, Holden Karnofsky of Givewell and the Open Philanthropy Project (Open Phil) said he is optimistic in particular about the ability of Open Phil to attract and influence the donations of more very high net-worth individuals, especially the ones similar to Dustin Moskovitiz (co-founder of Good Ventures, and Facebook) in Silicon Valley succeeding in startups, who Mr. Karnofsky believes take more consideration of the effectiveness of their donations. One example he cited is how earlier this year Alex Trigger and Mike Kriger, a founder of Instagram, have agreed to make grants from their foundation based on the recommendations of Open Phil alongside Good Ventures.
Also, how much information do you really have about how much the highest net-worth individuals effective altruism is in touch with are considering recommendations from effective altruism? Even if they haven’t donated one cent, aren’t they more likely to make a decision to donate five, six, or seven figure sums, rather than just donating $10 to AMF and saying “okay, thanks, EA, I’ve learned!”
Effective altruism is a social movement, and while much of it is centralized in formal organizations, more so than other social movements I’m aware of, that doesn’t mean it doesn’t have a big grassroots component to it as well. Individuals telling their friends and others about effective altruism is a lot different than a marketing campaign, and I’m not sure we can expect the same methods to effect optimal conversion rates for both. It’s more difficult to measure what works through word of mouth. Also, I personally think it might be too demanding to request effective altruists run experiments or something in their personal lives to figure out what works best in spreading effective altruism.
Can you please specificy what you mean by the “core of the movement”, and on what dimension in this case a lack of diversity means, and how it’s most problematic?
I don’t think that evidence from pitches of luxury cars necessarily is very relevant for us. Experiences from other social and political movements seem much more relevant. As far as I know most if not all of them used the tactics of getting as much attention as possible. Of course, that doesn’t necessarily show that they used the right tactics, but on the reasonable hypothesis that they are on average better than chance at choosing good tactics, it does provide some evidence that attention-seeking is a good tactics.
I don’t have any rigorous evidence on this matter, but my personal experience is certainly not that people who hear about EA somehow turn against it. I would need much more compelling evidence than is presented in this post to stop believing that.
Yes, we do need to collect more information about what works, but we should definitely not “hit the brakes”. Instead, we need to double our efforts to reach out. E.g., in my country, Sweden, very few have even heard of EA still, but when they do, most of them like it.
That’s interesting—what’s your best guess or evidence as to how many change their actions as a result, and in what ways they do so?
We’re at a too early stage for evaluating that. I’m just saying there seems to be something in between a “win” in the sense of immediate conversion, and a “loss” in the sense that the person is now less likely to join the EA movement. Therefore, I find these words unwarranted:
Josh continues:
I also think this strategy is hard to get to work from an organizational point of view. Today, people are working enthusiastically to grow the movement in a highly decentralized way. This strategy would need much more centralized planning, and it’s not clear to me how we would pull that off. Also, there is a risk that the grass-roots would lose a bit of their entusiasm if they had to co-ordinate all of their outreach activities with someone measuring their activities.