This is a pretty dramatic tone given the level of evidence backing up the claims:
The post never actually figures out a plausible number for a conversion rate that could be compared to anyone else’s. It’s not actually at all clear to me that our conversion rates are unusually low. Conversion rates for everything are low.
This goes double if you’re asking for relatively extreme actions like EA pitches do, and counting fairly tiny exposures like watching a YouTube video in the denominator. It seems pretty unreasonable to expect a high conversion rate from that.
The article doesn’t present any evidence (just hypothetical anecdotes) that non-conversions become unlikely to convert in the future. What’s the magnitude of this effect? How much of it is actually causal (vs. that person just having a very low likelihood of converting in the first place)?
For instance, in a world where the quality of the message doesn’t matter at all, but half the population has a conversion rate of 1 and the other half has a rate of 0, you’ll see this effect (because the people who don’t convert the first time will have a conversion rate of 0 as well) but optimizing your message doesn’t matter at all.
Indeed! I’d love to be able to use evidence. It’d be incredible to demonstrate the number of impressions we’ve garnered, how many people have taken the first subsequent action, and how many people have eventually converted to the movement.
Unfortunately, the measurement and metrics around EA are quite weak and have not yet been brought together. We do not, for example, have numbers around how many people are EAs (no matter what definitions we use), any understanding of total exposure, etc. My advocating for considerations of conversion rate I believe is very much in line with this feedback, as it would require robust measurement and evaluation.
While it is to me self-evident, I do not believe that we have to believe our conversion rate is low for my argument to be compelling. That we haven’t done structured conversion optimization in and of itself strongly implies that our efforts are likely to be highly suboptimal.
Bullet 3 should have been addressed in my post with strong evidence, and I do believe there is likely to be plenty of it out there (this effect is discussed frequently in psychology and marketing). In the absence of it, the rational argument I find compelling.
Unfortunately, the measurement and metrics around EA are quite weak and have not yet been brought together. We do not, for example, have numbers around how many people are EAs (no matter what definitions we use), any understanding of total exposure, etc.
Membership in most social movements seems leaky and ill-defined. Before it was receiving national news attenton every week, how many Americans were part of the civil rights movement? How many members of there are the Black Lives Matter movement today? Exactly how many vegans and/or vegetarians are there, and what proportion of them count as animal liberationists, rather than just individuals who changed their own diets and nothing more? What are the absolute numbers and percentaged breakdowns of progressives, conservatives, and libertarians in different politcal movements?
These aren’t totally rhetorical questions. I assume historians and social scientists have numbers for each of these questions varying in the degrees of their precision. However, I expect thy’d be rougher estimates than we’d be hoping for. Also, the numbers for this communities, I expect, were only able to be collected once the community had reached a sufficient size someone from outside bothered asking the question, and was able to find and notice where everyone was. Effective altruism is more unique among social movements in that it’s interested in gauging hard metrics of and for itself from the beginning. Unfortunately, this doesn’t mean we’re going to figure out how ot get them as soon as we want them.
I’m not claiming it’s an undoable task. I just think it’s more difficult than we’d like to admit. Effective altruism sometimes generates metrics which didn’t exist before, so we’ll not stop trying now. I think it’s still best we use metrics like money moved not to infer the number of persons invovled in effective altruism, but to estimate its influence. Even if most of the new money was coming from billionaires, it might be much harder to influence a billionaire to donate money to a cause than the typical person, because the bilionaire could be conscious we’re hoping they donate much more money than who we ususally approach.
Another metric, which is informative but is rougher, is the amount of awareness and media coverage on the issue. Obviously, there has been a great uptick in the coverage of A.I. risk in the last year. In the wake of the Syrian refugee crisis, I’ve seen more friends on Facebook sharing articles about the issue of open borders, from publications such as Vox, The Atlantic, and The Guardian. Of course, these articles I noticed were being shared by fellow effective altruists, so of course I have an inflated impression of how much real coverage the issue of open borders is receiving. However, one year ago, I doubt there would have been articles on the topic of open borders from major publications in the first place, with advocates being limited to posts from “openborders.info″, or their own blogs.
A more objective measure of increased coverage or awareness could be to check the number of Google searches, items with searches, and hits for webpages, e.g., including the phrase “open borders”. Any question of this would need to figure out the amount of positive, neutral, and negative news coverage of the issue in the total amount of coverage. That seems like it too would be a more subjective way of measuring impact, but I expect it would also be valuable.
Unfortunately, the measurement and metrics around EA are quite weak and have not yet been brought together. We do not, for example, have numbers around how many people are EAs (no matter what definitions we use), any understanding of total exposure, etc.
We could improve this by getting as many people as possible to take at least a first page of questions or two on the annual EA census. Among other things that helps establish a lower bound number of people who are EA’s by some definition and their basic characteristics. I remember that around the time the last one was done there were estimates that the total number of EAs was in the low 1000′s. I have no sense of how its grown since then but I have no sense that its exploded so you seem right to that extent.
This is a pretty dramatic tone given the level of evidence backing up the claims:
The post never actually figures out a plausible number for a conversion rate that could be compared to anyone else’s. It’s not actually at all clear to me that our conversion rates are unusually low. Conversion rates for everything are low.
This goes double if you’re asking for relatively extreme actions like EA pitches do, and counting fairly tiny exposures like watching a YouTube video in the denominator. It seems pretty unreasonable to expect a high conversion rate from that.
The article doesn’t present any evidence (just hypothetical anecdotes) that non-conversions become unlikely to convert in the future. What’s the magnitude of this effect? How much of it is actually causal (vs. that person just having a very low likelihood of converting in the first place)?
For instance, in a world where the quality of the message doesn’t matter at all, but half the population has a conversion rate of 1 and the other half has a rate of 0, you’ll see this effect (because the people who don’t convert the first time will have a conversion rate of 0 as well) but optimizing your message doesn’t matter at all.
Indeed! I’d love to be able to use evidence. It’d be incredible to demonstrate the number of impressions we’ve garnered, how many people have taken the first subsequent action, and how many people have eventually converted to the movement.
Unfortunately, the measurement and metrics around EA are quite weak and have not yet been brought together. We do not, for example, have numbers around how many people are EAs (no matter what definitions we use), any understanding of total exposure, etc. My advocating for considerations of conversion rate I believe is very much in line with this feedback, as it would require robust measurement and evaluation.
While it is to me self-evident, I do not believe that we have to believe our conversion rate is low for my argument to be compelling. That we haven’t done structured conversion optimization in and of itself strongly implies that our efforts are likely to be highly suboptimal.
Bullet 3 should have been addressed in my post with strong evidence, and I do believe there is likely to be plenty of it out there (this effect is discussed frequently in psychology and marketing). In the absence of it, the rational argument I find compelling.
Membership in most social movements seems leaky and ill-defined. Before it was receiving national news attenton every week, how many Americans were part of the civil rights movement? How many members of there are the Black Lives Matter movement today? Exactly how many vegans and/or vegetarians are there, and what proportion of them count as animal liberationists, rather than just individuals who changed their own diets and nothing more? What are the absolute numbers and percentaged breakdowns of progressives, conservatives, and libertarians in different politcal movements?
These aren’t totally rhetorical questions. I assume historians and social scientists have numbers for each of these questions varying in the degrees of their precision. However, I expect thy’d be rougher estimates than we’d be hoping for. Also, the numbers for this communities, I expect, were only able to be collected once the community had reached a sufficient size someone from outside bothered asking the question, and was able to find and notice where everyone was. Effective altruism is more unique among social movements in that it’s interested in gauging hard metrics of and for itself from the beginning. Unfortunately, this doesn’t mean we’re going to figure out how ot get them as soon as we want them.
I’m not claiming it’s an undoable task. I just think it’s more difficult than we’d like to admit. Effective altruism sometimes generates metrics which didn’t exist before, so we’ll not stop trying now. I think it’s still best we use metrics like money moved not to infer the number of persons invovled in effective altruism, but to estimate its influence. Even if most of the new money was coming from billionaires, it might be much harder to influence a billionaire to donate money to a cause than the typical person, because the bilionaire could be conscious we’re hoping they donate much more money than who we ususally approach.
Another metric, which is informative but is rougher, is the amount of awareness and media coverage on the issue. Obviously, there has been a great uptick in the coverage of A.I. risk in the last year. In the wake of the Syrian refugee crisis, I’ve seen more friends on Facebook sharing articles about the issue of open borders, from publications such as Vox, The Atlantic, and The Guardian. Of course, these articles I noticed were being shared by fellow effective altruists, so of course I have an inflated impression of how much real coverage the issue of open borders is receiving. However, one year ago, I doubt there would have been articles on the topic of open borders from major publications in the first place, with advocates being limited to posts from “openborders.info″, or their own blogs.
A more objective measure of increased coverage or awareness could be to check the number of Google searches, items with searches, and hits for webpages, e.g., including the phrase “open borders”. Any question of this would need to figure out the amount of positive, neutral, and negative news coverage of the issue in the total amount of coverage. That seems like it too would be a more subjective way of measuring impact, but I expect it would also be valuable.
We could improve this by getting as many people as possible to take at least a first page of questions or two on the annual EA census. Among other things that helps establish a lower bound number of people who are EA’s by some definition and their basic characteristics. I remember that around the time the last one was done there were estimates that the total number of EAs was in the low 1000′s. I have no sense of how its grown since then but I have no sense that its exploded so you seem right to that extent.