One thing that comes to mind, (though perhaps a bit strange), is to really consider Effective Altruism under a similar lens as you would a SaaS product or similar. In the SaaS (software as a service) industry, there are a fair bit of best practices around understanding retention rates, churn, and doing cohort analysis and the like. There’s also literature in evaluating the quality of a product on NPS score and better. It could be neat to have people rank “Effective Altruism” and “The EA Community” on NPS scores.
Likewise, it could be interesting to survey people with things like, “How would you rate the value you are getting from the EA ecosystem”, and then work to maximize this value. Consider the costs (donations, career changes) vs. the benefits and see if you can model total value better.
Hey Ozzie, that makes sense. I think the last EA survey did some things pretty similar to this, inc. asking about value adds & issues, and something similar to the NPS score, as well as why people don’t recommend it.
asked an ‘NPS’ question about EA, asked for qualitative information about positive/negative experiences of the community and examined predictors
asked about barriers to becoming more involved in EA
asked about reasons for people’s interest in EA declining or increasing
asked about what factors were important for retaining people in EA
asked about why people who the respondent knew dropped out
I’m pretty sceptical about the utility of Net Promoter Score in the classical sense for EA. I don’t think there’s any good evidence for the prescribed way of calculating Net Promoter Score (ignoring respondents who answer in the upper-middle of the scale, and then subtracting the proportion of people who selected one of the bottom 7 response levels from the proportion who selected one of the top two response items). And, as I mentioned in our original post, its validity and predictive power has been questioned. Furthermore, one of the most common uses is comparing the NPS score of an entity to an industry benchmark (e.g. the average scores for other companies in the same industry), but it’s very unclear what reference class would be relevant for EA, the community, as a whole, so it’s fundamentally not clear whether EA’s NPS score is good or bad. In the specific case of EA, I also suspect that the question of how excited one would be to recommend EA to a suitable friend may well be picking up on attitudes other than satisfaction with EA, i.e. literally how people would feel about recommending EA to someone. This might explain why the people with the highest ‘NPS’ scores (we just treated the measure as a straightforward ordinal varlable in our own analyses) were people who had just joined EA, and fairly reliably became lower over time.
Nice work with this!
One thing that comes to mind, (though perhaps a bit strange), is to really consider Effective Altruism under a similar lens as you would a SaaS product or similar. In the SaaS (software as a service) industry, there are a fair bit of best practices around understanding retention rates, churn, and doing cohort analysis and the like. There’s also literature in evaluating the quality of a product on NPS score and better. It could be neat to have people rank “Effective Altruism” and “The EA Community” on NPS scores.
Likewise, it could be interesting to survey people with things like, “How would you rate the value you are getting from the EA ecosystem”, and then work to maximize this value. Consider the costs (donations, career changes) vs. the benefits and see if you can model total value better.
Hey Ozzie, that makes sense. I think the last EA survey did some things pretty similar to this, inc. asking about value adds & issues, and something similar to the NPS score, as well as why people don’t recommend it.
Yeh much of this is in our Community Information post where we:
asked an ‘NPS’ question about EA, asked for qualitative information about positive/negative experiences of the community and examined predictors
asked about barriers to becoming more involved in EA
asked about reasons for people’s interest in EA declining or increasing
asked about what factors were important for retaining people in EA
asked about why people who the respondent knew dropped out
I’m pretty sceptical about the utility of Net Promoter Score in the classical sense for EA. I don’t think there’s any good evidence for the prescribed way of calculating Net Promoter Score (ignoring respondents who answer in the upper-middle of the scale, and then subtracting the proportion of people who selected one of the bottom 7 response levels from the proportion who selected one of the top two response items). And, as I mentioned in our original post, its validity and predictive power has been questioned. Furthermore, one of the most common uses is comparing the NPS score of an entity to an industry benchmark (e.g. the average scores for other companies in the same industry), but it’s very unclear what reference class would be relevant for EA, the community, as a whole, so it’s fundamentally not clear whether EA’s NPS score is good or bad. In the specific case of EA, I also suspect that the question of how excited one would be to recommend EA to a suitable friend may well be picking up on attitudes other than satisfaction with EA, i.e. literally how people would feel about recommending EA to someone. This might explain why the people with the highest ‘NPS’ scores (we just treated the measure as a straightforward ordinal varlable in our own analyses) were people who had just joined EA, and fairly reliably became lower over time.