Thanks for the post! This definitely isn’t addressed at you specifically (I think this applies to all EA groups and orgs), so I hope this doesn’t seem like unfairly singling you out over a very small part of your post, but I think EAs should stop calculating and reporting the ‘NPS score’ when they ask NPS or NPS-style questions.
I assume you calculated the NPS score in the ‘standard’ way i.e. asking people “Would you recommend the Fellowship to a friend?” on a 0-10 or 1-10 scale, and subtracting the percentage of people who answered with a 6 or lower (“Detractors”) from the percentage of people who answered with a 9 or 10 “Promoters”). The claim behind the NPS system is that people who give responses within these ranges are qualitatively different ‘clusters’ (and also the people responding with a 7-8 are also a distinct cluster “Passives” who basically don’t matter and so who don’t figure in the NPS scores at all) and that just subtracting the percentages of one cluster from another is the “easiest-to-understand, most effective summary of how a company [is] performing in this context.”
Unfortunately, it does not seem to me that there’s a sound empirical basis for analysing an NPS style scale in this way (and the company behind it are quite untransparent about this basis (see discussion here). This way of analysing responses to a scale is pretty unusual and obscures most of the information about the distribution of responses, which it seems like it would be pretty easy for an EA audience to understand. For example, it seems like it would be pretty easy to depict the distribution of responses, as we did in the EA Survey Community information post.
And it seems like calculating the mean and median response would also give a more informative, but equally easy to understand summary of performance on this measure (more so than the NPS score, which for example, completely ignores whether people respond with a 0 or a 6). This would also allow easy significance testing of the differences between events/groups.
Thanks for the post! This definitely isn’t addressed at you specifically (I think this applies to all EA groups and orgs), so I hope this doesn’t seem like unfairly singling you out over a very small part of your post, but I think EAs should stop calculating and reporting the ‘NPS score’ when they ask NPS or NPS-style questions.
I assume you calculated the NPS score in the ‘standard’ way i.e. asking people “Would you recommend the Fellowship to a friend?” on a 0-10 or 1-10 scale, and subtracting the percentage of people who answered with a 6 or lower (“Detractors”) from the percentage of people who answered with a 9 or 10 “Promoters”). The claim behind the NPS system is that people who give responses within these ranges are qualitatively different ‘clusters’ (and also the people responding with a 7-8 are also a distinct cluster “Passives” who basically don’t matter and so who don’t figure in the NPS scores at all) and that just subtracting the percentages of one cluster from another is the “easiest-to-understand, most effective summary of how a company [is] performing in this context.”
Unfortunately, it does not seem to me that there’s a sound empirical basis for analysing an NPS style scale in this way (and the company behind it are quite untransparent about this basis (see discussion here). This way of analysing responses to a scale is pretty unusual and obscures most of the information about the distribution of responses, which it seems like it would be pretty easy for an EA audience to understand. For example, it seems like it would be pretty easy to depict the distribution of responses, as we did in the EA Survey Community information post.
And it seems like calculating the mean and median response would also give a more informative, but equally easy to understand summary of performance on this measure (more so than the NPS score, which for example, completely ignores whether people respond with a 0 or a 6). This would also allow easy significance testing of the differences between events/groups.