Sure! I’ve asked the relevant people to respond with the NPS figures if it’s quick/easy for them to do so, but they might prioritize other things.
Btw, I disagree about how useful NPS is. I think it’s quite a weird metric (with very strong threshold effects between 6⁄7 and 8⁄9, and no discrimination between a 6 and a 1). That’s why we switched to the mean. I do think that looking at a histogram is often useful though- in most cases the mean doesn’t give you a strong sense of the distribution.
Thanks! I guess I think NPS is useful precisely because of those threshold effects, but agree not sure that it handles the discrimination between 6 and 1 well. Histograms seem great!
Hmm, I still think the threshold effects are kinda weird, and so NPS shouldn’t be the main measure. (I know you’re just asking for it as supplementary info, and I think we’d maybe both prefer mean + histogram.)
There’s a prima facie case, that’s like: the threshold effects say that you care totally about the 6⁄7 and 8⁄9 boundaries, and not-at-all about the 5⁄6, 7⁄8, 9⁄10 boundaries. That’s weird!
I could imagine a view that’s like “it’s really important to have enthusiastic promoters because they help spread the word about your product” or something, but then why would that view want you to care not-at-all about the 9⁄10 boundary? I imagine 10s are more enthusiastic promoters, and it seems plausible to me that the 9⁄10 differential is the same or greater than the 8⁄9 differential.
And why would it want you to care not-at-all about the 7⁄8 boundary? I imagine 8s could be enthusiastic promoters, more so than 7s.
And similar comments for a view that’s like “it’s really important to avoid having detractors, because they put people off”.
I could also imagine a kinda startup-y view that’s like “it’s really important to get excellent product market fit, which means focusing on getting some people to really love your product, rather than a large group of people to like it”. But on that view, why ignore the 9⁄10 boundary? And why care about detractors?
I also think that maybe all of the above views make more sense when your aim is to predict whether your product will grow virally (not our focus), vs. whether it’s generally high quality/providing something that people want (more our focus). So they might just not carry over well to our case.
Thanks for explaining your view! I don’t really have super strong views here, so don’t want to labour the point, but just thought I’d share my intuition for where I’m coming from. For me it makes sense to have a thresholds at the places because it does actually carve up the buckets of reactions better than the linear scale suggests.
For example, some people feel weird rating something really low and so they “express dislike” by rating it 6⁄10. So to me the lowest scorers and the 6/10ers are actually probably have more similar experiences than their linear score suggests. I claim this is driven by weird habits/something psychological of how people are used to rating things.
I think there’s a similar thing at the 7/8/9 distinction. I think when people think something is “okay” they just rate it 7⁄10. But when someone is actually impressed by something they rate it 9⁄10, which is only 2 points more but actually captures a quite different sentiment. From experience also I’ve noticed some people use 9⁄10 in place of 10⁄10 because they just never give anything 10⁄10 (e.g they understand what it means for something to be 10⁄10 differently to others)
The short of it is that I claim people don’t seem to use the linear scale as an actual linear scale , and so it makes sense to normalise things with the thresholds, and I claim that the thresholds are at the right place mostly just from my (very limited) experience
Sure! I’ve asked the relevant people to respond with the NPS figures if it’s quick/easy for them to do so, but they might prioritize other things.
Btw, I disagree about how useful NPS is. I think it’s quite a weird metric (with very strong threshold effects between 6⁄7 and 8⁄9, and no discrimination between a 6 and a 1). That’s why we switched to the mean. I do think that looking at a histogram is often useful though- in most cases the mean doesn’t give you a strong sense of the distribution.
Thanks! I guess I think NPS is useful precisely because of those threshold effects, but agree not sure that it handles the discrimination between 6 and 1 well. Histograms seem great!
Hmm, I still think the threshold effects are kinda weird, and so NPS shouldn’t be the main measure. (I know you’re just asking for it as supplementary info, and I think we’d maybe both prefer mean + histogram.)
There’s a prima facie case, that’s like: the threshold effects say that you care totally about the 6⁄7 and 8⁄9 boundaries, and not-at-all about the 5⁄6, 7⁄8, 9⁄10 boundaries. That’s weird!
I could imagine a view that’s like “it’s really important to have enthusiastic promoters because they help spread the word about your product” or something, but then why would that view want you to care not-at-all about the 9⁄10 boundary? I imagine 10s are more enthusiastic promoters, and it seems plausible to me that the 9⁄10 differential is the same or greater than the 8⁄9 differential.
And why would it want you to care not-at-all about the 7⁄8 boundary? I imagine 8s could be enthusiastic promoters, more so than 7s.
And similar comments for a view that’s like “it’s really important to avoid having detractors, because they put people off”.
I could also imagine a kinda startup-y view that’s like “it’s really important to get excellent product market fit, which means focusing on getting some people to really love your product, rather than a large group of people to like it”. But on that view, why ignore the 9⁄10 boundary? And why care about detractors?
I also think that maybe all of the above views make more sense when your aim is to predict whether your product will grow virally (not our focus), vs. whether it’s generally high quality/providing something that people want (more our focus). So they might just not carry over well to our case.
Thanks for explaining your view! I don’t really have super strong views here, so don’t want to labour the point, but just thought I’d share my intuition for where I’m coming from. For me it makes sense to have a thresholds at the places because it does actually carve up the buckets of reactions better than the linear scale suggests.
For example, some people feel weird rating something really low and so they “express dislike” by rating it 6⁄10. So to me the lowest scorers and the 6/10ers are actually probably have more similar experiences than their linear score suggests. I claim this is driven by weird habits/something psychological of how people are used to rating things.
I think there’s a similar thing at the 7/8/9 distinction. I think when people think something is “okay” they just rate it 7⁄10. But when someone is actually impressed by something they rate it 9⁄10, which is only 2 points more but actually captures a quite different sentiment. From experience also I’ve noticed some people use 9⁄10 in place of 10⁄10 because they just never give anything 10⁄10 (e.g they understand what it means for something to be 10⁄10 differently to others)
The short of it is that I claim people don’t seem to use the linear scale as an actual linear scale , and so it makes sense to normalise things with the thresholds, and I claim that the thresholds are at the right place mostly just from my (very limited) experience
Thanks for explaining! The guess about how people use the scale seems pretty plausible to me.