Thanks for being so open to feedback and non-defensive on this and thanks especially for updating the body of the post. I think there’s a big problem where people change their minds due to feedback but it never gets propagated out to readers b/c it’s buried in a comment section. With preliminaries out of the way. . .
This wasn’t the tone I was going for. On my reading of the post, it’s pretty hedge-y, with the exception of the title. Can you help me out by pointing out some ways that I seem overconfident in the empirical view?
Looking back at it, I do think the headline really primed me here. But I think the other things were:
“the NPS score coupled with the referral evidence suggests that as a brand EA has a very dedicated fanbase that is willing to promote the brand.” I see how this could feel like a nitpick because you said “suggests” and not “proves” but I think somebody reading the post quickly, glossing over the actual numbers, and trusting your description of their implications (which I think is pretty common/reasonable) would take this to mean that the NPS score unambiguously is evidence in this direction and the uncertainty is due to the fact that it’s just one form of evidence. I’m not sure exactly what I would’ve said instead but I would’ve said something more moderate when describing data that’s less than excellent on it’s own terms. Taking out the phrase “very dedicated” would’ve made a pretty big difference, I think.
I think it would’ve made a big difference if there was some explicit discussion of the fact that the NPS score was good-but-not-great and could reasonably have been better. Putting the data in there for comparison definitely helps but if I’m reading something about metrics I’ve never seen before, I kind of expect the writer to do the work for me and tell me how to interpret the comparison. If there’s some data and then the author says it suggests “very dedicated fanbase” I’m likely to assume that the score EA got is relatively close to the score Apple got. If it’s not, that seems like an important enough fact to grapple with instead of just present.
*The other places you implicitly described what the data mean are “highly loyal community willing to promote EA” and “high brand loyalty.” Combining all of this, I think the post really reads like EA killed it on the NPS front.
I don’t see why this is a reasonable conclusion to reach. I think the welcoming/unwelcoming distinction is a claim about the experience of being in the community and interacting with EAs. Since new people haven’t had a chance to interact with EAs, it would be surprising if they found the community unwelcoming.
It could be that the EA brand prevents people from wanting to join the community in the first place. That seems like a hypothesis worth testing, but it doesn’t seem to me like a claim about how welcoming the community is.
Fair points but I’m not convinced.
I’d guess that there’s a lot of people who have had enough contact with the EA community to have been affected by its welcomingness or lack thereof but who wouldn’t have counted as applying to EAG b/c of someone else in EA according to your metric. People with relatively weak connections to EA are likely to be most affected by welcomingness so it seems possible that the relevant margin here is whether people who fall into the non-referral group feel welcome to apply.
I think a major mechanism through which welcomingness has effects is welcomingness → experience of people interacting with the community → EA’s reputation/brand among people outside the community. So I’d actually expect welcomingness to have a big effect on whether EA has a brand that gets people to want to join the community in the first place.
For a fairly big, somewhat outward facing event like EAG where the pool of potential non-EA applicants is so large compared to the pool of potential EA applicants, it seems possible that this mechanism through which welcomingness decreases proportion of applicants coming through referrals could dominate your proposed mechanism through which welcomingness increases the proportion of applicants coming through referrals.
JTBC, I don’t have a net take on the above. My main point is just that the direction is ambiguous so I don’t think the data says much about welcomingness.
Thanks for being so open to feedback and non-defensive on this and thanks especially for updating the body of the post. I think there’s a big problem where people change their minds due to feedback but it never gets propagated out to readers b/c it’s buried in a comment section. With preliminaries out of the way. . .
Looking back at it, I do think the headline really primed me here. But I think the other things were: “the NPS score coupled with the referral evidence suggests that as a brand EA has a very dedicated fanbase that is willing to promote the brand.” I see how this could feel like a nitpick because you said “suggests” and not “proves” but I think somebody reading the post quickly, glossing over the actual numbers, and trusting your description of their implications (which I think is pretty common/reasonable) would take this to mean that the NPS score unambiguously is evidence in this direction and the uncertainty is due to the fact that it’s just one form of evidence. I’m not sure exactly what I would’ve said instead but I would’ve said something more moderate when describing data that’s less than excellent on it’s own terms. Taking out the phrase “very dedicated” would’ve made a pretty big difference, I think. I think it would’ve made a big difference if there was some explicit discussion of the fact that the NPS score was good-but-not-great and could reasonably have been better. Putting the data in there for comparison definitely helps but if I’m reading something about metrics I’ve never seen before, I kind of expect the writer to do the work for me and tell me how to interpret the comparison. If there’s some data and then the author says it suggests “very dedicated fanbase” I’m likely to assume that the score EA got is relatively close to the score Apple got. If it’s not, that seems like an important enough fact to grapple with instead of just present. *The other places you implicitly described what the data mean are “highly loyal community willing to promote EA” and “high brand loyalty.” Combining all of this, I think the post really reads like EA killed it on the NPS front.
I’d guess that there’s a lot of people who have had enough contact with the EA community to have been affected by its welcomingness or lack thereof but who wouldn’t have counted as applying to EAG b/c of someone else in EA according to your metric. People with relatively weak connections to EA are likely to be most affected by welcomingness so it seems possible that the relevant margin here is whether people who fall into the non-referral group feel welcome to apply. I think a major mechanism through which welcomingness has effects is welcomingness → experience of people interacting with the community → EA’s reputation/brand among people outside the community. So I’d actually expect welcomingness to have a big effect on whether EA has a brand that gets people to want to join the community in the first place. For a fairly big, somewhat outward facing event like EAG where the pool of potential non-EA applicants is so large compared to the pool of potential EA applicants, it seems possible that this mechanism through which welcomingness decreases proportion of applicants coming through referrals could dominate your proposed mechanism through which welcomingness increases the proportion of applicants coming through referrals. JTBC, I don’t have a net take on the above. My main point is just that the direction is ambiguous so I don’t think the data says much about welcomingness.