I think the tone of this post projected a confidence around your empirical views that initially led me to glaze over the actual data you present. But on a second read I noticed that the NPS at EAG was 5 points lower than the threshold you give for “excellent” (and far short of what you say is typical of Apple). This felt a bit jarring in light of the takeaways that “EAs really love EA”—so much so that being more welcoming isn’t a very pressing problem.
Nothing in this post is actually inconsistent with an NPS that’s short of excellent. I don’t even really have an opinion about whether NPS is a useful measure. But it does make me feel like the “potential implications” you list are things you already believed. Did the data affect your views much one way or another? Do you have a sense for the threshold at which you would have instead written a post that said “even though the NPS at EAG was only good, not great, I still believe that making EA more welcoming is not one of the most pressing problems facing EA?”
Similarly, I don’t really see how either of your conclusions could really be a “potential implication” of the fact that more than half of EAG applications were due to referrals. To me this data seems equally consistent with the exact opposite conclusions—only people who already know an EA well end up applying to EAG, which is evidence that EAG is unwelcoming and too cultish. Alternatively, if very few applications had come through referrals, say 5%, you could just as easily argue that this was evidence against the need to be more welcoming (tons of people who don’t know us feel welcome to apply) and against EA being too cultish (it’s not a closed system at all)!
Obviously it’s ok for you to have views that aren’t driven by this fairly limited data. But if the data isn’t really one of the stronger factors informing your views, I think I’d probably prefer to see them presented in separate posts. I think there’s otherwise a risk of building a habit of using “data as soldiers” (https://wiki.lesswrong.com/wiki/Arguments_as_soldiers) and losing opportunities to update.
I think the tone of this post projected a confidence around your empirical views that initially led me to glaze over the actual data you present.
This wasn’t the tone I was going for. On my reading of the post, it’s pretty hedge-y, with the exception of the title. Can you help me out by pointing out some ways that I seem overconfident in the empirical view?
To me this data seems equally consistent with the exact opposite conclusions—only people who already know an EA well end up applying to EAG, which is evidence that EAG is unwelcoming and too cultish.
I don’t see why this is a reasonable conclusion to reach. I think the welcoming/unwelcoming distinction is a claim about the experience of being in the community and interacting with EAs. Since new people haven’t had a chance to interact with EAs, it would be surprising if they found the community unwelcoming.
It could be that the EA brand prevents people from wanting to join the community in the first place. That seems like a hypothesis worth testing, but it doesn’t seem to me like a claim about how welcoming the community is.
That said, I am inferring from “EAs think their friends will like EA” to “therefore, the community is welcoming.” I agree that there are a number of ways that this inference could break down and I probably should have stuck more to the first claim instead of the second.
I added an edit to the post to reflect this concern.
Obviously it’s ok for you to have views that aren’t driven by this fairly limited data. But if the data isn’t really one of the stronger factors informing your views, I think I’d probably prefer to see them presented in separate posts.
This is fair. The data here does affect my view, but the view is also affected by some other data we collected around EA Global. I’m not sure a separate post would be the way to go, but a more nuanced discussion of whether this implies anything about the welcominness of the community would have been better.
I think welcoming/unwelcoming is one of those things that most people initially assess almost immediately upon contact with a community. Yes, people who stay in the community will update their perceptions over time, but I have definitely been to enough meetups, dances, and general social gatherings to have a sense after one interaction with a community of whether I feel welcome and to have noticed that this affects my probability of returning. It even affects my probability of returning if I go with a friend, or know a subset of people there; being welcomed by one or a few people is not necessarily enough if the community as a whole doesn’t feel welcoming, even if those people are deeply connected in the community and think I will want to return.
I also think that what it means for long-time community members to feel like a group is welcoming is pretty unclear, because they themselves actually do not need to be made welcome there. I would be much more interested in thinking about whether EA is welcoming based on how welcoming newcomers (say, people who have participated in local or online EA groups at least once, but for no longer than a year) think it is, what proportion of newcomers return or bring friends, etc.
Thanks for being so open to feedback and non-defensive on this and thanks especially for updating the body of the post. I think there’s a big problem where people change their minds due to feedback but it never gets propagated out to readers b/c it’s buried in a comment section. With preliminaries out of the way. . .
This wasn’t the tone I was going for. On my reading of the post, it’s pretty hedge-y, with the exception of the title. Can you help me out by pointing out some ways that I seem overconfident in the empirical view?
Looking back at it, I do think the headline really primed me here. But I think the other things were:
“the NPS score coupled with the referral evidence suggests that as a brand EA has a very dedicated fanbase that is willing to promote the brand.” I see how this could feel like a nitpick because you said “suggests” and not “proves” but I think somebody reading the post quickly, glossing over the actual numbers, and trusting your description of their implications (which I think is pretty common/reasonable) would take this to mean that the NPS score unambiguously is evidence in this direction and the uncertainty is due to the fact that it’s just one form of evidence. I’m not sure exactly what I would’ve said instead but I would’ve said something more moderate when describing data that’s less than excellent on it’s own terms. Taking out the phrase “very dedicated” would’ve made a pretty big difference, I think.
I think it would’ve made a big difference if there was some explicit discussion of the fact that the NPS score was good-but-not-great and could reasonably have been better. Putting the data in there for comparison definitely helps but if I’m reading something about metrics I’ve never seen before, I kind of expect the writer to do the work for me and tell me how to interpret the comparison. If there’s some data and then the author says it suggests “very dedicated fanbase” I’m likely to assume that the score EA got is relatively close to the score Apple got. If it’s not, that seems like an important enough fact to grapple with instead of just present.
*The other places you implicitly described what the data mean are “highly loyal community willing to promote EA” and “high brand loyalty.” Combining all of this, I think the post really reads like EA killed it on the NPS front.
I don’t see why this is a reasonable conclusion to reach. I think the welcoming/unwelcoming distinction is a claim about the experience of being in the community and interacting with EAs. Since new people haven’t had a chance to interact with EAs, it would be surprising if they found the community unwelcoming.
It could be that the EA brand prevents people from wanting to join the community in the first place. That seems like a hypothesis worth testing, but it doesn’t seem to me like a claim about how welcoming the community is.
Fair points but I’m not convinced.
I’d guess that there’s a lot of people who have had enough contact with the EA community to have been affected by its welcomingness or lack thereof but who wouldn’t have counted as applying to EAG b/c of someone else in EA according to your metric. People with relatively weak connections to EA are likely to be most affected by welcomingness so it seems possible that the relevant margin here is whether people who fall into the non-referral group feel welcome to apply.
I think a major mechanism through which welcomingness has effects is welcomingness → experience of people interacting with the community → EA’s reputation/brand among people outside the community. So I’d actually expect welcomingness to have a big effect on whether EA has a brand that gets people to want to join the community in the first place.
For a fairly big, somewhat outward facing event like EAG where the pool of potential non-EA applicants is so large compared to the pool of potential EA applicants, it seems possible that this mechanism through which welcomingness decreases proportion of applicants coming through referrals could dominate your proposed mechanism through which welcomingness increases the proportion of applicants coming through referrals.
JTBC, I don’t have a net take on the above. My main point is just that the direction is ambiguous so I don’t think the data says much about welcomingness.
I think the tone of this post projected a confidence around your empirical views that initially led me to glaze over the actual data you present. But on a second read I noticed that the NPS at EAG was 5 points lower than the threshold you give for “excellent” (and far short of what you say is typical of Apple). This felt a bit jarring in light of the takeaways that “EAs really love EA”—so much so that being more welcoming isn’t a very pressing problem.
Nothing in this post is actually inconsistent with an NPS that’s short of excellent. I don’t even really have an opinion about whether NPS is a useful measure. But it does make me feel like the “potential implications” you list are things you already believed. Did the data affect your views much one way or another? Do you have a sense for the threshold at which you would have instead written a post that said “even though the NPS at EAG was only good, not great, I still believe that making EA more welcoming is not one of the most pressing problems facing EA?”
Similarly, I don’t really see how either of your conclusions could really be a “potential implication” of the fact that more than half of EAG applications were due to referrals. To me this data seems equally consistent with the exact opposite conclusions—only people who already know an EA well end up applying to EAG, which is evidence that EAG is unwelcoming and too cultish. Alternatively, if very few applications had come through referrals, say 5%, you could just as easily argue that this was evidence against the need to be more welcoming (tons of people who don’t know us feel welcome to apply) and against EA being too cultish (it’s not a closed system at all)!
Obviously it’s ok for you to have views that aren’t driven by this fairly limited data. But if the data isn’t really one of the stronger factors informing your views, I think I’d probably prefer to see them presented in separate posts. I think there’s otherwise a risk of building a habit of using “data as soldiers” (https://wiki.lesswrong.com/wiki/Arguments_as_soldiers) and losing opportunities to update.
This wasn’t the tone I was going for. On my reading of the post, it’s pretty hedge-y, with the exception of the title. Can you help me out by pointing out some ways that I seem overconfident in the empirical view?
I don’t see why this is a reasonable conclusion to reach. I think the welcoming/unwelcoming distinction is a claim about the experience of being in the community and interacting with EAs. Since new people haven’t had a chance to interact with EAs, it would be surprising if they found the community unwelcoming.
It could be that the EA brand prevents people from wanting to join the community in the first place. That seems like a hypothesis worth testing, but it doesn’t seem to me like a claim about how welcoming the community is.
That said, I am inferring from “EAs think their friends will like EA” to “therefore, the community is welcoming.” I agree that there are a number of ways that this inference could break down and I probably should have stuck more to the first claim instead of the second.
I added an edit to the post to reflect this concern.
This is fair. The data here does affect my view, but the view is also affected by some other data we collected around EA Global. I’m not sure a separate post would be the way to go, but a more nuanced discussion of whether this implies anything about the welcominness of the community would have been better.
I think welcoming/unwelcoming is one of those things that most people initially assess almost immediately upon contact with a community. Yes, people who stay in the community will update their perceptions over time, but I have definitely been to enough meetups, dances, and general social gatherings to have a sense after one interaction with a community of whether I feel welcome and to have noticed that this affects my probability of returning. It even affects my probability of returning if I go with a friend, or know a subset of people there; being welcomed by one or a few people is not necessarily enough if the community as a whole doesn’t feel welcoming, even if those people are deeply connected in the community and think I will want to return.
I also think that what it means for long-time community members to feel like a group is welcoming is pretty unclear, because they themselves actually do not need to be made welcome there. I would be much more interested in thinking about whether EA is welcoming based on how welcoming newcomers (say, people who have participated in local or online EA groups at least once, but for no longer than a year) think it is, what proportion of newcomers return or bring friends, etc.
Thanks for being so open to feedback and non-defensive on this and thanks especially for updating the body of the post. I think there’s a big problem where people change their minds due to feedback but it never gets propagated out to readers b/c it’s buried in a comment section. With preliminaries out of the way. . .
Looking back at it, I do think the headline really primed me here. But I think the other things were: “the NPS score coupled with the referral evidence suggests that as a brand EA has a very dedicated fanbase that is willing to promote the brand.” I see how this could feel like a nitpick because you said “suggests” and not “proves” but I think somebody reading the post quickly, glossing over the actual numbers, and trusting your description of their implications (which I think is pretty common/reasonable) would take this to mean that the NPS score unambiguously is evidence in this direction and the uncertainty is due to the fact that it’s just one form of evidence. I’m not sure exactly what I would’ve said instead but I would’ve said something more moderate when describing data that’s less than excellent on it’s own terms. Taking out the phrase “very dedicated” would’ve made a pretty big difference, I think. I think it would’ve made a big difference if there was some explicit discussion of the fact that the NPS score was good-but-not-great and could reasonably have been better. Putting the data in there for comparison definitely helps but if I’m reading something about metrics I’ve never seen before, I kind of expect the writer to do the work for me and tell me how to interpret the comparison. If there’s some data and then the author says it suggests “very dedicated fanbase” I’m likely to assume that the score EA got is relatively close to the score Apple got. If it’s not, that seems like an important enough fact to grapple with instead of just present. *The other places you implicitly described what the data mean are “highly loyal community willing to promote EA” and “high brand loyalty.” Combining all of this, I think the post really reads like EA killed it on the NPS front.
I’d guess that there’s a lot of people who have had enough contact with the EA community to have been affected by its welcomingness or lack thereof but who wouldn’t have counted as applying to EAG b/c of someone else in EA according to your metric. People with relatively weak connections to EA are likely to be most affected by welcomingness so it seems possible that the relevant margin here is whether people who fall into the non-referral group feel welcome to apply. I think a major mechanism through which welcomingness has effects is welcomingness → experience of people interacting with the community → EA’s reputation/brand among people outside the community. So I’d actually expect welcomingness to have a big effect on whether EA has a brand that gets people to want to join the community in the first place. For a fairly big, somewhat outward facing event like EAG where the pool of potential non-EA applicants is so large compared to the pool of potential EA applicants, it seems possible that this mechanism through which welcomingness decreases proportion of applicants coming through referrals could dominate your proposed mechanism through which welcomingness increases the proportion of applicants coming through referrals. JTBC, I don’t have a net take on the above. My main point is just that the direction is ambiguous so I don’t think the data says much about welcomingness.