How many of these findings are likely to be of practical (as opposed to statistical) significance? (I’m looking mostly at the 21-30 group and to a lesser extent at the 31-40 group given EA demographics and the relatedly low number of 40+ respondents.)
It’s been a long time since I was a graduate student, but it looks like most of the differences are +/- 0.25 points on a seven-point scale even before we consider the error bars. In other words, it seems that the differences we can be statistically confident in are significantly less than .25 in almost all cases. Is a difference of .1 or .25 on a seven-point scale likely to be substantively meaningful?
Likewise, most of the correlations between traits and donation were .03 or less and not statistically significant. But then again, I vaguely recall that correlations can be fairly low and still meaningful when one of the variables is binary?
The exception looks to be lower rates of male neuroticism, which seems more likely to be of practical significance.
To add some specifics to my earlier comment, if we look at the confidence intervals for the effect sizes in terms of cohen’s d (visualizer), we see that:
The effect size for conscientiousness for men age 21-30, was 0.182 (0.0979 to 0.266). So between an almost 0 effect size and a smallish one.
The confidence intervals for effect size for neuroticism for men aged 21-30 was 0.388 (0.255 to 0.421). So a small to medium effect size.
The confidence intervals for effect size for neuroticism for men aged 31-40 was 0.366 (0.231 to 0.500). So between a small and medium effect size.
Interpreting effect sizes is, of course, not straightforward. The conventional standards are somewhat arbitrary, and it’s quite widely agreed that the classic cohen’s d benchmarks for small/medium/large effect sizes are quite conservative (the average effect size in psychology is no more than d=0.3). This empirically generated set of benchmarks (if you convert r into d), would suggest that around 0.2 is small and 0.4 is medium. But whether a particular effect size is practically meaningful varies depending on the context. For example, a small effect may make very little difference at the individual level, but make large differences at the aggregate level / in the long-run.
In my personal view, very few or none, if you are looking at the association between personality and outcomes. As we note the associations between personality, donation behavior and cause prioritization were “only small”. I think that in itself is an important finding, since some people would expect large influences on things like donation or cause prioritisation.
If you’re talking about the differences in personality between EAs and the general population, I think these are potentially more practically significant (for example, if you’re considering things like influences on outreach / recruitment, or the influence of these differences on the kinds of people EA attracts). Here even small differences could be significant in a non-linear way (for example, if EA is disproportionately appealing to people who are very high in need for cognition, or similar traits, this could have a big effect). Some of these apparent differences between EA and the general population are not obviously small, though the level of confidence we can have for the analyses where we can benchmark to population levels at both the gender and age level (i.e. big 5) do not give us clarity about what the exact magnitude of the differences are. Even with 1600 respondents, the sample size is not so large when you are accounting for age and gender in this way (and we argue that you do need to do this in this case for the results to be interpretable).
I had never thought about it, but I can see some features of the intellectual and practical ecosystem that would select against people with higher neuroticism (at least to the extent it manifested as anxiety and certain other dimensions):
People who are higher in neuroticism might find EA’s attitude toward risk tolerance challenging to accept.
The prevalence of grant-based funding is difficult for many people more temperamentally prone to anxiety to tolerate.
More generally, effectiveness orientation is more anxiety-inducing than a belief in Lake Wobegon charity where all efforts are above average.
So I wonder if the [edit: people who predicted high neuroticism] are conflating EA’s pressures can cause neurotic symptoms with EA attracts relatively more people with neurotic temperaments. My rough (and extreme) analogy might be air-traffic controllers; the work seems very likely to cause anxiety and some other negative emotional states. At the same time, people who are higher in neuroticism may be more likely to choose another career (or be screened out by psychological testing).
Unfortunately, to test these hypotheses, one would probably need to sample an appropriate non-EA population and correlate certain attitudes toward generalized openness to the EA way of thought.
I wonder if the predictors are conflating EA’s pressures can cause neurotic symptoms with EA attracts relatively more people with neurotic temperaments.
My default assumption would be that we’re measuring trait-neuroticism rather than just temporary, locally caused anxiety. That’s partly because personality traits are relatively stable, but also because I’d be surprised if EA were having a large effect on people’s tendency to describe themselves as being “anxious, easily upset” etc. (and that doesn’t seem to be the case in our results on the effect of EA on mental health). Of course, it’s also worth noting that our results in this study tended towards lower neuroticism for EAs.
I do think that whether the results are driven more by EA selecting for people who are higher in emotional stability at the outset, or whether the community is losing people with higher trait-neuroticism, is a significant question however. I agree that we couldn’t empirically tackle this without further data, such as by measuring personality across years and tracking dropout.
Sorry, by predictors I had meant “people who had predicted you would find higher levels of neuroticism,” not your survey questions. Edited my comment to clarify.
How many of these findings are likely to be of practical (as opposed to statistical) significance? (I’m looking mostly at the 21-30 group and to a lesser extent at the 31-40 group given EA demographics and the relatedly low number of 40+ respondents.)
It’s been a long time since I was a graduate student, but it looks like most of the differences are +/- 0.25 points on a seven-point scale even before we consider the error bars. In other words, it seems that the differences we can be statistically confident in are significantly less than .25 in almost all cases. Is a difference of .1 or .25 on a seven-point scale likely to be substantively meaningful?
Likewise, most of the correlations between traits and donation were .03 or less and not statistically significant. But then again, I vaguely recall that correlations can be fairly low and still meaningful when one of the variables is binary?
The exception looks to be lower rates of male neuroticism, which seems more likely to be of practical significance.
To add some specifics to my earlier comment, if we look at the confidence intervals for the effect sizes in terms of cohen’s d (visualizer), we see that:
The effect size for conscientiousness for men age 21-30, was 0.182 (0.0979 to 0.266). So between an almost 0 effect size and a smallish one.
The confidence intervals for effect size for neuroticism for men aged 21-30 was 0.388 (0.255 to 0.421). So a small to medium effect size.
The confidence intervals for effect size for neuroticism for men aged 31-40 was 0.366 (0.231 to 0.500). So between a small and medium effect size.
Interpreting effect sizes is, of course, not straightforward. The conventional standards are somewhat arbitrary, and it’s quite widely agreed that the classic cohen’s d benchmarks for small/medium/large effect sizes are quite conservative (the average effect size in psychology is no more than d=0.3). This empirically generated set of benchmarks (if you convert r into d), would suggest that around 0.2 is small and 0.4 is medium. But whether a particular effect size is practically meaningful varies depending on the context. For example, a small effect may make very little difference at the individual level, but make large differences at the aggregate level / in the long-run.
In my personal view, very few or none, if you are looking at the association between personality and outcomes. As we note the associations between personality, donation behavior and cause prioritization were “only small”. I think that in itself is an important finding, since some people would expect large influences on things like donation or cause prioritisation.
If you’re talking about the differences in personality between EAs and the general population, I think these are potentially more practically significant (for example, if you’re considering things like influences on outreach / recruitment, or the influence of these differences on the kinds of people EA attracts). Here even small differences could be significant in a non-linear way (for example, if EA is disproportionately appealing to people who are very high in need for cognition, or similar traits, this could have a big effect). Some of these apparent differences between EA and the general population are not obviously small, though the level of confidence we can have for the analyses where we can benchmark to population levels at both the gender and age level (i.e. big 5) do not give us clarity about what the exact magnitude of the differences are. Even with 1600 respondents, the sample size is not so large when you are accounting for age and gender in this way (and we argue that you do need to do this in this case for the results to be interpretable).
I think this is also notable, whatever the effect size, because people have often predicted a directionally opposite result.
I had never thought about it, but I can see some features of the intellectual and practical ecosystem that would select against people with higher neuroticism (at least to the extent it manifested as anxiety and certain other dimensions):
People who are higher in neuroticism might find EA’s attitude toward risk tolerance challenging to accept.
The prevalence of grant-based funding is difficult for many people more temperamentally prone to anxiety to tolerate.
More generally, effectiveness orientation is more anxiety-inducing than a belief in Lake Wobegon charity where all efforts are above average.
So I wonder if the [edit: people who predicted high neuroticism] are conflating EA’s pressures can cause neurotic symptoms with EA attracts relatively more people with neurotic temperaments. My rough (and extreme) analogy might be air-traffic controllers; the work seems very likely to cause anxiety and some other negative emotional states. At the same time, people who are higher in neuroticism may be more likely to choose another career (or be screened out by psychological testing).
Unfortunately, to test these hypotheses, one would probably need to sample an appropriate non-EA population and correlate certain attitudes toward generalized openness to the EA way of thought.
My default assumption would be that we’re measuring trait-neuroticism rather than just temporary, locally caused anxiety. That’s partly because personality traits are relatively stable, but also because I’d be surprised if EA were having a large effect on people’s tendency to describe themselves as being “anxious, easily upset” etc. (and that doesn’t seem to be the case in our results on the effect of EA on mental health). Of course, it’s also worth noting that our results in this study tended towards lower neuroticism for EAs.
I do think that whether the results are driven more by EA selecting for people who are higher in emotional stability at the outset, or whether the community is losing people with higher trait-neuroticism, is a significant question however. I agree that we couldn’t empirically tackle this without further data, such as by measuring personality across years and tracking dropout.
Sorry, by predictors I had meant “people who had predicted you would find higher levels of neuroticism,” not your survey questions. Edited my comment to clarify.