Thanks for sharing the results and thanks, in particular, for including the results for the particular measures, rather than just the composite score.
high writing scores predicted less engagement… Model (3) shows what is driving this: our measures of open-mindedness and commitment. It is unclear why this is. One story for open-mindedness could be that open-minded applicants are less likely to go all-in on EA socials and events and prefer to read widely. And a story for commitment could be that those most committed to the fellowship spent more time reading the extra readings and thus had less time for non-fellowship engagement.
Taking the results at face value, it seems like this could be explained by your measures systematically measuring something other than what you take them to be measuring (e.g. the problem is construct validity). For example, perhaps your measures of “open-mindedness” or “commitment” actually just tracked people’s inclination to acquiesce to social pressure, or something associated with it. Of course, I don’t know how you actually measured open-mindedness or commitment, so my speculation isn’t based on having any particular reason to think your measures were bad.
Of course, not taking the results at face value, it could just be idiosyncracies of what you note was a small sample. It could be interesting to see plots of the relationship between some of the variables, to help get a sense of whether some of the effects could be driven by outliers etc.
I think it’s completely plausible that these two measures were systematically measuring something other than what we took them to be measuring. The confusing thing is what it indeed was measuring and why these traits had negative effects.
(The way we judged open-mindedness, for example, was by asking applicants to write down an instance where they changed their minds in response to evidence.)
But I do think the most likely case is the small sample.
Thanks for sharing the results and thanks, in particular, for including the results for the particular measures, rather than just the composite score.
Taking the results at face value, it seems like this could be explained by your measures systematically measuring something other than what you take them to be measuring (e.g. the problem is construct validity). For example, perhaps your measures of “open-mindedness” or “commitment” actually just tracked people’s inclination to acquiesce to social pressure, or something associated with it. Of course, I don’t know how you actually measured open-mindedness or commitment, so my speculation isn’t based on having any particular reason to think your measures were bad.
Of course, not taking the results at face value, it could just be idiosyncracies of what you note was a small sample. It could be interesting to see plots of the relationship between some of the variables, to help get a sense of whether some of the effects could be driven by outliers etc.
Thanks for the comment!
I think it’s completely plausible that these two measures were systematically measuring something other than what we took them to be measuring. The confusing thing is what it indeed was measuring and why these traits had negative effects.
(The way we judged open-mindedness, for example, was by asking applicants to write down an instance where they changed their minds in response to evidence.)
But I do think the most likely case is the small sample.