Caveat: I work for Rethink Priorities, as do the authors of this post
Great to have this timely research and good antidote to some of the gloomy outlooks on how this has affected the community!
I especially liked the two convergent approaches for measuring the effect of the FTX crisis on EA satisfaction. I noticed in the satisfaction over time analysis, you can kind of eyeball a (slight) negative pre-trend. This made me wonder how you thought about the causal inference concerns both in that specific analysis and for this report generally? By extension, I wonder if the summary could be a bit more explicit:
the FTX crisis had decreased satisfaction with the EA community
Maybe this should read something like “FTX crisis was associated with, and may have caused, decreased satisfaction”?
Yeah it’s fair to worry about the causal interpretation of the satisfaction results, although I think the pre-trend you mention is mostly a result of the GAM model pulling the regression line down a bit due to the later observations. Limiting the analysis to the period before, say, November 9th, shows no such pre-trend. I’m personally more worried about possible confounds such as the one we note about engagement. It could also be that the FTX crisis motivated a certain group of people to take the survey and share their dissatisfaction. So, it’s true that the design of the survey does not lend itself to easily answer questions of causality, leaving it a matter of interpretation based on the results as a whole and context.
Regarding the overall interpretation, I think it might still be fair to conclude that the FTX crisis has decreased satisfaction if we take into account the other results, including the recalled satisfaction pre and before the FTX crisis and the explicit reports of concerns related to FTX, decreased trust etc.
Caveat: I work for Rethink Priorities, as do the authors of this post
Great to have this timely research and good antidote to some of the gloomy outlooks on how this has affected the community!
I especially liked the two convergent approaches for measuring the effect of the FTX crisis on EA satisfaction. I noticed in the satisfaction over time analysis, you can kind of eyeball a (slight) negative pre-trend. This made me wonder how you thought about the causal inference concerns both in that specific analysis and for this report generally? By extension, I wonder if the summary could be a bit more explicit:
Maybe this should read something like “FTX crisis was associated with, and may have caused, decreased satisfaction”?
Yeah it’s fair to worry about the causal interpretation of the satisfaction results, although I think the pre-trend you mention is mostly a result of the GAM model pulling the regression line down a bit due to the later observations. Limiting the analysis to the period before, say, November 9th, shows no such pre-trend. I’m personally more worried about possible confounds such as the one we note about engagement. It could also be that the FTX crisis motivated a certain group of people to take the survey and share their dissatisfaction. So, it’s true that the design of the survey does not lend itself to easily answer questions of causality, leaving it a matter of interpretation based on the results as a whole and context.
Regarding the overall interpretation, I think it might still be fair to conclude that the FTX crisis has decreased satisfaction if we take into account the other results, including the recalled satisfaction pre and before the FTX crisis and the explicit reports of concerns related to FTX, decreased trust etc.
Thanks, Willem, that all makes sense! I agree, the overall conclusion certainly seems fair, especially given the convergent evidence you cite.