I think these surveys are complementary and each have different strengths and weaknesses relevant for different purposes.[1] However, I think what the strengths and weaknesses are and how to interpret the surveys in light of them is not immediately obvious. And I know that in at least some cases, decision-makers have had straightforwardly mistaken factual beliefs about the surveys which has mislead them about how to interpret them. This is a problem if people mistakenly rely on the results of only one of the surveys, or assign the wrong weights to each survey, when answering different questions.
A post about this would outline the key strengths and weaknesses of the different surveys for different purposes, touching on questions such as:
How much our confidence should change when we have a small sample size from a small population.
How concerned we should be about biases in the samples for each survey and what population we should be targeting.
How much the different questions in each survey allows us to check and verify the answers within each survey.
How much the results of each survey can be verified and cross-referenced with each other (e.g. by identifying specific highly engaged LTists within the EAS).
Reassuringly, they also seem to generate very similar results, when we directly compare them, adjusting for differences in composition, i.e. only looking at highly engaged longtermists within the EA Survey.
How to interpret the EA Survey and Open Phil EA/LT Survey.
I think these surveys are complementary and each have different strengths and weaknesses relevant for different purposes.[1] However, I think what the strengths and weaknesses are and how to interpret the surveys in light of them is not immediately obvious. And I know that in at least some cases, decision-makers have had straightforwardly mistaken factual beliefs about the surveys which has mislead them about how to interpret them. This is a problem if people mistakenly rely on the results of only one of the surveys, or assign the wrong weights to each survey, when answering different questions.
A post about this would outline the key strengths and weaknesses of the different surveys for different purposes, touching on questions such as:
How much our confidence should change when we have a small sample size from a small population.
How concerned we should be about biases in the samples for each survey and what population we should be targeting.
How much the different questions in each survey allows us to check and verify the answers within each survey.
How much the results of each survey can be verified and cross-referenced with each other (e.g. by identifying specific highly engaged LTists within the EAS).
Reassuringly, they also seem to generate very similar results, when we directly compare them, adjusting for differences in composition, i.e. only looking at highly engaged longtermists within the EA Survey.
Nice. I’d find this super interesting!