In the world where changes to the survey explain the drop, I’d expect to see a similar number of people click through to the survey (especially in 2019) but a lower completion rate. Do you happen to have data on the completion rate by year?
If the number of people visiting the survey has dropped, then that seems consistent with the hypothesis that the drop is explained by the movement shrinking unless the increased time cost of completing the survey was made very clear upfront in 2019 and 2020.
If the number of people visiting the survey has dropped, then that seems consistent with the hypothesis that the drop is explained by the movement shrinking unless the increased time cost of completing the survey was made very clear upfront in 2019 and 2020
Unfortunately (for testing your hypothesis in this manner) the length of the survey is made very explicit upfront. The estimated length of the EAS2019 was 2-3x longer than EAS2018 (as it happened, this was an over-estimate, though it was still much longer than in 2018), while the estimated length of EAS2020 was a mere 2x longer than EAS2018.
Also, I would expect a longer, more demanding survey to lead to fewer total respondents in the year of the survey itself (and not merely lagged a year), since I think current-year uptake can be influenced by word of mouth and sharing (I imagine people would be less likely to share and recommend others take the survey if they found the survey long or annoying).
That said, as I noted in my original comment, I would expect to see lag effects (the survey being too long reduces response to the next year’s survey) and I might expect these effects to be larger (and to stack if the next year’s survey is itself too long) and this is exactly what we see: we see a moderate did from 2018 to 2019 and then a much larger dip from 2019 to 2020.
I’d expect to see a similar number of people click through to the survey (especially in 2019) but a lower completion rate
“Completion rate” is not entirely straightforward, because we explicitly instruct respondents that the final questions of the survey are especially optional “extra credit” questions and they should feel free to quit the survey before these. We can, however, look at the final questions of the main section of the survey (before the extra credit section) and here we see roughly the predicted pattern: a big drop in those ‘completing’ the main section from 2018 to 2019 followed by a smaller absolute drop 2019 to 2020, even though the percentage of those who started the survey completing the main section actually increased between 2019 and 2020 (which we might expect if some people, who are less inclined to take the survey, were put off taking it).
Another (and I think better way) of examining whether we are simply sampling fewer people or the population has shrunk is comparing numbers for subpopulations of the EA Survey(s) to known population sizes as we did here.
In 2019, we estimated that we sampled around 40% of the ‘highly engaged’ EA population. In 2020, using updated numbers, we estimated that we sampled around 35% of the highly engaged EA population.
If the true EA population had remained the same size 2019-2020 and we just sampled 35% rather than 40% overall, we would expect the number of EAs sampled in 2020 to decrease from 2513 to 2199 (which is pretty close to the 2166 we actually sampled).
But, as noted, we believe that we usually sample high and low engagement EAs at different rates (sampling relatively fewer less engaged EAs). And, if our sampling rate reduces overall, (a priori) I would expect this to hold up better among highly engaged EAs than among less engaged EAs (who may be less motivated to take the survey and more responsive to the survey becoming too onerous to complete).
The total number of highly engaged EAs in our sample this year was similar/slightly higher than 2019, implying that population slightly increased in size (as I would expect). We don’t have any good proxies for the size of the less engaged EA population (this becomes harder and harder as we consider the progressively larger populations of progressively less engaged EAs), but I would guess that we probably experienced a yet larger reduction in sampling rate for this population, and that the true size of the population of less engaged EAs has probably increased (I would probably look to things like the EA newsletter, subscribers to 80,000 Hours mailing lists and so on for proxies for that group, but Aaron/people at 80K may disagree).
Of course, much of this growth in the number of highly engaged EAs is likely due to EAs becoming more engaged, rather than there becoming more EAs. As it happens, EAS2020 had more 4′s but fewer 5′s, which I think can plausibly be explained by the general reduction in rate of people sampled, mentioned above, but a number of 1-3s moving into the 4 category and fewer 4s moving into the 5 category (which is more stringent, e.g. EA org employee, group leader etc.).
In the world where changes to the survey explain the drop, I’d expect to see a similar number of people click through to the survey (especially in 2019) but a lower completion rate. Do you happen to have data on the completion rate by year?
If the number of people visiting the survey has dropped, then that seems consistent with the hypothesis that the drop is explained by the movement shrinking unless the increased time cost of completing the survey was made very clear upfront in 2019 and 2020.
Unfortunately (for testing your hypothesis in this manner) the length of the survey is made very explicit upfront. The estimated length of the EAS2019 was 2-3x longer than EAS2018 (as it happened, this was an over-estimate, though it was still much longer than in 2018), while the estimated length of EAS2020 was a mere 2x longer than EAS2018.
Also, I would expect a longer, more demanding survey to lead to fewer total respondents in the year of the survey itself (and not merely lagged a year), since I think current-year uptake can be influenced by word of mouth and sharing (I imagine people would be less likely to share and recommend others take the survey if they found the survey long or annoying).
That said, as I noted in my original comment, I would expect to see lag effects (the survey being too long reduces response to the next year’s survey) and I might expect these effects to be larger (and to stack if the next year’s survey is itself too long) and this is exactly what we see: we see a moderate did from 2018 to 2019 and then a much larger dip from 2019 to 2020.
“Completion rate” is not entirely straightforward, because we explicitly instruct respondents that the final questions of the survey are especially optional “extra credit” questions and they should feel free to quit the survey before these. We can, however, look at the final questions of the main section of the survey (before the extra credit section) and here we see roughly the predicted pattern: a big drop in those ‘completing’ the main section from 2018 to 2019 followed by a smaller absolute drop 2019 to 2020, even though the percentage of those who started the survey completing the main section actually increased between 2019 and 2020 (which we might expect if some people, who are less inclined to take the survey, were put off taking it).
Another (and I think better way) of examining whether we are simply sampling fewer people or the population has shrunk is comparing numbers for subpopulations of the EA Survey(s) to known population sizes as we did here.
In 2019, we estimated that we sampled around 40% of the ‘highly engaged’ EA population. In 2020, using updated numbers, we estimated that we sampled around 35% of the highly engaged EA population.
If the true EA population had remained the same size 2019-2020 and we just sampled 35% rather than 40% overall, we would expect the number of EAs sampled in 2020 to decrease from 2513 to 2199 (which is pretty close to the 2166 we actually sampled).
But, as noted, we believe that we usually sample high and low engagement EAs at different rates (sampling relatively fewer less engaged EAs). And, if our sampling rate reduces overall, (a priori) I would expect this to hold up better among highly engaged EAs than among less engaged EAs (who may be less motivated to take the survey and more responsive to the survey becoming too onerous to complete).
The total number of highly engaged EAs in our sample this year was similar/slightly higher than 2019, implying that population slightly increased in size (as I would expect). We don’t have any good proxies for the size of the less engaged EA population (this becomes harder and harder as we consider the progressively larger populations of progressively less engaged EAs), but I would guess that we probably experienced a yet larger reduction in sampling rate for this population, and that the true size of the population of less engaged EAs has probably increased (I would probably look to things like the EA newsletter, subscribers to 80,000 Hours mailing lists and so on for proxies for that group, but Aaron/people at 80K may disagree).
If the sampling rate of highly engaged EAs has gone down from 40% to 35%, but the number of them was the same, that would imply 14% growth.
You then say:
So the total growth should be 14% + growth in highly engaged EAs.
Could you give me the exact figure?
926 highly engaged in EAS2019, 933 in EAS2020.
Of course, much of this growth in the number of highly engaged EAs is likely due to EAs becoming more engaged, rather than there becoming more EAs. As it happens, EAS2020 had more 4′s but fewer 5′s, which I think can plausibly be explained by the general reduction in rate of people sampled, mentioned above, but a number of 1-3s moving into the 4 category and fewer 4s moving into the 5 category (which is more stringent, e.g. EA org employee, group leader etc.).