It is also worth noting that the survey was asking people who identify as EA in 2017 how much they donated in 2015 and 2016. These people weren’t necessarily EAs in 2015 or 2016.
Looking at the raw data of when respondents said that they first became involved in EA, I’m getting that:
7% became EAs in 2017 28% became EAs in 2016 24% became EAs in 2015 41% became EAs in 2014 or earlier
(assuming that everyone who took the “Donations Only” survey became an EA before 2015, and leaving out everyone else who didn’t answer the question about when they became an EA.)
So if we’re looking at donations made in 2015, 35% of the people weren’t EAs then and another 24% had only just become EAs that year. For 2016, 35% of the people weren’t EAs yet at the start of the year and 7% weren’t EAs at the end of the year.
These not-yet-EAs can have a large influence on the median, and to a lesser extent on the percentiles and the mean. They would also tend to create an upward trend in the longitudinal analysis (e.g., if many of the 184 individuals became EAs in 2015).
You’re right there’s a long lag time between asking about donations and the time of the donations… for the most part this is unavoidable, though we’re hoping to time the survey much better in the future (asking only about one year of donations and asking just a month or two after the year is over). This will come with better organization in our team.
In the meantime, it is pretty easy to filter the data accordingly—if you look only at donations made by EAs who stated that they joined on 2014 or before, the median donation is $1280.20 for 2015 and $1500 for 2016.
I agree that asking about 2016 donations in early 2017 is an improvement for this. If future surveys are just going to ask about one year of donations then that’s pretty much all you can do with the timing of the survey.
In the meantime, it is pretty easy to filter the data accordingly—if you look only at donations made by EAs who stated that they joined on 2014 or before, the median donation is $1280.20 for 2015 and $1500 for 2016.
This seems like a better way to do the analyses. I think that the post would be more informative & easier to interpret if all of the analyses used this kind of filter. (For 2016 donations you could also include people who became involved in EA in 2015.)
For example, someone who hears a number for the median non-student donation in 2016 will by default assume that this refers to people who were non-student EAs throughout 2016. If possible, it’s good to give the number which matches the scenario that they’re imagining rather than needing to give caveats about how 35% of the people weren’t EAs yet at the start of 2016. When people hear a non-intuitive analysis with a caveat then they’re fairly likely to either a) forget about the caveat and mistakenly think that the number refers to the thing that they initially assumed that it meant or b) not know what to make of the caveated analysis and therefore not learn anything.
It is also worth noting that the survey was asking people who identify as EA in 2017 how much they donated in 2015 and 2016. These people weren’t necessarily EAs in 2015 or 2016.
Looking at the raw data of when respondents said that they first became involved in EA, I’m getting that:
7% became EAs in 2017
28% became EAs in 2016
24% became EAs in 2015
41% became EAs in 2014 or earlier
(assuming that everyone who took the “Donations Only” survey became an EA before 2015, and leaving out everyone else who didn’t answer the question about when they became an EA.)
So if we’re looking at donations made in 2015, 35% of the people weren’t EAs then and another 24% had only just become EAs that year. For 2016, 35% of the people weren’t EAs yet at the start of the year and 7% weren’t EAs at the end of the year.
(There were similar issues with the 2015 survey.)
These not-yet-EAs can have a large influence on the median, and to a lesser extent on the percentiles and the mean. They would also tend to create an upward trend in the longitudinal analysis (e.g., if many of the 184 individuals became EAs in 2015).
You’re right there’s a long lag time between asking about donations and the time of the donations… for the most part this is unavoidable, though we’re hoping to time the survey much better in the future (asking only about one year of donations and asking just a month or two after the year is over). This will come with better organization in our team.
In the meantime, it is pretty easy to filter the data accordingly—if you look only at donations made by EAs who stated that they joined on 2014 or before, the median donation is $1280.20 for 2015 and $1500 for 2016.
I agree that asking about 2016 donations in early 2017 is an improvement for this. If future surveys are just going to ask about one year of donations then that’s pretty much all you can do with the timing of the survey.
This seems like a better way to do the analyses. I think that the post would be more informative & easier to interpret if all of the analyses used this kind of filter. (For 2016 donations you could also include people who became involved in EA in 2015.)
For example, someone who hears a number for the median non-student donation in 2016 will by default assume that this refers to people who were non-student EAs throughout 2016. If possible, it’s good to give the number which matches the scenario that they’re imagining rather than needing to give caveats about how 35% of the people weren’t EAs yet at the start of 2016. When people hear a non-intuitive analysis with a caveat then they’re fairly likely to either a) forget about the caveat and mistakenly think that the number refers to the thing that they initially assumed that it meant or b) not know what to make of the caveated analysis and therefore not learn anything.
The median 2016 reported donation total of people who joined on 2015 or before was $655.
We’ll talk amongst the team about if we want to update the post or not. Thanks!