Assessing the impact of Brazilian donors and EA community
We’re thinking about testing if our actions for promoting EA in this year (translations, meetings, networking...) have led to an observable increase in donations from Brazil—particularly outside the group of more “engaged” members. Even if we haven’t observed an increase in high-quality engagement (such as GWWC pledges), we do see an increase in some “cheaper signals”, such as the number of Facebook group members and the amount of donations to AMF (which, curiously, are concentrated in basically two metropolitan areas—Sao Paulo and Porto Alegre; I know there are some EAs living in Minas and in the North, but currently I’m not aware of any donation coming from Rio and Brasilia, despite them being high-income metropolitan area). We’d like to test if that’s a coincidence.
I would appreciate any suggestion/help on that. I think it would demand more than EA survey data. First, we thought about requesting to EA charities data about the amount of donations:
1.1 from Brazil between Oct 23th, 2018 and Oct 23th, 2019 (controlling for month) with the amount of donations from the previous year;
1.2 from similar countries (I’m not sure which countries we should pick: Argentina, Chile, Mexico, S. Africa, Portugal?...China?), in the same periods – to check if any of them presented a similar increase/decrease.
Second, I wonder if we could get in touch with at least some identified donors and ask them how they came to the decision of donating. Possibly, tracking people using the names they provided to those websites might be considered too invasive, but I wonder if the organization itself could send an e-mail inviting them to get in touch with us.
I think that Point 1 will be difficult to test in this way. What you want to do sounds a bit like a regression discontinuity analysis, but (as I understand it) there isn’t really a sharp time point for when you started promoting EA more; the translations/meetings etc. increased steadily since Oct 2018, right? I think this will make it harder to see the effect during the first year that you are scaling up outreach (particularly if compared by month, as there is probably seasonal variation in both donation and outreach). Brazil has also had a fairly distinct set of news worthy events (i.e. election and major political change, arrest of two former presidents during ongoing corruption scandals, amazon fires, etc.) over the same time period you increased outreach. If these events influence donation behaviour, then comparisons to other countries might not be particularly relevant (and it further complicates your monthly comparison). I think a better way to try and observe a quantitative effect would be if you compare the total donations for three years: pre-Oct 2018, Oct 2018-Oct 2019, post-Oct 2019 (provided you keep your level of outreach similar for the next year, and are patient). Aggregating over year will remove the seasonal effect of donations and some of the effect of current events, and if this shows an increase for 2019-2020, then you could (cautiously) look at comparing the monthly donation behaviour (three years of data will be better to compensate for monthly variation).
At this point, I think tracking your impact more subjectively by using questionnaires and interviews would produce more useful information. Not sure if charities would link their donors to you (maybe getting the contact of Brazilians who report donating in the EA survey would be more likely), but you could also try adding a annual questionnaire link to your newsletter/facebook/site like 80,000 hours does. I’d specifically try to ask people who made their first donations, or who increased their donations, this year what motivated them to do so.
Assessing the impact of Brazilian donors and EA community
We’re thinking about testing if our actions for promoting EA in this year (translations, meetings, networking...) have led to an observable increase in donations from Brazil—particularly outside the group of more “engaged” members. Even if we haven’t observed an increase in high-quality engagement (such as GWWC pledges), we do see an increase in some “cheaper signals”, such as the number of Facebook group members and the amount of donations to AMF (which, curiously, are concentrated in basically two metropolitan areas—Sao Paulo and Porto Alegre; I know there are some EAs living in Minas and in the North, but currently I’m not aware of any donation coming from Rio and Brasilia, despite them being high-income metropolitan area). We’d like to test if that’s a coincidence.
I would appreciate any suggestion/help on that. I think it would demand more than EA survey data. First, we thought about requesting to EA charities data about the amount of donations:
1.1 from Brazil between Oct 23th, 2018 and Oct 23th, 2019 (controlling for month) with the amount of donations from the previous year;
1.2 from similar countries (I’m not sure which countries we should pick: Argentina, Chile, Mexico, S. Africa, Portugal?...China?), in the same periods – to check if any of them presented a similar increase/decrease.
Second, I wonder if we could get in touch with at least some identified donors and ask them how they came to the decision of donating. Possibly, tracking people using the names they provided to those websites might be considered too invasive, but I wonder if the organization itself could send an e-mail inviting them to get in touch with us.
Hi Ramiro.
I think that Point 1 will be difficult to test in this way. What you want to do sounds a bit like a regression discontinuity analysis, but (as I understand it) there isn’t really a sharp time point for when you started promoting EA more; the translations/meetings etc. increased steadily since Oct 2018, right? I think this will make it harder to see the effect during the first year that you are scaling up outreach (particularly if compared by month, as there is probably seasonal variation in both donation and outreach). Brazil has also had a fairly distinct set of news worthy events (i.e. election and major political change, arrest of two former presidents during ongoing corruption scandals, amazon fires, etc.) over the same time period you increased outreach. If these events influence donation behaviour, then comparisons to other countries might not be particularly relevant (and it further complicates your monthly comparison). I think a better way to try and observe a quantitative effect would be if you compare the total donations for three years: pre-Oct 2018, Oct 2018-Oct 2019, post-Oct 2019 (provided you keep your level of outreach similar for the next year, and are patient). Aggregating over year will remove the seasonal effect of donations and some of the effect of current events, and if this shows an increase for 2019-2020, then you could (cautiously) look at comparing the monthly donation behaviour (three years of data will be better to compensate for monthly variation).
At this point, I think tracking your impact more subjectively by using questionnaires and interviews would produce more useful information. Not sure if charities would link their donors to you (maybe getting the contact of Brazilians who report donating in the EA survey would be more likely), but you could also try adding a annual questionnaire link to your newsletter/facebook/site like 80,000 hours does. I’d specifically try to ask people who made their first donations, or who increased their donations, this year what motivated them to do so.