I think that Point 1 will be difficult to test in this way. What you want to do sounds a bit like a regression discontinuity analysis, but (as I understand it) there isn’t really a sharp time point for when you started promoting EA more; the translations/meetings etc. increased steadily since Oct 2018, right? I think this will make it harder to see the effect during the first year that you are scaling up outreach (particularly if compared by month, as there is probably seasonal variation in both donation and outreach). Brazil has also had a fairly distinct set of news worthy events (i.e. election and major political change, arrest of two former presidents during ongoing corruption scandals, amazon fires, etc.) over the same time period you increased outreach. If these events influence donation behaviour, then comparisons to other countries might not be particularly relevant (and it further complicates your monthly comparison). I think a better way to try and observe a quantitative effect would be if you compare the total donations for three years: pre-Oct 2018, Oct 2018-Oct 2019, post-Oct 2019 (provided you keep your level of outreach similar for the next year, and are patient). Aggregating over year will remove the seasonal effect of donations and some of the effect of current events, and if this shows an increase for 2019-2020, then you could (cautiously) look at comparing the monthly donation behaviour (three years of data will be better to compensate for monthly variation).
At this point, I think tracking your impact more subjectively by using questionnaires and interviews would produce more useful information. Not sure if charities would link their donors to you (maybe getting the contact of Brazilians who report donating in the EA survey would be more likely), but you could also try adding a annual questionnaire link to your newsletter/facebook/site like 80,000 hours does. I’d specifically try to ask people who made their first donations, or who increased their donations, this year what motivated them to do so.
Hi Ramiro.
I think that Point 1 will be difficult to test in this way. What you want to do sounds a bit like a regression discontinuity analysis, but (as I understand it) there isn’t really a sharp time point for when you started promoting EA more; the translations/meetings etc. increased steadily since Oct 2018, right? I think this will make it harder to see the effect during the first year that you are scaling up outreach (particularly if compared by month, as there is probably seasonal variation in both donation and outreach). Brazil has also had a fairly distinct set of news worthy events (i.e. election and major political change, arrest of two former presidents during ongoing corruption scandals, amazon fires, etc.) over the same time period you increased outreach. If these events influence donation behaviour, then comparisons to other countries might not be particularly relevant (and it further complicates your monthly comparison). I think a better way to try and observe a quantitative effect would be if you compare the total donations for three years: pre-Oct 2018, Oct 2018-Oct 2019, post-Oct 2019 (provided you keep your level of outreach similar for the next year, and are patient). Aggregating over year will remove the seasonal effect of donations and some of the effect of current events, and if this shows an increase for 2019-2020, then you could (cautiously) look at comparing the monthly donation behaviour (three years of data will be better to compensate for monthly variation).
At this point, I think tracking your impact more subjectively by using questionnaires and interviews would produce more useful information. Not sure if charities would link their donors to you (maybe getting the contact of Brazilians who report donating in the EA survey would be more likely), but you could also try adding a annual questionnaire link to your newsletter/facebook/site like 80,000 hours does. I’d specifically try to ask people who made their first donations, or who increased their donations, this year what motivated them to do so.