I think the results of this study are reason to think that animal advocacy via online advertising in this context is less effective than I previously thought it to be. This is because this study suggests that it’s unlikely the effects of online advertising in this context are above a threshold which I previously assigned some probability to and I have now lessened the probability that I put on an effect like this in light of the findings of this study.
Which threshold was that and how did you arrive at that conclusion? I don’t really know one way or another yet, but upgrading or downgrading confidence seems premature without concrete numbers.
-
As a result I would direct slightly less resources to online advertising in this context relative to other techniques then I would have prior to being aware of the results of this study.
It seems unfair to deallocate money from online ads where studies are potentially inconclusive to areas where studies don’t exist, unless you have strong pre-existing reasons to distinguish those interventions as higher potential.
-
Would it be better to do a pre-analysis plan for future studies?
We pre-registered the methodology. It would have been nicer to also pre-register the survey questions and the data analysis plan. However, I was working with other people and limited time, so I didn’t manage to make that happen.
-
Would it be better to do pre treatment/intervention and post treatment/intervention data collection rather than just post treatment/intervention data collection for future studies? By this I mean something like a baseline survey and an endline survey which seemed to be used in a lot of social science RCTs.
I think it is too unclear how the baseline survey could be effectively collected without adding another large source of declining participation.
-
Was it worth using Edge Research to analyze the data for this study?
No.
-
Will external bodies like Edge Research do data analysis for future MFA studies?
I don’t know, but likely not, especially now that MFA has hired more in-house statistical expertise.
However, Edge Research in particular will not be used again.
-
Why was the study so low powered?
The response rate conversion of people getting the follow-up survey from the people who got the original treatment was lower than expected and lower than what had been determined via piloting the study.
Was it originally thought that online ads were more effective
No.
or perhaps the study’s power was constrained by inadequate funding?
No. The study was fully funded and was instead limited by how large of a Facebook ad campaign MFA could logistically manage at the time. I was told that if we wanted to increase the sample size further we’d likely have to either (a) relax the requirement that we target females age 13-25 or (b) go into foreign language ads.
-
“Edge Research then “weighted” the data so the respondents from each group were identical in gender, geography, and age.” I am not totally sure what this means and it seems important. It would be great if someone could please explain more about what the “weighting” process entails.
I also am clueless about this weighting process and think it should be disclosed. Though I think ACE re-analyzed the data without any weighting and came to pretty similar conclusions.
-
I had a quick look but it isn’t initially clear to me how to reach 684 participants in the control group and 749 people in the experiment group. My guess is that all of the participants who were 12 years or younger, didn’t fully complete the survey and were not female were excluded from the final analysis. Is that right? Or was there some additional exclusion criteria?
I think they just excluded all people who didn’t answer every question.
Which threshold was that and how did you arrive at that conclusion? I don’t really know one way or another yet, but upgrading or downgrading confidence seems premature without concrete numbers.
The threshold was a 10% difference in animal product consumption between the experiment and the control group. I arrived at this conclusion because I thought that there was some chance that these ads would cause the experiment group to report a 10% or more decrease in animal product consumption when compared to the control group. Since the study didn’t detect a difference at this level I assign a lower probability to a change of this magnitude being present than I did previously.
A predicted change of 10% or more might have been overly optimistic and I didn’t have a great sense of what I thought the effectiveness of online ads would be prior to this experiment. The ads were targeted at what was thought to be the most receptive demographic and those who click on these ads seem particularly predisposed to decreasing their animal product consumption. You’re right though, upgrading or downgrading confidence might be premature without concrete numbers.
I think there are some other reasons for why I seem to be updating in the negative direction for the effectiveness of online ads. These other reasons are:
I feel that that my lower bound for the effectiveness of online ads also moved in the negative direction. I previously assigned next to no probability that the ads caused an increase in animal product consumption. However the results seem to suggest that there may have been an increase in animal product consumption in the experiment group. So I have increased the probability that I put on that outcome.
ACE also seems to be updating in the negative direction.
I did a very rough and simple calculation in this spreadsheet using that the experiment group would have 1% of people reduce their animal product consumption by 10%, 1% of people convert to vegetarianism and .1% of people convert to veganism. I don’t put too much weight on this because I did do these calculations after I had already somewhat committed to negatively updating in this post which may have induced a bias towards producing negative results. Still, this suggests that something like my best guess was systematically too positive across the board.
On this last bullet point I wonder if there is a way that we can do a bayesian analysis of the data. If we were to set our prior and then inform it with the results from this experiment. It would be very interesting to see if this would cause us to update.
It seems unfair to deallocate money from online ads where studies are potentially inconclusive to areas where studies don’t exist, unless you have strong pre-existing reasons to distinguish those interventions as higher potential.
I think we agree that if the study is inconclusive it shouldn’t cause us to change the allocation of resources to online ads. However, I think if the study causes updates in the negative direction or positive direction about the effectiveness of online ads this is reason to change the allocation of resources to online ads. I currently interpret the study as causing me to update in the negative direction for online ads. I think this means that other interventions appear relatively more effective in comparison to online advertising compared to my prior views of their effectiveness in comparison to online advertising. This seems to be reason to allocate some increased amount of resources to these other interventions and some decreased amount of resources to online ads.
Which threshold was that and how did you arrive at that conclusion? I don’t really know one way or another yet, but upgrading or downgrading confidence seems premature without concrete numbers.
-
It seems unfair to deallocate money from online ads where studies are potentially inconclusive to areas where studies don’t exist, unless you have strong pre-existing reasons to distinguish those interventions as higher potential.
-
We pre-registered the methodology. It would have been nicer to also pre-register the survey questions and the data analysis plan. However, I was working with other people and limited time, so I didn’t manage to make that happen.
-
I think it is too unclear how the baseline survey could be effectively collected without adding another large source of declining participation.
-
No.
-
I don’t know, but likely not, especially now that MFA has hired more in-house statistical expertise.
However, Edge Research in particular will not be used again.
-
The response rate conversion of people getting the follow-up survey from the people who got the original treatment was lower than expected and lower than what had been determined via piloting the study.
No.
No. The study was fully funded and was instead limited by how large of a Facebook ad campaign MFA could logistically manage at the time. I was told that if we wanted to increase the sample size further we’d likely have to either (a) relax the requirement that we target females age 13-25 or (b) go into foreign language ads.
-
I also am clueless about this weighting process and think it should be disclosed. Though I think ACE re-analyzed the data without any weighting and came to pretty similar conclusions.
-
I think they just excluded all people who didn’t answer every question.
The threshold was a 10% difference in animal product consumption between the experiment and the control group. I arrived at this conclusion because I thought that there was some chance that these ads would cause the experiment group to report a 10% or more decrease in animal product consumption when compared to the control group. Since the study didn’t detect a difference at this level I assign a lower probability to a change of this magnitude being present than I did previously.
A predicted change of 10% or more might have been overly optimistic and I didn’t have a great sense of what I thought the effectiveness of online ads would be prior to this experiment. The ads were targeted at what was thought to be the most receptive demographic and those who click on these ads seem particularly predisposed to decreasing their animal product consumption. You’re right though, upgrading or downgrading confidence might be premature without concrete numbers.
I think there are some other reasons for why I seem to be updating in the negative direction for the effectiveness of online ads. These other reasons are:
I feel that that my lower bound for the effectiveness of online ads also moved in the negative direction. I previously assigned next to no probability that the ads caused an increase in animal product consumption. However the results seem to suggest that there may have been an increase in animal product consumption in the experiment group. So I have increased the probability that I put on that outcome.
ACE also seems to be updating in the negative direction.
I did a very rough and simple calculation in this spreadsheet using that the experiment group would have 1% of people reduce their animal product consumption by 10%, 1% of people convert to vegetarianism and .1% of people convert to veganism. I don’t put too much weight on this because I did do these calculations after I had already somewhat committed to negatively updating in this post which may have induced a bias towards producing negative results. Still, this suggests that something like my best guess was systematically too positive across the board.
On this last bullet point I wonder if there is a way that we can do a bayesian analysis of the data. If we were to set our prior and then inform it with the results from this experiment. It would be very interesting to see if this would cause us to update.
I think we agree that if the study is inconclusive it shouldn’t cause us to change the allocation of resources to online ads. However, I think if the study causes updates in the negative direction or positive direction about the effectiveness of online ads this is reason to change the allocation of resources to online ads. I currently interpret the study as causing me to update in the negative direction for online ads. I think this means that other interventions appear relatively more effective in comparison to online advertising compared to my prior views of their effectiveness in comparison to online advertising. This seems to be reason to allocate some increased amount of resources to these other interventions and some decreased amount of resources to online ads.