You should not update significantly towards “casual outreach about EA is ineffective”, or “outreach has a very low probability of success” since the study is FAR too underpowered to detect even large effects. For example, if talking about GWWC to likely candidates has a 10% chance of making them take the pledge in the next 15-20 days, and the 14 people who were contacted are exactly representative of the pool of “likely candidates”, then we have a .9^14=23% chance of getting 0 pledges.
Given that it was already unlikely that being put in contact with a GWWC member would have a 10% chance of making them take the pledge, we can now call it very unlikely.
I’m not sure how you’re operationalizing the difference between unlikely and very unlikely, but I think we should not be able to make sizable updates from this data unless the prior is REALLY big.
(You probably already understand this, but other people might read your comment as suggesting something more strongly than you’re actually referring to, and this is a point that I really wanted to clarify anyway because I expect it to be a fairly common mistake)
Roughly: Unsurprising conclusions from experiments with low sample sizes should not change your mind significantly, regardless of what your prior beliefs are.
This is true (mostly) regardless of the size of your prior. If a null result when you have a high prior wouldn’t cause a large update downwards, then a null result on something when you have a low prior shouldn’t cause a large shift downwards either.
[Math with made-up numbers below]
As mentioned earlier:
If your hypothesis is 10%: 23% probability experiment confirms it.
If your hypothesis is 1%: 87% probability experiment is in line with this
5%: 49%
20%: 4.4%
Say your prior belief is that there’s a 70% chance of talking to new people having no effect (or meaningfully close enough to zero that it doesn’t matter), a 25% chance that it has a 1% effect, and a 5% chance that it has a 10% effect.
Then by Bayes’ Theorem, your posterior probability should be:
75.3% chance it has no effect
23.4% chance it has a 1% effect
1.24% chance it has a 10% effect.
If, on the other hand, you originally believed that there’s a 50% chance of it have no effect, and a 50% chance of it having a 10% effect, then your posterior should be:
81.3% chance it has no effect
18.7% chance it has a 10% effect.
Finally, if your prior is that it already has a relatively small effect, this study is far too underpowered to basically make any conclusions at all. For example, if you originally believed that there’s a 70% chance of it having no effect, and a 30% chance of it having a .1% effect, then your posterior should be:
70.3% chance of no effect
29.7% chance of a .1% effect.
This is all assuming ideal conditions.Model uncertainty and uncertainty about the quality of my experiment should only decrease the size of your update, not increase it.
Do you agree here? If so, do you think I should rephrase the original post to make this clearer?
More quick Bayes: Suppose we have a Beta(0.01, 0.32) prior on the proportion of people who will pledge. I choose this prior because it gives a point-estimate of a ~3% chance of pledging, and a probability of ~95% that the chance of pledging is less than 10%, which seems prima facie reasonable.
Updating on your data using a binomial model yields a Beta(0.01, 0.32 + 14) distribution, which gives a point estimate of < 0.1% and a ~99.9% probability that the true chance of pledging is less than 10%.
Given that it was already unlikely that being put in contact with a GWWC member would have a 10% chance of making them take the pledge, we can now call it very unlikely.
I’m not sure how you’re operationalizing the difference between unlikely and very unlikely, but I think we should not be able to make sizable updates from this data unless the prior is REALLY big.
(You probably already understand this, but other people might read your comment as suggesting something more strongly than you’re actually referring to, and this is a point that I really wanted to clarify anyway because I expect it to be a fairly common mistake)
Roughly: Unsurprising conclusions from experiments with low sample sizes should not change your mind significantly, regardless of what your prior beliefs are.
This is true (mostly) regardless of the size of your prior. If a null result when you have a high prior wouldn’t cause a large update downwards, then a null result on something when you have a low prior shouldn’t cause a large shift downwards either.
[Math with made-up numbers below]
As mentioned earlier:
If your hypothesis is 10%: 23% probability experiment confirms it.
If your hypothesis is 1%: 87% probability experiment is in line with this
5%: 49%
20%: 4.4%
Say your prior belief is that there’s a 70% chance of talking to new people having no effect (or meaningfully close enough to zero that it doesn’t matter), a 25% chance that it has a 1% effect, and a 5% chance that it has a 10% effect.
Then by Bayes’ Theorem, your posterior probability should be: 75.3% chance it has no effect
23.4% chance it has a 1% effect
1.24% chance it has a 10% effect.
If, on the other hand, you originally believed that there’s a 50% chance of it have no effect, and a 50% chance of it having a 10% effect, then your posterior should be:
81.3% chance it has no effect
18.7% chance it has a 10% effect.
Finally, if your prior is that it already has a relatively small effect, this study is far too underpowered to basically make any conclusions at all. For example, if you originally believed that there’s a 70% chance of it having no effect, and a 30% chance of it having a .1% effect, then your posterior should be:
70.3% chance of no effect
29.7% chance of a .1% effect.
This is all assuming ideal conditions.Model uncertainty and uncertainty about the quality of my experiment should only decrease the size of your update, not increase it.
Do you agree here? If so, do you think I should rephrase the original post to make this clearer?
More quick Bayes: Suppose we have a Beta(0.01, 0.32) prior on the proportion of people who will pledge. I choose this prior because it gives a point-estimate of a ~3% chance of pledging, and a probability of ~95% that the chance of pledging is less than 10%, which seems prima facie reasonable.
Updating on your data using a binomial model yields a Beta(0.01, 0.32 + 14) distribution, which gives a point estimate of < 0.1% and a ~99.9% probability that the true chance of pledging is less than 10%.
I trust that you can explain Bayes theorem, I’m just adding that we now can be fairly confident that the intervention has less than 10% effectiveness.
Yeah that makes sense!