The difference between matching and challenge grants was not statistically significant, actually. More generally, that study’s evidence is suggestive at best; it was underpowered (couldn’t have distinguished a 30% increase in donations from noise) and didn’t correct for multiple (12 in the field, 10 in the lab) hypothesis tests. They also mis-described what a p-value means, which doesn’t directly invalidate their results but makes me pretty generally worried.
The difference between matching and challenge grants was not statistically significant, actually. More generally, that study’s evidence is suggestive at best; it was underpowered (couldn’t have distinguished a 30% increase in donations from noise) and didn’t correct for multiple (12 in the field, 10 in the lab) hypothesis tests. They also mis-described what a p-value means, which doesn’t directly invalidate their results but makes me pretty generally worried.