This is a very different way of interpreting the study results than any of the writeups I’ve seen. Edge’s Report, Mercy for Animals, and Animal Charity Evaluators all conclude that there was basically no effect.
From your results it still would be reasonable to conclude that there’s “no effect” since the p-value is >0.05, but the p-value is low enough that I would give a follow-up study a reasonably high chance of getting a statistically significant result.
Also: it looks like you’re using a one-sided t-test to get your p-value. I don’t know much about significance testing but wouldn’t it be better to use a two-sided t-test? My understanding is that one-sided tests are sort of cheating by making your p-value half of what it really should be.
it looks like you’re using a one-sided t-test to get your p-value.
I agree that a two-sided test would be the right thing to use here, and p-value calculations aren’t something I fully understand. Is this calculation one-sided or two-sided?
I can’t tell what’s being done in that calculation.
I’m getting a p-value of 0.108 from a Pearson chi-square test (with cell values 55, 809; 78, 856). A chi-square test and a two-tailed t-test should give very similar results with these data, so I agree with Michael that it looks like your p=0.053 comes from a one-tailed test.
From your results it still would be reasonable to conclude that there’s “no effect” since the p-value is >0.05, but the p-value is low enough that I would give a follow-up study a reasonably high chance of getting a statistically significant result.
Also: it looks like you’re using a one-sided t-test to get your p-value. I don’t know much about significance testing but wouldn’t it be better to use a two-sided t-test? My understanding is that one-sided tests are sort of cheating by making your p-value half of what it really should be.
I agree that a two-sided test would be the right thing to use here, and p-value calculations aren’t something I fully understand. Is this calculation one-sided or two-sided?
It looks like the NORMDIST function on your sheet is taking the integral from 0 to
z_score
, which is one-sided. A two-sided test would takeI can’t tell what’s being done in that calculation.
I’m getting a p-value of 0.108 from a Pearson chi-square test (with cell values 55, 809; 78, 856). A chi-square test and a two-tailed t-test should give very similar results with these data, so I agree with Michael that it looks like your p=0.053 comes from a one-tailed test.
Yes, you’re right. Sorry! I redid it computationally and also got 0.108. Post updated.