Hmm, do you maybe mean “based on a real effect” when you say significant? Because we already now that 10 of the 55 tests came out significant, so I don’t understand why we would want to calculate the probability of these results being significant. I was calculating the probability of seeing the 10 significant differences that we saw, assuming all the differences we observed are not based on real effects but on random variation, or basically
p(observing differences in the comparisons that are so high that they the t-test with a 5% threshold says ‘significant’ ten out of 55 times | the differences we saw are all just based on random variation in the data).
In case you find this confusing, that is totally on me. I find signicance testing very unintuitive and maybe shouldn’t even have tried to explain it. :’) Just in case, chapter 11 in Doing Bayesian Data Analysis introduces the topic from a Bayesian perspective and was really useful for me.
It seems like you are calculating the chance that NONE of these results are significant, not the chance that MOST of them ARE (?)
Hmm, do you maybe mean “based on a real effect” when you say significant? Because we already now that 10 of the 55 tests came out significant, so I don’t understand why we would want to calculate the probability of these results being significant. I was calculating the probability of seeing the 10 significant differences that we saw, assuming all the differences we observed are not based on real effects but on random variation, or basically
p(observing differences in the comparisons that are so high that they the t-test with a 5% threshold says ‘significant’ ten out of 55 times | the differences we saw are all just based on random variation in the data).
In case you find this confusing, that is totally on me. I find signicance testing very unintuitive and maybe shouldn’t even have tried to explain it. :’) Just in case, chapter 11 in Doing Bayesian Data Analysis introduces the topic from a Bayesian perspective and was really useful for me.
I think I understand what you are doing, and disagree with it being a way of meaningfully addressing my concern.