Hi Gregory, Thank you for helping try to establish these probabilities. I am not sure I follow the math (Iâm not used to doing these calculations). Could you explain how you calculated it? Thanks again!
If you use a two tailed test and find a positive effect with p<0.05 itâs <0.025 likely youâd get a positive effect that big by chance. If you donât understand that then you should look up two tailed tests.
OK, I will. I donât have your input data, nor the assumptions on which you based your analysis to apply the two-tailed test. These are necessary to understand your results.
Hi Ryan, I need to know what input data and assumptions he used to be able to verify/âreplicate/âinterpret his math. Without this information, I cannot comment further. Thanks!
You could cast about for various relevant base-rates (âWhat is the chance of any given proposed conjecture in medical science being true?â âWhat is the chance of a given medical trial giving a positive result?â). Crisp data on these questions are hard to find, but the proportion for either is comfortably less than even. (Maybe ~5% for the first, ~20% for the second).
From something like this one can make further adjustments based on the particular circumstances, which are generally in the adverse direction:
Typical trials have more than n=6 non-consecutive case series behind them, and so this should be less likely to replicate than the typical member of this class.
(Particularly, heterodox theories of pathogenesis tend to do worse, and on cursory search I can find a alternative theories of Crohnâs which seem about as facially plausible as this).
The wild theory also imposes a penalty: even if the minimal prediction doesnât demand the wider âmalasezzia causes it etc.â, that the hypothesis is generated through these means is a further cost.
Thereâs also information I have from medical training which speaks against this (i.e. if antifungals had such dramatic effects as proposed, it probably would have risen to attention somewhat sooner).
All the second order things I noted in my first comment.
As Ryan has explained, standard significance testing puts a floor of 2.5% of a (false) positive result in any trial even if the true effect is zero. There is some chance the ground truth really is that itraconazole cures Crohnâs (given some evidence of TNFa downstream effects, background knowledge of fungal microbiota disregulation, and the very slender case series), which gives it a small boost above this, although this in itself is somewhat discounted by the limited power of the proposed study (i.e. even if Itraconazole works, the study might miss it).
Hi Gregory, Thanks for the detailed answer. Iâm still not clear on how the numbers quoted above (0.005, 3%, 2.5%) were calculated, nor how they affect the probability of Samuel et al 2010 replicating successfully. It is worthwhile to break down the problem in two parts:
(I) Does Samuel et al 2010 give us any information to support the hypothesis that Crohnâs might be cured by itraconazole? If so, how much?
(II) How large does an RCT need to be to properly test this hypothesis?
Answering these two questions is essential to determine if Samuel et al 2010 should be replicated or not (obviously with proper controls this time). This is what I am trying to determine with this forum post: should we raise ~500K$ to replicate it or not? What is the expected return on giving for this experiment?
~3% (Standard significance testing means thereâs a 2.5% chance of a false positive result favouring the treatment group under the null).
Hi Gregory, Thank you for helping try to establish these probabilities. I am not sure I follow the math (Iâm not used to doing these calculations). Could you explain how you calculated it? Thanks again!
If you use a two tailed test and find a positive effect with p<0.05 itâs <0.025 likely youâd get a positive effect that big by chance. If you donât understand that then you should look up two tailed tests.
OK, I will. I donât have your input data, nor the assumptions on which you based your analysis to apply the two-tailed test. These are necessary to understand your results.
Heâs just saying he thinks thereâs a 0.005 chance of detecting a real effect.
Hi Ryan, I need to know what input data and assumptions he used to be able to verify/âreplicate/âinterpret his math. Without this information, I cannot comment further. Thanks!
In hope but little expectation:
You could cast about for various relevant base-rates (âWhat is the chance of any given proposed conjecture in medical science being true?â âWhat is the chance of a given medical trial giving a positive result?â). Crisp data on these questions are hard to find, but the proportion for either is comfortably less than even. (Maybe ~5% for the first, ~20% for the second).
From something like this one can make further adjustments based on the particular circumstances, which are generally in the adverse direction:
Typical trials have more than n=6 non-consecutive case series behind them, and so this should be less likely to replicate than the typical member of this class.
(Particularly, heterodox theories of pathogenesis tend to do worse, and on cursory search I can find a alternative theories of Crohnâs which seem about as facially plausible as this).
The wild theory also imposes a penalty: even if the minimal prediction doesnât demand the wider âmalasezzia causes it etc.â, that the hypothesis is generated through these means is a further cost.
Thereâs also information I have from medical training which speaks against this (i.e. if antifungals had such dramatic effects as proposed, it probably would have risen to attention somewhat sooner).
All the second order things I noted in my first comment.
As Ryan has explained, standard significance testing puts a floor of 2.5% of a (false) positive result in any trial even if the true effect is zero. There is some chance the ground truth really is that itraconazole cures Crohnâs (given some evidence of TNFa downstream effects, background knowledge of fungal microbiota disregulation, and the very slender case series), which gives it a small boost above this, although this in itself is somewhat discounted by the limited power of the proposed study (i.e. even if Itraconazole works, the study might miss it).
Hi Gregory, Thanks for the detailed answer. Iâm still not clear on how the numbers quoted above (0.005, 3%, 2.5%) were calculated, nor how they affect the probability of Samuel et al 2010 replicating successfully. It is worthwhile to break down the problem in two parts:
(I) Does Samuel et al 2010 give us any information to support the hypothesis that Crohnâs might be cured by itraconazole? If so, how much?
(II) How large does an RCT need to be to properly test this hypothesis?
Answering these two questions is essential to determine if Samuel et al 2010 should be replicated or not (obviously with proper controls this time). This is what I am trying to determine with this forum post: should we raise ~500K$ to replicate it or not? What is the expected return on giving for this experiment?