X = odds that Samuel et al 2010’s results will replicate (range 0 − 1).
Category 1 options: studies which can bring X’s value close to 1.
(1a) A well powered RCT testing itraconazole in Crohn’s (success = curing Crohn’s).
Category 2 options: cheaper studies which can increase X, but not bring it close enough to 1 to change clinical practice. However, they would raise awareness that Crohn’s might be caused by a fungus, and thus might be cured by itraconazole. Hopefully someone will do (1a) based on the results of these category 2 options.
(2a) Test Samuel et al 2010 by using a larger medical database than that available at the Mayo Clinic in 2010 (ideally in the mid-West where histoplasmosis is endemic).
(2b) Antibodies against Malassezia are associated with psoriasis (Squiquera et al 1994; Liang et al 2003). We could try replicating these studies in Crohn’s disease.
(2c) In psoriasis, white blood cells release interferon gamma when exposed to Malassezia antigens (Kanda et al 2002), likely because T cells are specifically targeting Malassezia on the skin. We could replicate this study in Crohn’s disease.
Note that (2c) will likely be successful because vedolizumab is known to cause psoriasis in ~10% of Crohn’s patients by sending T cells from the gut to the skin (Tadbiri et al 2018).
The idea of doing an intermediate piece of work is so one can abandon the project if it is negative whilst having spent less than 500k. Even independent of the adverse indicators I note above, the prior on case series finding replicating out in RCT is very low.
Another cheap option would be talking to the original investigators. They may have reasons why they haven’t followed this finding up themselves.
I attempted to contact them, but they did not reply. These are top Crohn’s researchers, and must be very solicited from all sides, so their lack of response is expected.
(2b) (2c) (2d) are being run right now by different groups. I don’t know how long it will take for them to publish (best guess ~1-2 years).
What numerical value do you assign to the probability of replication of Samuel et al 2010 (variable X)?
Hi Gregory, Thank you for helping try to establish these probabilities. I am not sure I follow the math (I’m not used to doing these calculations). Could you explain how you calculated it? Thanks again!
If you use a two tailed test and find a positive effect with p<0.05 it’s <0.025 likely you’d get a positive effect that big by chance. If you don’t understand that then you should look up two tailed tests.
OK, I will. I don’t have your input data, nor the assumptions on which you based your analysis to apply the two-tailed test. These are necessary to understand your results.
Hi Ryan, I need to know what input data and assumptions he used to be able to verify/replicate/interpret his math. Without this information, I cannot comment further. Thanks!
You could cast about for various relevant base-rates (“What is the chance of any given proposed conjecture in medical science being true?” “What is the chance of a given medical trial giving a positive result?”). Crisp data on these questions are hard to find, but the proportion for either is comfortably less than even. (Maybe ~5% for the first, ~20% for the second).
From something like this one can make further adjustments based on the particular circumstances, which are generally in the adverse direction:
Typical trials have more than n=6 non-consecutive case series behind them, and so this should be less likely to replicate than the typical member of this class.
(Particularly, heterodox theories of pathogenesis tend to do worse, and on cursory search I can find a alternative theories of Crohn’s which seem about as facially plausible as this).
The wild theory also imposes a penalty: even if the minimal prediction doesn’t demand the wider ‘malasezzia causes it etc.’, that the hypothesis is generated through these means is a further cost.
There’s also information I have from medical training which speaks against this (i.e. if antifungals had such dramatic effects as proposed, it probably would have risen to attention somewhat sooner).
All the second order things I noted in my first comment.
As Ryan has explained, standard significance testing puts a floor of 2.5% of a (false) positive result in any trial even if the true effect is zero. There is some chance the ground truth really is that itraconazole cures Crohn’s (given some evidence of TNFa downstream effects, background knowledge of fungal microbiota disregulation, and the very slender case series), which gives it a small boost above this, although this in itself is somewhat discounted by the limited power of the proposed study (i.e. even if Itraconazole works, the study might miss it).
Hi Gregory, Thanks for the detailed answer. I’m still not clear on how the numbers quoted above (0.005, 3%, 2.5%) were calculated, nor how they affect the probability of Samuel et al 2010 replicating successfully. It is worthwhile to break down the problem in two parts:
(I) Does Samuel et al 2010 give us any information to support the hypothesis that Crohn’s might be cured by itraconazole? If so, how much?
(II) How large does an RCT need to be to properly test this hypothesis?
Answering these two questions is essential to determine if Samuel et al 2010 should be replicated or not (obviously with proper controls this time). This is what I am trying to determine with this forum post: should we raise ~500K$ to replicate it or not? What is the expected return on giving for this experiment?
Hi Gregory, here are some more options:
X = odds that Samuel et al 2010’s results will replicate (range 0 − 1).
Category 1 options: studies which can bring X’s value close to 1.
(1a) A well powered RCT testing itraconazole in Crohn’s (success = curing Crohn’s).
Category 2 options: cheaper studies which can increase X, but not bring it close enough to 1 to change clinical practice. However, they would raise awareness that Crohn’s might be caused by a fungus, and thus might be cured by itraconazole. Hopefully someone will do (1a) based on the results of these category 2 options.
(2a) Test Samuel et al 2010 by using a larger medical database than that available at the Mayo Clinic in 2010 (ideally in the mid-West where histoplasmosis is endemic).
(2b) Antibodies against Malassezia are associated with psoriasis (Squiquera et al 1994; Liang et al 2003). We could try replicating these studies in Crohn’s disease.
(2c) In psoriasis, white blood cells release interferon gamma when exposed to Malassezia antigens (Kanda et al 2002), likely because T cells are specifically targeting Malassezia on the skin. We could replicate this study in Crohn’s disease.
(2d) We could replicate Kellermayer et al 2012 or Richard 2018, who found extremely strong associations between Malassezia and IBD.
Note that (2c) will likely be successful because vedolizumab is known to cause psoriasis in ~10% of Crohn’s patients by sending T cells from the gut to the skin (Tadbiri et al 2018).
Other ideas are welcome!
The idea of doing an intermediate piece of work is so one can abandon the project if it is negative whilst having spent less than 500k. Even independent of the adverse indicators I note above, the prior on case series finding replicating out in RCT is very low.
Another cheap option would be talking to the original investigators. They may have reasons why they haven’t followed this finding up themselves.
I attempted to contact them, but they did not reply. These are top Crohn’s researchers, and must be very solicited from all sides, so their lack of response is expected.
(2b) (2c) (2d) are being run right now by different groups. I don’t know how long it will take for them to publish (best guess ~1-2 years).
What numerical value do you assign to the probability of replication of Samuel et al 2010 (variable X)?
~3% (Standard significance testing means there’s a 2.5% chance of a false positive result favouring the treatment group under the null).
Hi Gregory, Thank you for helping try to establish these probabilities. I am not sure I follow the math (I’m not used to doing these calculations). Could you explain how you calculated it? Thanks again!
If you use a two tailed test and find a positive effect with p<0.05 it’s <0.025 likely you’d get a positive effect that big by chance. If you don’t understand that then you should look up two tailed tests.
OK, I will. I don’t have your input data, nor the assumptions on which you based your analysis to apply the two-tailed test. These are necessary to understand your results.
He’s just saying he thinks there’s a 0.005 chance of detecting a real effect.
Hi Ryan, I need to know what input data and assumptions he used to be able to verify/replicate/interpret his math. Without this information, I cannot comment further. Thanks!
In hope but little expectation:
You could cast about for various relevant base-rates (“What is the chance of any given proposed conjecture in medical science being true?” “What is the chance of a given medical trial giving a positive result?”). Crisp data on these questions are hard to find, but the proportion for either is comfortably less than even. (Maybe ~5% for the first, ~20% for the second).
From something like this one can make further adjustments based on the particular circumstances, which are generally in the adverse direction:
Typical trials have more than n=6 non-consecutive case series behind them, and so this should be less likely to replicate than the typical member of this class.
(Particularly, heterodox theories of pathogenesis tend to do worse, and on cursory search I can find a alternative theories of Crohn’s which seem about as facially plausible as this).
The wild theory also imposes a penalty: even if the minimal prediction doesn’t demand the wider ‘malasezzia causes it etc.’, that the hypothesis is generated through these means is a further cost.
There’s also information I have from medical training which speaks against this (i.e. if antifungals had such dramatic effects as proposed, it probably would have risen to attention somewhat sooner).
All the second order things I noted in my first comment.
As Ryan has explained, standard significance testing puts a floor of 2.5% of a (false) positive result in any trial even if the true effect is zero. There is some chance the ground truth really is that itraconazole cures Crohn’s (given some evidence of TNFa downstream effects, background knowledge of fungal microbiota disregulation, and the very slender case series), which gives it a small boost above this, although this in itself is somewhat discounted by the limited power of the proposed study (i.e. even if Itraconazole works, the study might miss it).
Hi Gregory, Thanks for the detailed answer. I’m still not clear on how the numbers quoted above (0.005, 3%, 2.5%) were calculated, nor how they affect the probability of Samuel et al 2010 replicating successfully. It is worthwhile to break down the problem in two parts:
(I) Does Samuel et al 2010 give us any information to support the hypothesis that Crohn’s might be cured by itraconazole? If so, how much?
(II) How large does an RCT need to be to properly test this hypothesis?
Answering these two questions is essential to determine if Samuel et al 2010 should be replicated or not (obviously with proper controls this time). This is what I am trying to determine with this forum post: should we raise ~500K$ to replicate it or not? What is the expected return on giving for this experiment?