A cheaper alternative (also by about an order of magnitude) is to do a hospital record study where you look at subsequent Crohnâs admissions or similar proxies of disease activity in those recently prescribed antifungals versus those who arenât.
I also imagine it would get better data than a poorly powered RCT.
Hi Hauke, Thank you very much for this suggestion. Yes, animal models would be another category 2 option. You might know that Barry Marshall had much trouble developing animals models of Helicobacter pylori-induced gastritis, so this approach is hit-and-miss at best, and it is hard to know ahead of time what the probability of a âhitâ would be. It is also less ethical than the other solutions, and for this reason, Iâd prefer avoiding animal models (if possible).
Hi Gregory, Great suggestion! The main issue with this approach is that it seems long-term use of itraconazole is required (>3 months), which rarely occurs in practice. Most on-label uses of itraconazole are for much shorter periods, which is one reason why Samuel et al 2010 was such an exception: histoplasmosis is only prevalent in the mid-West, and requires a very long course of itraconazole.
A second problem is that once treatment is discontinued, Crohnâs symptoms seem to return after a few months (again per Samuel et al 2010). This is very much like dandruff (caused by the fungus Malassezia): once antifungal shampoos are discontinued, Malassezia return, and so does dandruff! So weâd have to be able to test using the medical database if these Crohnâs patients got a flare or not during the treatment period (as compared to properly selected controlsâgetting comparable/âunbiased controls using this methods is not trivial).
In addition, Samuel et al 2010 was very well positioned to detect the effect of itraconazole, because they stopped giving their patients immunosuppressantsâso they were expecting severe flares during treatment. This is not expected to occur in most cases from medical databases.
Finally, I donât think medical database studies like this can be used to change medical practice. Would the FDA allow a new indication without an RCT? I doubt it. So running a database could not reach the stated impact.
How many patients do you think we would need in a RCT to have sufficient power? The researchers I am working with think itra=20, placebo=20 would be sufficient. I donât have the expertise to evaluate this. Samuel et al 2010 noticed a marked effect on 5 patients, although there were no controls, so they were judging this using their clinical experience. FWIW, the last author of Samuel et al 2010 is one of the top Crohnâs researchers in the world.
X = odds that Samuel et al 2010âs results will replicate (range 0 â 1).
Category 1 options: studies which can bring Xâs value close to 1.
(1a) A well powered RCT testing itraconazole in Crohnâs (success = curing Crohnâs).
Category 2 options: cheaper studies which can increase X, but not bring it close enough to 1 to change clinical practice. However, they would raise awareness that Crohnâs might be caused by a fungus, and thus might be cured by itraconazole. Hopefully someone will do (1a) based on the results of these category 2 options.
(2a) Test Samuel et al 2010 by using a larger medical database than that available at the Mayo Clinic in 2010 (ideally in the mid-West where histoplasmosis is endemic).
(2b) Antibodies against Malassezia are associated with psoriasis (Squiquera et al 1994; Liang et al 2003). We could try replicating these studies in Crohnâs disease.
(2c) In psoriasis, white blood cells release interferon gamma when exposed to Malassezia antigens (Kanda et al 2002), likely because T cells are specifically targeting Malassezia on the skin. We could replicate this study in Crohnâs disease.
Note that (2c) will likely be successful because vedolizumab is known to cause psoriasis in ~10% of Crohnâs patients by sending T cells from the gut to the skin (Tadbiri et al 2018).
The idea of doing an intermediate piece of work is so one can abandon the project if it is negative whilst having spent less than 500k. Even independent of the adverse indicators I note above, the prior on case series finding replicating out in RCT is very low.
Another cheap option would be talking to the original investigators. They may have reasons why they havenât followed this finding up themselves.
I attempted to contact them, but they did not reply. These are top Crohnâs researchers, and must be very solicited from all sides, so their lack of response is expected.
(2b) (2c) (2d) are being run right now by different groups. I donât know how long it will take for them to publish (best guess ~1-2 years).
What numerical value do you assign to the probability of replication of Samuel et al 2010 (variable X)?
Hi Gregory, Thank you for helping try to establish these probabilities. I am not sure I follow the math (Iâm not used to doing these calculations). Could you explain how you calculated it? Thanks again!
If you use a two tailed test and find a positive effect with p<0.05 itâs <0.025 likely youâd get a positive effect that big by chance. If you donât understand that then you should look up two tailed tests.
OK, I will. I donât have your input data, nor the assumptions on which you based your analysis to apply the two-tailed test. These are necessary to understand your results.
Hi Ryan, I need to know what input data and assumptions he used to be able to verify/âreplicate/âinterpret his math. Without this information, I cannot comment further. Thanks!
You could cast about for various relevant base-rates (âWhat is the chance of any given proposed conjecture in medical science being true?â âWhat is the chance of a given medical trial giving a positive result?â). Crisp data on these questions are hard to find, but the proportion for either is comfortably less than even. (Maybe ~5% for the first, ~20% for the second).
From something like this one can make further adjustments based on the particular circumstances, which are generally in the adverse direction:
Typical trials have more than n=6 non-consecutive case series behind them, and so this should be less likely to replicate than the typical member of this class.
(Particularly, heterodox theories of pathogenesis tend to do worse, and on cursory search I can find a alternative theories of Crohnâs which seem about as facially plausible as this).
The wild theory also imposes a penalty: even if the minimal prediction doesnât demand the wider âmalasezzia causes it etc.â, that the hypothesis is generated through these means is a further cost.
Thereâs also information I have from medical training which speaks against this (i.e. if antifungals had such dramatic effects as proposed, it probably would have risen to attention somewhat sooner).
All the second order things I noted in my first comment.
As Ryan has explained, standard significance testing puts a floor of 2.5% of a (false) positive result in any trial even if the true effect is zero. There is some chance the ground truth really is that itraconazole cures Crohnâs (given some evidence of TNFa downstream effects, background knowledge of fungal microbiota disregulation, and the very slender case series), which gives it a small boost above this, although this in itself is somewhat discounted by the limited power of the proposed study (i.e. even if Itraconazole works, the study might miss it).
Hi Gregory, Thanks for the detailed answer. Iâm still not clear on how the numbers quoted above (0.005, 3%, 2.5%) were calculated, nor how they affect the probability of Samuel et al 2010 replicating successfully. It is worthwhile to break down the problem in two parts:
(I) Does Samuel et al 2010 give us any information to support the hypothesis that Crohnâs might be cured by itraconazole? If so, how much?
(II) How large does an RCT need to be to properly test this hypothesis?
Answering these two questions is essential to determine if Samuel et al 2010 should be replicated or not (obviously with proper controls this time). This is what I am trying to determine with this forum post: should we raise ~500K$ to replicate it or not? What is the expected return on giving for this experiment?
A cheaper alternative (also by about an order of magnitude) is to do a hospital record study where you look at subsequent Crohnâs admissions or similar proxies of disease activity in those recently prescribed antifungals versus those who arenât.
I also imagine it would get better data than a poorly powered RCT.
This might be naive and I have only skimmed this thread, but wouldnât using a cheap study using mouse model be best here? Maybe contact the authors of the papers cited in this paper âMouse models of inflammatory bowel disease for investigating mucosal immunity in the intestineâ to collaborate on such a study.
Hi Hauke, Thank you very much for this suggestion. Yes, animal models would be another category 2 option. You might know that Barry Marshall had much trouble developing animals models of Helicobacter pylori-induced gastritis, so this approach is hit-and-miss at best, and it is hard to know ahead of time what the probability of a âhitâ would be. It is also less ethical than the other solutions, and for this reason, Iâd prefer avoiding animal models (if possible).
Hi Gregory, Great suggestion! The main issue with this approach is that it seems long-term use of itraconazole is required (>3 months), which rarely occurs in practice. Most on-label uses of itraconazole are for much shorter periods, which is one reason why Samuel et al 2010 was such an exception: histoplasmosis is only prevalent in the mid-West, and requires a very long course of itraconazole.
A second problem is that once treatment is discontinued, Crohnâs symptoms seem to return after a few months (again per Samuel et al 2010). This is very much like dandruff (caused by the fungus Malassezia): once antifungal shampoos are discontinued, Malassezia return, and so does dandruff! So weâd have to be able to test using the medical database if these Crohnâs patients got a flare or not during the treatment period (as compared to properly selected controlsâgetting comparable/âunbiased controls using this methods is not trivial).
In addition, Samuel et al 2010 was very well positioned to detect the effect of itraconazole, because they stopped giving their patients immunosuppressantsâso they were expecting severe flares during treatment. This is not expected to occur in most cases from medical databases.
Finally, I donât think medical database studies like this can be used to change medical practice. Would the FDA allow a new indication without an RCT? I doubt it. So running a database could not reach the stated impact.
How many patients do you think we would need in a RCT to have sufficient power? The researchers I am working with think itra=20, placebo=20 would be sufficient. I donât have the expertise to evaluate this. Samuel et al 2010 noticed a marked effect on 5 patients, although there were no controls, so they were judging this using their clinical experience. FWIW, the last author of Samuel et al 2010 is one of the top Crohnâs researchers in the world.
Hi Gregory, here are some more options:
X = odds that Samuel et al 2010âs results will replicate (range 0 â 1).
Category 1 options: studies which can bring Xâs value close to 1.
(1a) A well powered RCT testing itraconazole in Crohnâs (success = curing Crohnâs).
Category 2 options: cheaper studies which can increase X, but not bring it close enough to 1 to change clinical practice. However, they would raise awareness that Crohnâs might be caused by a fungus, and thus might be cured by itraconazole. Hopefully someone will do (1a) based on the results of these category 2 options.
(2a) Test Samuel et al 2010 by using a larger medical database than that available at the Mayo Clinic in 2010 (ideally in the mid-West where histoplasmosis is endemic).
(2b) Antibodies against Malassezia are associated with psoriasis (Squiquera et al 1994; Liang et al 2003). We could try replicating these studies in Crohnâs disease.
(2c) In psoriasis, white blood cells release interferon gamma when exposed to Malassezia antigens (Kanda et al 2002), likely because T cells are specifically targeting Malassezia on the skin. We could replicate this study in Crohnâs disease.
(2d) We could replicate Kellermayer et al 2012 or Richard 2018, who found extremely strong associations between Malassezia and IBD.
Note that (2c) will likely be successful because vedolizumab is known to cause psoriasis in ~10% of Crohnâs patients by sending T cells from the gut to the skin (Tadbiri et al 2018).
Other ideas are welcome!
The idea of doing an intermediate piece of work is so one can abandon the project if it is negative whilst having spent less than 500k. Even independent of the adverse indicators I note above, the prior on case series finding replicating out in RCT is very low.
Another cheap option would be talking to the original investigators. They may have reasons why they havenât followed this finding up themselves.
I attempted to contact them, but they did not reply. These are top Crohnâs researchers, and must be very solicited from all sides, so their lack of response is expected.
(2b) (2c) (2d) are being run right now by different groups. I donât know how long it will take for them to publish (best guess ~1-2 years).
What numerical value do you assign to the probability of replication of Samuel et al 2010 (variable X)?
~3% (Standard significance testing means thereâs a 2.5% chance of a false positive result favouring the treatment group under the null).
Hi Gregory, Thank you for helping try to establish these probabilities. I am not sure I follow the math (Iâm not used to doing these calculations). Could you explain how you calculated it? Thanks again!
If you use a two tailed test and find a positive effect with p<0.05 itâs <0.025 likely youâd get a positive effect that big by chance. If you donât understand that then you should look up two tailed tests.
OK, I will. I donât have your input data, nor the assumptions on which you based your analysis to apply the two-tailed test. These are necessary to understand your results.
Heâs just saying he thinks thereâs a 0.005 chance of detecting a real effect.
Hi Ryan, I need to know what input data and assumptions he used to be able to verify/âreplicate/âinterpret his math. Without this information, I cannot comment further. Thanks!
In hope but little expectation:
You could cast about for various relevant base-rates (âWhat is the chance of any given proposed conjecture in medical science being true?â âWhat is the chance of a given medical trial giving a positive result?â). Crisp data on these questions are hard to find, but the proportion for either is comfortably less than even. (Maybe ~5% for the first, ~20% for the second).
From something like this one can make further adjustments based on the particular circumstances, which are generally in the adverse direction:
Typical trials have more than n=6 non-consecutive case series behind them, and so this should be less likely to replicate than the typical member of this class.
(Particularly, heterodox theories of pathogenesis tend to do worse, and on cursory search I can find a alternative theories of Crohnâs which seem about as facially plausible as this).
The wild theory also imposes a penalty: even if the minimal prediction doesnât demand the wider âmalasezzia causes it etc.â, that the hypothesis is generated through these means is a further cost.
Thereâs also information I have from medical training which speaks against this (i.e. if antifungals had such dramatic effects as proposed, it probably would have risen to attention somewhat sooner).
All the second order things I noted in my first comment.
As Ryan has explained, standard significance testing puts a floor of 2.5% of a (false) positive result in any trial even if the true effect is zero. There is some chance the ground truth really is that itraconazole cures Crohnâs (given some evidence of TNFa downstream effects, background knowledge of fungal microbiota disregulation, and the very slender case series), which gives it a small boost above this, although this in itself is somewhat discounted by the limited power of the proposed study (i.e. even if Itraconazole works, the study might miss it).
Hi Gregory, Thanks for the detailed answer. Iâm still not clear on how the numbers quoted above (0.005, 3%, 2.5%) were calculated, nor how they affect the probability of Samuel et al 2010 replicating successfully. It is worthwhile to break down the problem in two parts:
(I) Does Samuel et al 2010 give us any information to support the hypothesis that Crohnâs might be cured by itraconazole? If so, how much?
(II) How large does an RCT need to be to properly test this hypothesis?
Answering these two questions is essential to determine if Samuel et al 2010 should be replicated or not (obviously with proper controls this time). This is what I am trying to determine with this forum post: should we raise ~500K$ to replicate it or not? What is the expected return on giving for this experiment?