I strong-downvoted this post. I had hoped the reasons why would be obvious. Alas not.
Scientific (in)credibility
The comments so far have mainly focused on the cost-effectiveness calculation. Yet it is the science itself that is replete with red flags: from grandiose free-wheeling, to misreporting cited results, to gross medical and scientific misunderstanding. [As background: I am a doctor who has published on the genetics of inflammatory bowel disease]
Several examples before I succumbed:
Samuel et al. 2010 is a retrospective database review of 6 patients treated with itraconazole for histoplasmosis in Crohnâs Disease (CD) (N.B. Observational, not controlled, and as a letter, editor- rather than peer-reviewed). It did not âreport it cured patients with CD by clearing fungus from the gutâ: the authorsâ own (appropriately tentativeâunlike the OP) conjecture was any therapeutic effect was mediated by immunomodulatory effects of azole drugs downstream of TNF-a. It emphatically didnât âsuggest oral itraconazole may be effective against Malassezia in the gutâ (as claimed in the linked websiteâs FAQ) as the presence or subsequent elimination of Malassezia was never assessedânor was Malassezia mentioned.
Crohnâs disease is not a spondyloarthritis! (and neither is psoriasis, ulcerative colitis, or acute anterior uveitis). As the name suggests, spondyloarthritides are arthritides (i.e. diseases principally of jointsâthe âspondyloâ prefix points to joints between vertebrae); Crohnâs a disease of the GI tract. Crohnâs can be associated with a spondyloarthritis (enteropathic spondyloarthritis). As the word âassociatedâ suggests, these are not one and the same: only a minority of those with Crohnâs develop joint sequelae. (cf. Standardlistsofspondyloarthridesânote Crohnâs simpliciter isnât on them).
Chronic inflammation isnât a symptom (âspondoyloarthritideâ or otherwise), and symptoms (rather than diseases) are only cured in the colloquial use of the term.
However one parses â[P]roving beyond all doubt that Crohnâs disease is caused by this fungus will very likely lead to a cure for all spondyloarthritide symptoms using antifungal drugs.â (âMerelyâ relieving all back pain from spondyloarthritides? Relieving all symptoms that arise from the set of (correctly defined) spondyloarthritides? Curing all spondyloarthritides? Curing (and/âor relieving all symptoms) from the authorâs grab bag of symptoms/âdiseases which include CD, Ulcerative Collitis, Ankylosing spondylitis, Psoriasis and chronic back pain?) The antecedent (one n=40 therapeutic study wonât prove Malassezia causes Crohnâs, especially with a competing immunomodulatory mechanism already proposed); the consequent (anti-fungal drugs as some autoimmune disease panacea of uncertain scope); and the implication (even if Malasezzia is proven to cause Crohnâs, the likelihood of this result (and therapy) generalising is basically nil) are all absurd.
The âI love details!â page notes at then end âThese findings satisfy Kochâs postulates for disease causation, albeit scattered across several related diseases.â Which demonstrate the author doesnât understand Kochâs postulates: you canât âmix and matchâ across diseases, and the postulates need to be satisfied in sequence (i.e. you find the microorganism only present in cases of the disease (1), culture it (2), induce the disease in a healthy individual with such a culture (3), and extract the organism again from such individuals (4)).
The work reported in that page, here, and elsewhere also directly contradict Kochâs first postulate. Malasezzia is not found in abundance in cases of disease (pick any of them) and not in healthy individuals (postulate 1): the author himself states Malasezzia is ubiquitous across individuals, diseased or not (and this ubiquity is cited as why this genus is being proposed in the first place).
Intermezzo
Iâd also rather not know how much has been spent on this so far. Whatever it is, investing another half a million dollars is profoundly ill-advised (putting the money in a pile and burning it is mildly preferable, even when one factors in climate change impacts). At least an order of magnitude cheaper is buying the time of someone who works in Crohnâs to offer their assessment. I doubt it would be less scathing than mine.
Meta moaning
Most EAs had the good judgement to avoid the terrible mistake of a medical degree. One of the few downsides of so doing is (usually) not possessing the background knowledge to appraise something like this. As a community, we might worry about our collective understanding being led astray without the happy accident of someone with specialised knowledge (yet atrocious time-management and prioritisation skills among manifold other relevant personal failings) happening onto the right threads.
Have no fear: I have some handy advice/âdespairing pleas:
Medical science isnât completely civilizationally inadequate, and thus projects that resort to being pitched directly to inexpert funders have a pretty poor base rate (cf. DRACO)
Although these are imperfect, if the person behind the project doesnât have credentials in a relevant field (bioinformatics rather than gastroenterology, say), and/âor a fairly slender relevant publication record, and scant/âno interest from recognised experts, these are also adverse indicators. (Remember the nobel-prize winner endorsed Vit C megadosing?)
It can be hard to set the right incredulity prior: we all want take advantage of our risk neutrality to chase hits, but not all upsides that vindicate a low likelihood are credible. A rule-of-thumb I commend is 10^-(3+n(miracles)). So when someone suggests they have discovered the key mechanism of action (and consequent fix) for Crohnâs disease, and ulcerative colitis, and ankylosing spondylitis, and reactive arthritis, and psoriasis, and psoriatic arthritis, and acute anterior uveitis, and oligoarthritis, and multiple sclerosis, and rheumatoid arthritis, and systemic lupus erythematosus, and prostate cancer,andbenign prostatichyperplasia, and chronic back pain (n~14), there may be some cause for concern.
Spot-checking bits of the write-up can be a great âsniff testâ, especially in key areas where one isnât sure of oneâs ground (âWell, the fermi seems reasonable, but I wonder what this extra-sensory perception thing is all aboutâ).
Post value tends to be multiplicative (e.g. the antecedent of âIf we have a cure for Crohnâs, how good would it be?â may be the crucial consideration), and so its key to have an to develop an understanding across the key topics. Otherwise one risks conversational bikeshedding. Worse, there could be Sokal-hoax-esque effects where nonsense can end up well-received (say, moderately upvoted) provided it sends the right signals on non-substantive metrics like style, approach, sentiment, etc.
I see these aspects of epistemic culture as an important team sport, with âamateurâ participation encouraged (for my part, implored). I had hoped when I clicked the âdownvoteâ arrow for a few seconds I could leave this to fade in obscurity thereafter. When instead I find it being both upvoted and discussed like it has been, I become worried that it might actually attract resources from other EAs who might mistakenly take conversation thus-far to represent the balance of reason, and detract from EAâs reputation with those who recognise it does not (cf. âThe scientific revolution for altruismâ aspiration). So I feel I have to am to write something more comprehensive. This took a lot longer than a few seconds, although fortunately my time is essentially worthless. Next time we may not be so lucky.
I mostly agree with this, but I think itâs also wrong in a couple of places.
Crohnâs disease is not a spondyloarthritis! (and neither is psoriasis, ulcerative colitis, or acute anterior uveitis). As the name suggests, spondyloarthritides are arthritides (i.e. diseases principally of jointsâthe âspondyloâ prefix points to joints between vertebrae); Crohnâs a disease of the GI tract.
I think this is just restating the hypothesis, that Crohnâs shares (most of) its pathophysiology with the spondyloarthritides⊠Which is a well-known open possibility. The incidence of Crohns is >10% in people with AS and vice versa. They share heredity, HLA-B27. Apparently 2â3 of those with AS also have silent gut signs [1].
Also, I think the following is off the mark:
Although these are imperfect, if the person behind the project doesnât have credentials in a relevant field (bioinformatics rather than gastroenterology, say), and/âor a fairly slender relevant publication record, and scant/âno interest from recognised experts, these are also adverse indicators. (Remember the nobel-prize winner endorsed Vit C megadosing?)
Note that the author did manage to co-author his latest piece with an ophthalmologist/ârheumatologist with a professorship in inflammation research and 20k cites.
Overall, the parts of the objection that I agree most with are i) that it seems very unlikely that one or two fungi would be implicated with all of these 14 various diseases, and that treating the fungus would cure the inflammatory disease (rather than the fungus just acting as an initial trigger), and ii) that there are mistakes, especially semantic ones, and especially on malassezia.org (as opposed to in the papers), with some of the medical science.
The interesting questions seems to me to be whether an overconfident-seeming author could nonetheless be correct about the minimal prediction that some antifungals would work well in at least Crohnâs disease. I donât yet see why this is <1% likely.
Hi Gregory, Thanks for the detailed response. I understand where you are coming from: if tables were turned, I would have posted a similar comment. Iâd be happy go over the science in greater detail with you; perhaps we can start another thread to cover this, as I expect our science discussion to be very long, detailed and technical. Right now, we have an important question to answer: is it worth spending 500K$ USD in an attempt to replicate Samuel et al 2010 with more patients and proper controls?
The 500K$ USD figure is from a detailed budget produced by a credible university-affiliated clinical research team eager to start this study. I reviewed this budget with them, and it is reasonable.
Zeke estimates the direct financial upside of a successful replication to be about 33B$/âyear. This is a 66000:1 ratio (33B/â500K = 66000). We need to assign probabilities to the following explanations of Samuel et al 2010âs results:
1. Their results are correct: itraconazole cures Crohnâs. 2. Their results are a fluke: itraconazole isnât affecting Crohnâs symptoms, and natural waxing and waning of symptoms made it look like itraconazole cures Crohnâs.
Itraconazole is a cheap, widely available off-patent broad-spectrum antifungal drug. The main immunological signature of Crohnâs disease are antibodies against conserved fungal sugars (mannan, beta-glucan, and chitin). By principle of parsimony, this means that Crohnâs patientsâ immune systems are likely fighting a fungus which is the root cause of Crohnâs disease. There are a number of other possible explanations for the above observations, but these are more complex and difficult to prove.
(A) What are the odds that Samuel et al 2010âs results will replicate? X (B) At what odds is this replication project a good candidate for EA funding? Y (C) If X > Y and X is low, can funding agencies and foundations tolerate the risk of failure, or must we find these funds in a less non-conventional manner?
I am getting a much better understanding of Y with the help of people on this forum, thanks to Seke and Ryan (I have no experience in doing these estimates). I was hoping to get a better idea of the value of X too. In most situations, people round-down the value of X to zero before starting their analysis. This means they consider further effort evaluating X or Y to be largely futile (in Bayesian terms, prior probabilities of zero cannot be changed by further analysis). EA folks are used to dealing with low X values, so I thought theyâd be less likely to round down to zero
If X < Y, then I will move on to other things. If X > Y, then I will do all I can to fund this study, as this is likely the highest-impact charitable project available to me.
Iâd very much appreciate it if Gregory, Ryan, Seke, Aaron could help me quantify X and Y. Other diseases listed here can somewhat decrease Y, but calculating by how much is complex, so letâs stick to Crohnâs for now. I included other diseases in this post because they were the main focus of my research for six years, and they might well have the same fungal etiology as Crohnâs disease. I realize that despite this prior research *strengthening* the case a fungal etiology in Crohnâs, many people instinctively *reduce* the prior probability of any of this being correct due to the unusually large scope of this project.
A cheaper alternative (also by about an order of magnitude) is to do a hospital record study where you look at subsequent Crohnâs admissions or similar proxies of disease activity in those recently prescribed antifungals versus those who arenât.
I also imagine it would get better data than a poorly powered RCT.
Hi Hauke, Thank you very much for this suggestion. Yes, animal models would be another category 2 option. You might know that Barry Marshall had much trouble developing animals models of Helicobacter pylori-induced gastritis, so this approach is hit-and-miss at best, and it is hard to know ahead of time what the probability of a âhitâ would be. It is also less ethical than the other solutions, and for this reason, Iâd prefer avoiding animal models (if possible).
Hi Gregory, Great suggestion! The main issue with this approach is that it seems long-term use of itraconazole is required (>3 months), which rarely occurs in practice. Most on-label uses of itraconazole are for much shorter periods, which is one reason why Samuel et al 2010 was such an exception: histoplasmosis is only prevalent in the mid-West, and requires a very long course of itraconazole.
A second problem is that once treatment is discontinued, Crohnâs symptoms seem to return after a few months (again per Samuel et al 2010). This is very much like dandruff (caused by the fungus Malassezia): once antifungal shampoos are discontinued, Malassezia return, and so does dandruff! So weâd have to be able to test using the medical database if these Crohnâs patients got a flare or not during the treatment period (as compared to properly selected controlsâgetting comparable/âunbiased controls using this methods is not trivial).
In addition, Samuel et al 2010 was very well positioned to detect the effect of itraconazole, because they stopped giving their patients immunosuppressantsâso they were expecting severe flares during treatment. This is not expected to occur in most cases from medical databases.
Finally, I donât think medical database studies like this can be used to change medical practice. Would the FDA allow a new indication without an RCT? I doubt it. So running a database could not reach the stated impact.
How many patients do you think we would need in a RCT to have sufficient power? The researchers I am working with think itra=20, placebo=20 would be sufficient. I donât have the expertise to evaluate this. Samuel et al 2010 noticed a marked effect on 5 patients, although there were no controls, so they were judging this using their clinical experience. FWIW, the last author of Samuel et al 2010 is one of the top Crohnâs researchers in the world.
X = odds that Samuel et al 2010âs results will replicate (range 0 â 1).
Category 1 options: studies which can bring Xâs value close to 1.
(1a) A well powered RCT testing itraconazole in Crohnâs (success = curing Crohnâs).
Category 2 options: cheaper studies which can increase X, but not bring it close enough to 1 to change clinical practice. However, they would raise awareness that Crohnâs might be caused by a fungus, and thus might be cured by itraconazole. Hopefully someone will do (1a) based on the results of these category 2 options.
(2a) Test Samuel et al 2010 by using a larger medical database than that available at the Mayo Clinic in 2010 (ideally in the mid-West where histoplasmosis is endemic).
(2b) Antibodies against Malassezia are associated with psoriasis (Squiquera et al 1994; Liang et al 2003). We could try replicating these studies in Crohnâs disease.
(2c) In psoriasis, white blood cells release interferon gamma when exposed to Malassezia antigens (Kanda et al 2002), likely because T cells are specifically targeting Malassezia on the skin. We could replicate this study in Crohnâs disease.
Note that (2c) will likely be successful because vedolizumab is known to cause psoriasis in ~10% of Crohnâs patients by sending T cells from the gut to the skin (Tadbiri et al 2018).
The idea of doing an intermediate piece of work is so one can abandon the project if it is negative whilst having spent less than 500k. Even independent of the adverse indicators I note above, the prior on case series finding replicating out in RCT is very low.
Another cheap option would be talking to the original investigators. They may have reasons why they havenât followed this finding up themselves.
I attempted to contact them, but they did not reply. These are top Crohnâs researchers, and must be very solicited from all sides, so their lack of response is expected.
(2b) (2c) (2d) are being run right now by different groups. I donât know how long it will take for them to publish (best guess ~1-2 years).
What numerical value do you assign to the probability of replication of Samuel et al 2010 (variable X)?
Hi Gregory, Thank you for helping try to establish these probabilities. I am not sure I follow the math (Iâm not used to doing these calculations). Could you explain how you calculated it? Thanks again!
If you use a two tailed test and find a positive effect with p<0.05 itâs <0.025 likely youâd get a positive effect that big by chance. If you donât understand that then you should look up two tailed tests.
OK, I will. I donât have your input data, nor the assumptions on which you based your analysis to apply the two-tailed test. These are necessary to understand your results.
Hi Ryan, I need to know what input data and assumptions he used to be able to verify/âreplicate/âinterpret his math. Without this information, I cannot comment further. Thanks!
You could cast about for various relevant base-rates (âWhat is the chance of any given proposed conjecture in medical science being true?â âWhat is the chance of a given medical trial giving a positive result?â). Crisp data on these questions are hard to find, but the proportion for either is comfortably less than even. (Maybe ~5% for the first, ~20% for the second).
From something like this one can make further adjustments based on the particular circumstances, which are generally in the adverse direction:
Typical trials have more than n=6 non-consecutive case series behind them, and so this should be less likely to replicate than the typical member of this class.
(Particularly, heterodox theories of pathogenesis tend to do worse, and on cursory search I can find a alternative theories of Crohnâs which seem about as facially plausible as this).
The wild theory also imposes a penalty: even if the minimal prediction doesnât demand the wider âmalasezzia causes it etc.â, that the hypothesis is generated through these means is a further cost.
Thereâs also information I have from medical training which speaks against this (i.e. if antifungals had such dramatic effects as proposed, it probably would have risen to attention somewhat sooner).
All the second order things I noted in my first comment.
As Ryan has explained, standard significance testing puts a floor of 2.5% of a (false) positive result in any trial even if the true effect is zero. There is some chance the ground truth really is that itraconazole cures Crohnâs (given some evidence of TNFa downstream effects, background knowledge of fungal microbiota disregulation, and the very slender case series), which gives it a small boost above this, although this in itself is somewhat discounted by the limited power of the proposed study (i.e. even if Itraconazole works, the study might miss it).
Hi Gregory, Thanks for the detailed answer. Iâm still not clear on how the numbers quoted above (0.005, 3%, 2.5%) were calculated, nor how they affect the probability of Samuel et al 2010 replicating successfully. It is worthwhile to break down the problem in two parts:
(I) Does Samuel et al 2010 give us any information to support the hypothesis that Crohnâs might be cured by itraconazole? If so, how much?
(II) How large does an RCT need to be to properly test this hypothesis?
Answering these two questions is essential to determine if Samuel et al 2010 should be replicated or not (obviously with proper controls this time). This is what I am trying to determine with this forum post: should we raise ~500K$ to replicate it or not? What is the expected return on giving for this experiment?
>Zeke estimates the direct financial upside of a successful replication to be about 33B$/âyear. This is a 66000:1 ratio (33B/â500K = 66000).
This is not directly relevant, because the money is being saved by other people and governments, who are not normally using their money very well. EAsâ money is much more valuable as it is used much more efficiently than Western people and governments usually do. NB: this is also the reason why EA should generally be considered funders of last resort.
If the study has a 0.5% (??? I have no idea) chance of leading to global approval and effective treatment then itâs 35k QALY in expectation per my estimate which means a point estimate of $14/âQALY. iirc, thatâs comparable to global poverty interventions but at a much lower robustness of evidence, some other top EA efforts with a similar degree of robustness will presumably have a much higher EV. Of course the other diseases you can work on may be much worse causes.
Also that $33B comes from a study on the impact of the disease. Just because you replicate well doesnât mean the treatment truly works, and is approved globally, etc. Hence the 0.5% number being very low.
Hi Zeke, Thanks for the clarification and the estimate for Y. If I understand correctly:
(1) Minimum success probability for project viability is ~0.5% (Y=0.5%)
(2) Upside following success is 33B$*10 years = 330B$ (per your earlier estimate, this needs to be adjusted for many different reasons, both up and down, but these adjustments are beyond my capabilities).
(3) Cost is 500K$.
(4) Expected ROI is = (330B$ * 0.5%) /â 500K$ = 3300.
So this means if you find a 100$ bill on the sidewalk and giving it away to someone else statistically gives them ~300K$, you will keep it, but if it statistically gives them 400K$ you will give it away. Is that right?
I strong-downvoted this post. I had hoped the reasons why would be obvious. Alas not.
Scientific (in)credibility
The comments so far have mainly focused on the cost-effectiveness calculation. Yet it is the science itself that is replete with red flags: from grandiose free-wheeling, to misreporting cited results, to gross medical and scientific misunderstanding. [As background: I am a doctor who has published on the genetics of inflammatory bowel disease]
Several examples before I succumbed:
Samuel et al. 2010 is a retrospective database review of 6 patients treated with itraconazole for histoplasmosis in Crohnâs Disease (CD) (N.B. Observational, not controlled, and as a letter, editor- rather than peer-reviewed). It did not âreport it cured patients with CD by clearing fungus from the gutâ: the authorsâ own (appropriately tentativeâunlike the OP) conjecture was any therapeutic effect was mediated by immunomodulatory effects of azole drugs downstream of TNF-a. It emphatically didnât âsuggest oral itraconazole may be effective against Malassezia in the gutâ (as claimed in the linked websiteâs FAQ) as the presence or subsequent elimination of Malassezia was never assessedânor was Malassezia mentioned.
Crohnâs disease is not a spondyloarthritis! (and neither is psoriasis, ulcerative colitis, or acute anterior uveitis). As the name suggests, spondyloarthritides are arthritides (i.e. diseases principally of jointsâthe âspondyloâ prefix points to joints between vertebrae); Crohnâs a disease of the GI tract. Crohnâs can be associated with a spondyloarthritis (enteropathic spondyloarthritis). As the word âassociatedâ suggests, these are not one and the same: only a minority of those with Crohnâs develop joint sequelae. (cf. Standard lists of spondyloarthridesânote Crohnâs simpliciter isnât on them).
Chronic inflammation isnât a symptom (âspondoyloarthritideâ or otherwise), and symptoms (rather than diseases) are only cured in the colloquial use of the term.
However one parses â[P]roving beyond all doubt that Crohnâs disease is caused by this fungus will very likely lead to a cure for all spondyloarthritide symptoms using antifungal drugs.â (âMerelyâ relieving all back pain from spondyloarthritides? Relieving all symptoms that arise from the set of (correctly defined) spondyloarthritides? Curing all spondyloarthritides? Curing (and/âor relieving all symptoms) from the authorâs grab bag of symptoms/âdiseases which include CD, Ulcerative Collitis, Ankylosing spondylitis, Psoriasis and chronic back pain?) The antecedent (one n=40 therapeutic study wonât prove Malassezia causes Crohnâs, especially with a competing immunomodulatory mechanism already proposed); the consequent (anti-fungal drugs as some autoimmune disease panacea of uncertain scope); and the implication (even if Malasezzia is proven to cause Crohnâs, the likelihood of this result (and therapy) generalising is basically nil) are all absurd.
The âI love details!â page notes at then end âThese findings satisfy Kochâs postulates for disease causation, albeit scattered across several related diseases.â Which demonstrate the author doesnât understand Kochâs postulates: you canât âmix and matchâ across diseases, and the postulates need to be satisfied in sequence (i.e. you find the microorganism only present in cases of the disease (1), culture it (2), induce the disease in a healthy individual with such a culture (3), and extract the organism again from such individuals (4)).
The work reported in that page, here, and elsewhere also directly contradict Kochâs first postulate. Malasezzia is not found in abundance in cases of disease (pick any of them) and not in healthy individuals (postulate 1): the author himself states Malasezzia is ubiquitous across individuals, diseased or not (and this ubiquity is cited as why this genus is being proposed in the first place).
Intermezzo
Iâd also rather not know how much has been spent on this so far. Whatever it is, investing another half a million dollars is profoundly ill-advised (putting the money in a pile and burning it is mildly preferable, even when one factors in climate change impacts). At least an order of magnitude cheaper is buying the time of someone who works in Crohnâs to offer their assessment. I doubt it would be less scathing than mine.
Meta moaning
Most EAs had the good judgement to avoid the terrible mistake of a medical degree. One of the few downsides of so doing is (usually) not possessing the background knowledge to appraise something like this. As a community, we might worry about our collective understanding being led astray without the happy accident of someone with specialised knowledge (yet atrocious time-management and prioritisation skills among manifold other relevant personal failings) happening onto the right threads.
Have no fear: I have some handy advice/âdespairing pleas:
Medical science isnât completely civilizationally inadequate, and thus projects that resort to being pitched directly to inexpert funders have a pretty poor base rate (cf. DRACO)
Although these are imperfect, if the person behind the project doesnât have credentials in a relevant field (bioinformatics rather than gastroenterology, say), and/âor a fairly slender relevant publication record, and scant/âno interest from recognised experts, these are also adverse indicators. (Remember the nobel-prize winner endorsed Vit C megadosing?)
It can be hard to set the right incredulity prior: we all want take advantage of our risk neutrality to chase hits, but not all upsides that vindicate a low likelihood are credible. A rule-of-thumb I commend is 10^-(3+n(miracles)). So when someone suggests they have discovered the key mechanism of action (and consequent fix) for Crohnâs disease, and ulcerative colitis, and ankylosing spondylitis, and reactive arthritis, and psoriasis, and psoriatic arthritis, and acute anterior uveitis, and oligoarthritis, and multiple sclerosis, and rheumatoid arthritis, and systemic lupus erythematosus, and prostate cancer, and benign prostatic hyperplasia, and chronic back pain (n~14), there may be some cause for concern.
Spot-checking bits of the write-up can be a great âsniff testâ, especially in key areas where one isnât sure of oneâs ground (âWell, the fermi seems reasonable, but I wonder what this extra-sensory perception thing is all aboutâ).
Post value tends to be multiplicative (e.g. the antecedent of âIf we have a cure for Crohnâs, how good would it be?â may be the crucial consideration), and so its key to have an to develop an understanding across the key topics. Otherwise one risks conversational bikeshedding. Worse, there could be Sokal-hoax-esque effects where nonsense can end up well-received (say, moderately upvoted) provided it sends the right signals on non-substantive metrics like style, approach, sentiment, etc.
I see these aspects of epistemic culture as an important team sport, with âamateurâ participation encouraged (for my part, implored). I had hoped when I clicked the âdownvoteâ arrow for a few seconds I could leave this to fade in obscurity thereafter. When instead I find it being both upvoted and discussed like it has been, I become worried that it might actually attract resources from other EAs who might mistakenly take conversation thus-far to represent the balance of reason, and detract from EAâs reputation with those who recognise it does not (cf. âThe scientific revolution for altruismâ aspiration). So I feel I have to am to write something more comprehensive. This took a lot longer than a few seconds, although fortunately my time is essentially worthless. Next time we may not be so lucky.
I mostly agree with this, but I think itâs also wrong in a couple of places.
I think this is just restating the hypothesis, that Crohnâs shares (most of) its pathophysiology with the spondyloarthritides⊠Which is a well-known open possibility. The incidence of Crohns is >10% in people with AS and vice versa. They share heredity, HLA-B27. Apparently 2â3 of those with AS also have silent gut signs [1].
Also, I think the following is off the mark:
Note that the author did manage to co-author his latest piece with an ophthalmologist/ârheumatologist with a professorship in inflammation research and 20k cites.
Overall, the parts of the objection that I agree most with are i) that it seems very unlikely that one or two fungi would be implicated with all of these 14 various diseases, and that treating the fungus would cure the inflammatory disease (rather than the fungus just acting as an initial trigger), and ii) that there are mistakes, especially semantic ones, and especially on malassezia.org (as opposed to in the papers), with some of the medical science.
The interesting questions seems to me to be whether an overconfident-seeming author could nonetheless be correct about the minimal prediction that some antifungals would work well in at least Crohnâs disease. I donât yet see why this is <1% likely.
1. https://ââwww.ncbi.nlm.nih.gov/ââpmc/ââarticles/ââPMC2996322/ââ
2. https://ââwww.ncbi.nlm.nih.gov/ââpubmed/ââ29675414, https://ââen.wikipedia.org/ââwiki/ââJames_T._Rosenbaum
Hi Ryan, Thanks for double checking Sekeâs impact numbers. Could you help me draft the impact phrase, Iâm afraid of getting it wrong again.
Hi Gregory, Thanks for the detailed response. I understand where you are coming from: if tables were turned, I would have posted a similar comment. Iâd be happy go over the science in greater detail with you; perhaps we can start another thread to cover this, as I expect our science discussion to be very long, detailed and technical. Right now, we have an important question to answer: is it worth spending 500K$ USD in an attempt to replicate Samuel et al 2010 with more patients and proper controls?
The 500K$ USD figure is from a detailed budget produced by a credible university-affiliated clinical research team eager to start this study. I reviewed this budget with them, and it is reasonable.
Zeke estimates the direct financial upside of a successful replication to be about 33B$/âyear. This is a 66000:1 ratio (33B/â500K = 66000). We need to assign probabilities to the following explanations of Samuel et al 2010âs results:
1. Their results are correct: itraconazole cures Crohnâs.
2. Their results are a fluke: itraconazole isnât affecting Crohnâs symptoms, and natural waxing and waning of symptoms made it look like itraconazole cures Crohnâs.
Itraconazole is a cheap, widely available off-patent broad-spectrum antifungal drug. The main immunological signature of Crohnâs disease are antibodies against conserved fungal sugars (mannan, beta-glucan, and chitin). By principle of parsimony, this means that Crohnâs patientsâ immune systems are likely fighting a fungus which is the root cause of Crohnâs disease. There are a number of other possible explanations for the above observations, but these are more complex and difficult to prove.
(A) What are the odds that Samuel et al 2010âs results will replicate? X
(B) At what odds is this replication project a good candidate for EA funding? Y
(C) If X > Y and X is low, can funding agencies and foundations tolerate the risk of failure, or must we find these funds in a less non-conventional manner?
I am getting a much better understanding of Y with the help of people on this forum, thanks to Seke and Ryan (I have no experience in doing these estimates). I was hoping to get a better idea of the value of X too. In most situations, people round-down the value of X to zero before starting their analysis. This means they consider further effort evaluating X or Y to be largely futile (in Bayesian terms, prior probabilities of zero cannot be changed by further analysis). EA folks are used to dealing with low X values, so I thought theyâd be less likely to round down to zero
If X < Y, then I will move on to other things. If X > Y, then I will do all I can to fund this study, as this is likely the highest-impact charitable project available to me.
Iâd very much appreciate it if Gregory, Ryan, Seke, Aaron could help me quantify X and Y. Other diseases listed here can somewhat decrease Y, but calculating by how much is complex, so letâs stick to Crohnâs for now. I included other diseases in this post because they were the main focus of my research for six years, and they might well have the same fungal etiology as Crohnâs disease. I realize that despite this prior research *strengthening* the case a fungal etiology in Crohnâs, many people instinctively *reduce* the prior probability of any of this being correct due to the unusually large scope of this project.
A cheaper alternative (also by about an order of magnitude) is to do a hospital record study where you look at subsequent Crohnâs admissions or similar proxies of disease activity in those recently prescribed antifungals versus those who arenât.
I also imagine it would get better data than a poorly powered RCT.
This might be naive and I have only skimmed this thread, but wouldnât using a cheap study using mouse model be best here? Maybe contact the authors of the papers cited in this paper âMouse models of inflammatory bowel disease for investigating mucosal immunity in the intestineâ to collaborate on such a study.
Hi Hauke, Thank you very much for this suggestion. Yes, animal models would be another category 2 option. You might know that Barry Marshall had much trouble developing animals models of Helicobacter pylori-induced gastritis, so this approach is hit-and-miss at best, and it is hard to know ahead of time what the probability of a âhitâ would be. It is also less ethical than the other solutions, and for this reason, Iâd prefer avoiding animal models (if possible).
Hi Gregory, Great suggestion! The main issue with this approach is that it seems long-term use of itraconazole is required (>3 months), which rarely occurs in practice. Most on-label uses of itraconazole are for much shorter periods, which is one reason why Samuel et al 2010 was such an exception: histoplasmosis is only prevalent in the mid-West, and requires a very long course of itraconazole.
A second problem is that once treatment is discontinued, Crohnâs symptoms seem to return after a few months (again per Samuel et al 2010). This is very much like dandruff (caused by the fungus Malassezia): once antifungal shampoos are discontinued, Malassezia return, and so does dandruff! So weâd have to be able to test using the medical database if these Crohnâs patients got a flare or not during the treatment period (as compared to properly selected controlsâgetting comparable/âunbiased controls using this methods is not trivial).
In addition, Samuel et al 2010 was very well positioned to detect the effect of itraconazole, because they stopped giving their patients immunosuppressantsâso they were expecting severe flares during treatment. This is not expected to occur in most cases from medical databases.
Finally, I donât think medical database studies like this can be used to change medical practice. Would the FDA allow a new indication without an RCT? I doubt it. So running a database could not reach the stated impact.
How many patients do you think we would need in a RCT to have sufficient power? The researchers I am working with think itra=20, placebo=20 would be sufficient. I donât have the expertise to evaluate this. Samuel et al 2010 noticed a marked effect on 5 patients, although there were no controls, so they were judging this using their clinical experience. FWIW, the last author of Samuel et al 2010 is one of the top Crohnâs researchers in the world.
Hi Gregory, here are some more options:
X = odds that Samuel et al 2010âs results will replicate (range 0 â 1).
Category 1 options: studies which can bring Xâs value close to 1.
(1a) A well powered RCT testing itraconazole in Crohnâs (success = curing Crohnâs).
Category 2 options: cheaper studies which can increase X, but not bring it close enough to 1 to change clinical practice. However, they would raise awareness that Crohnâs might be caused by a fungus, and thus might be cured by itraconazole. Hopefully someone will do (1a) based on the results of these category 2 options.
(2a) Test Samuel et al 2010 by using a larger medical database than that available at the Mayo Clinic in 2010 (ideally in the mid-West where histoplasmosis is endemic).
(2b) Antibodies against Malassezia are associated with psoriasis (Squiquera et al 1994; Liang et al 2003). We could try replicating these studies in Crohnâs disease.
(2c) In psoriasis, white blood cells release interferon gamma when exposed to Malassezia antigens (Kanda et al 2002), likely because T cells are specifically targeting Malassezia on the skin. We could replicate this study in Crohnâs disease.
(2d) We could replicate Kellermayer et al 2012 or Richard 2018, who found extremely strong associations between Malassezia and IBD.
Note that (2c) will likely be successful because vedolizumab is known to cause psoriasis in ~10% of Crohnâs patients by sending T cells from the gut to the skin (Tadbiri et al 2018).
Other ideas are welcome!
The idea of doing an intermediate piece of work is so one can abandon the project if it is negative whilst having spent less than 500k. Even independent of the adverse indicators I note above, the prior on case series finding replicating out in RCT is very low.
Another cheap option would be talking to the original investigators. They may have reasons why they havenât followed this finding up themselves.
I attempted to contact them, but they did not reply. These are top Crohnâs researchers, and must be very solicited from all sides, so their lack of response is expected.
(2b) (2c) (2d) are being run right now by different groups. I donât know how long it will take for them to publish (best guess ~1-2 years).
What numerical value do you assign to the probability of replication of Samuel et al 2010 (variable X)?
~3% (Standard significance testing means thereâs a 2.5% chance of a false positive result favouring the treatment group under the null).
Hi Gregory, Thank you for helping try to establish these probabilities. I am not sure I follow the math (Iâm not used to doing these calculations). Could you explain how you calculated it? Thanks again!
If you use a two tailed test and find a positive effect with p<0.05 itâs <0.025 likely youâd get a positive effect that big by chance. If you donât understand that then you should look up two tailed tests.
OK, I will. I donât have your input data, nor the assumptions on which you based your analysis to apply the two-tailed test. These are necessary to understand your results.
Heâs just saying he thinks thereâs a 0.005 chance of detecting a real effect.
Hi Ryan, I need to know what input data and assumptions he used to be able to verify/âreplicate/âinterpret his math. Without this information, I cannot comment further. Thanks!
In hope but little expectation:
You could cast about for various relevant base-rates (âWhat is the chance of any given proposed conjecture in medical science being true?â âWhat is the chance of a given medical trial giving a positive result?â). Crisp data on these questions are hard to find, but the proportion for either is comfortably less than even. (Maybe ~5% for the first, ~20% for the second).
From something like this one can make further adjustments based on the particular circumstances, which are generally in the adverse direction:
Typical trials have more than n=6 non-consecutive case series behind them, and so this should be less likely to replicate than the typical member of this class.
(Particularly, heterodox theories of pathogenesis tend to do worse, and on cursory search I can find a alternative theories of Crohnâs which seem about as facially plausible as this).
The wild theory also imposes a penalty: even if the minimal prediction doesnât demand the wider âmalasezzia causes it etc.â, that the hypothesis is generated through these means is a further cost.
Thereâs also information I have from medical training which speaks against this (i.e. if antifungals had such dramatic effects as proposed, it probably would have risen to attention somewhat sooner).
All the second order things I noted in my first comment.
As Ryan has explained, standard significance testing puts a floor of 2.5% of a (false) positive result in any trial even if the true effect is zero. There is some chance the ground truth really is that itraconazole cures Crohnâs (given some evidence of TNFa downstream effects, background knowledge of fungal microbiota disregulation, and the very slender case series), which gives it a small boost above this, although this in itself is somewhat discounted by the limited power of the proposed study (i.e. even if Itraconazole works, the study might miss it).
Hi Gregory, Thanks for the detailed answer. Iâm still not clear on how the numbers quoted above (0.005, 3%, 2.5%) were calculated, nor how they affect the probability of Samuel et al 2010 replicating successfully. It is worthwhile to break down the problem in two parts:
(I) Does Samuel et al 2010 give us any information to support the hypothesis that Crohnâs might be cured by itraconazole? If so, how much?
(II) How large does an RCT need to be to properly test this hypothesis?
Answering these two questions is essential to determine if Samuel et al 2010 should be replicated or not (obviously with proper controls this time). This is what I am trying to determine with this forum post: should we raise ~500K$ to replicate it or not? What is the expected return on giving for this experiment?
>Zeke estimates the direct financial upside of a successful replication to be about 33B$/âyear. This is a 66000:1 ratio (33B/â500K = 66000).
This is not directly relevant, because the money is being saved by other people and governments, who are not normally using their money very well. EAsâ money is much more valuable as it is used much more efficiently than Western people and governments usually do. NB: this is also the reason why EA should generally be considered funders of last resort.
If the study has a 0.5% (??? I have no idea) chance of leading to global approval and effective treatment then itâs 35k QALY in expectation per my estimate which means a point estimate of $14/âQALY. iirc, thatâs comparable to global poverty interventions but at a much lower robustness of evidence, some other top EA efforts with a similar degree of robustness will presumably have a much higher EV. Of course the other diseases you can work on may be much worse causes.
Also that $33B comes from a study on the impact of the disease. Just because you replicate well doesnât mean the treatment truly works, and is approved globally, etc. Hence the 0.5% number being very low.
Hi Zeke, Thanks for the clarification and the estimate for Y. If I understand correctly:
(1) Minimum success probability for project viability is ~0.5% (Y=0.5%)
(2) Upside following success is 33B$*10 years = 330B$ (per your earlier estimate, this needs to be adjusted for many different reasons, both up and down, but these adjustments are beyond my capabilities).
(3) Cost is 500K$.
(4) Expected ROI is = (330B$ * 0.5%) /â 500K$ = 3300.
So this means if you find a 100$ bill on the sidewalk and giving it away to someone else statistically gives them ~300K$, you will keep it, but if it statistically gives them 400K$ you will give it away. Is that right?
Only if this project is assumed to be the best available use of funds. Other things may be better.