Not sure how helpful percentages are given effect sizes and interventions are varied.
Per my OP, Iâd benchmark getting results similar or better to SSRIs (i.e. modestly effective for a few mental illnesses) to be the top 3% ish of what Iâd expect research to confirm. Iâd give 25% for essentially nothing replicating and it going the way of power poses, priming, or other psych dead ends Scott mentions.
The remaining 70% is smeared across much less impressive results (and worth noting SSRIs are hardly a miracle cure): maybe sort-of helpful for one condition, maybe helpful but only for a subset of motivated individuals, etc. etc.
Do you feel like you updated after reading the studies I point to?
e.g. were you initially like âthereâs literally a 0% chance this is realâ and now youâre like âwell, maybe thereâs a 3% chance that psychedelics are an effective treatment & 70% that psychedelics do something but arenât more efficacious than SSRIsâ ?
Do you assign a non-negligible chance to psychedelics meaningfully outperforming current treatments like SSRIs? (â3% for similar to or better than SSRIsâ blurs together the case where psychedelics are just as efficacious as SSRIs and the case where they are massively more efficacious.)
This is important because if thereâs a small, non-negligible chance of a large effect over that of current treatment, investment could still be warranted.
1) Generally my probability mass is skewed to the lower ends of the intervals Iâm noting. Thus the 70% band is more with multiple caveats rather than just one (e.g. a bit likeâas Scott describes itâKetamine: only really useful for depression, and even then generally modest effects even as second line therapy). Likewise the 3% is mostly âaround SSRIs and maybe slightly betterâ, with subpercentile mass as the dramatic breakthrough I think you have in mind.
2) Re. updates: There wasnât a huge update on reading the studies (not that I claim to have examined them closely), because I was at least dimly aware since medical school of psychedelics having some promise in mental health.
Although this was before I appreciated the importance of being quantitative, I imagine I would have given higher estimates back then, with the difference mainly accounted for by my appreciation of how treacherous replication has proven in both medicine and psychology.
Seeing that at least some of the studies were conducted reasonably given their limitations has attenuated this hit, but I had mostly priced this in as I expected to see this (i.e. I wasnât expecting to see the body of psychedelic work was obviously junk science etc.).
3) Aside: Givewellâs view doesnât appear to be â1-2% that deworming effects are realâ, but:
The â1-2% chanceâ doesnât mean that we think that thereâs a 98-99% chance that deworming programs have no effect at all, but that we think itâs appropriate to use a 1-2% multiplier compared to the impact found in the original trials â this could be thought of as assigning some chance that deworming programs have no impact, and some chance that the impact exists but will be smaller than was measured in those trials.
I.e. Their central estimate prices across a range of âno effectâ âmodest effectâ âas good as the index study advertisedâ, but weighted towards the lower end.
One could argue whether, if applied to psychedelics, whether the discount factor they suggest should be higher or lower than this (multiple studies would probably push to a more generous discount factors, but an emphasis on quality might point to more pessimistic ones, as the Kremer index study has I think a stronger methodologyâand a lot more vettingâthan the work noted here). But even something like a discount of ~0.1 would make a lot of the results noted above considerably less exciting (e.g. The Calhart-Harris effect size drops to d~0.3, which is good but puts it back into the ranges seen with existing interventions like CBD).
VoI is distinct from this best guess (analogously, a further deworming RCT to reduce uncertainty may have higher or lower value than âexploitingâ based on current uncertainty), but Iâd return to my prior remarks to suggest the likelihood of ending up with something â(roughly) as good as initial results advertiseâ is low/ânegligible enough not to make it a good EA buy.
4) Further aside: Given the OP was about psychedelics generally (inc advocacy and research) rather than the particular points on whether confirmatory research was a good idea, Iâd take other (counter-) arguments addressed more generally than this to be in scope.
Not sure how helpful percentages are given effect sizes and interventions are varied.
Per my OP, Iâd benchmark getting results similar or better to SSRIs (i.e. modestly effective for a few mental illnesses) to be the top 3% ish of what Iâd expect research to confirm. Iâd give 25% for essentially nothing replicating and it going the way of power poses, priming, or other psych dead ends Scott mentions.
The remaining 70% is smeared across much less impressive results (and worth noting SSRIs are hardly a miracle cure): maybe sort-of helpful for one condition, maybe helpful but only for a subset of motivated individuals, etc. etc.
Do you feel like you updated after reading the studies I point to?
e.g. were you initially like âthereâs literally a 0% chance this is realâ and now youâre like âwell, maybe thereâs a 3% chance that psychedelics are an effective treatment & 70% that psychedelics do something but arenât more efficacious than SSRIsâ ?
I see, thanks.
Do you assign a non-negligible chance to psychedelics meaningfully outperforming current treatments like SSRIs? (â3% for similar to or better than SSRIsâ blurs together the case where psychedelics are just as efficacious as SSRIs and the case where they are massively more efficacious.)
This is important because if thereâs a small, non-negligible chance of a large effect over that of current treatment, investment could still be warranted.
Comparison point: GiveWell has directed tens of millions USD to deworming programs, even though most GiveWell staffers think thereâs only a 1-2% chance that deworming effects are real.
1) Generally my probability mass is skewed to the lower ends of the intervals Iâm noting. Thus the 70% band is more with multiple caveats rather than just one (e.g. a bit likeâas Scott describes itâKetamine: only really useful for depression, and even then generally modest effects even as second line therapy). Likewise the 3% is mostly âaround SSRIs and maybe slightly betterâ, with subpercentile mass as the dramatic breakthrough I think you have in mind.
2) Re. updates: There wasnât a huge update on reading the studies (not that I claim to have examined them closely), because I was at least dimly aware since medical school of psychedelics having some promise in mental health.
Although this was before I appreciated the importance of being quantitative, I imagine I would have given higher estimates back then, with the difference mainly accounted for by my appreciation of how treacherous replication has proven in both medicine and psychology.
Seeing that at least some of the studies were conducted reasonably given their limitations has attenuated this hit, but I had mostly priced this in as I expected to see this (i.e. I wasnât expecting to see the body of psychedelic work was obviously junk science etc.).
3) Aside: Givewellâs view doesnât appear to be â1-2% that deworming effects are realâ, but:
I.e. Their central estimate prices across a range of âno effectâ âmodest effectâ âas good as the index study advertisedâ, but weighted towards the lower end.
One could argue whether, if applied to psychedelics, whether the discount factor they suggest should be higher or lower than this (multiple studies would probably push to a more generous discount factors, but an emphasis on quality might point to more pessimistic ones, as the Kremer index study has I think a stronger methodologyâand a lot more vettingâthan the work noted here). But even something like a discount of ~0.1 would make a lot of the results noted above considerably less exciting (e.g. The Calhart-Harris effect size drops to d~0.3, which is good but puts it back into the ranges seen with existing interventions like CBD).
VoI is distinct from this best guess (analogously, a further deworming RCT to reduce uncertainty may have higher or lower value than âexploitingâ based on current uncertainty), but Iâd return to my prior remarks to suggest the likelihood of ending up with something â(roughly) as good as initial results advertiseâ is low/ânegligible enough not to make it a good EA buy.
4) Further aside: Given the OP was about psychedelics generally (inc advocacy and research) rather than the particular points on whether confirmatory research was a good idea, Iâd take other (counter-) arguments addressed more generally than this to be in scope.