1) Generally my probability mass is skewed to the lower ends of the intervals Iâm noting. Thus the 70% band is more with multiple caveats rather than just one (e.g. a bit likeâas Scott describes itâKetamine: only really useful for depression, and even then generally modest effects even as second line therapy). Likewise the 3% is mostly âaround SSRIs and maybe slightly betterâ, with subpercentile mass as the dramatic breakthrough I think you have in mind.
2) Re. updates: There wasnât a huge update on reading the studies (not that I claim to have examined them closely), because I was at least dimly aware since medical school of psychedelics having some promise in mental health.
Although this was before I appreciated the importance of being quantitative, I imagine I would have given higher estimates back then, with the difference mainly accounted for by my appreciation of how treacherous replication has proven in both medicine and psychology.
Seeing that at least some of the studies were conducted reasonably given their limitations has attenuated this hit, but I had mostly priced this in as I expected to see this (i.e. I wasnât expecting to see the body of psychedelic work was obviously junk science etc.).
3) Aside: Givewellâs view doesnât appear to be â1-2% that deworming effects are realâ, but:
The â1-2% chanceâ doesnât mean that we think that thereâs a 98-99% chance that deworming programs have no effect at all, but that we think itâs appropriate to use a 1-2% multiplier compared to the impact found in the original trials â this could be thought of as assigning some chance that deworming programs have no impact, and some chance that the impact exists but will be smaller than was measured in those trials.
I.e. Their central estimate prices across a range of âno effectâ âmodest effectâ âas good as the index study advertisedâ, but weighted towards the lower end.
One could argue whether, if applied to psychedelics, whether the discount factor they suggest should be higher or lower than this (multiple studies would probably push to a more generous discount factors, but an emphasis on quality might point to more pessimistic ones, as the Kremer index study has I think a stronger methodologyâand a lot more vettingâthan the work noted here). But even something like a discount of ~0.1 would make a lot of the results noted above considerably less exciting (e.g. The Calhart-Harris effect size drops to d~0.3, which is good but puts it back into the ranges seen with existing interventions like CBD).
VoI is distinct from this best guess (analogously, a further deworming RCT to reduce uncertainty may have higher or lower value than âexploitingâ based on current uncertainty), but Iâd return to my prior remarks to suggest the likelihood of ending up with something â(roughly) as good as initial results advertiseâ is low/ânegligible enough not to make it a good EA buy.
4) Further aside: Given the OP was about psychedelics generally (inc advocacy and research) rather than the particular points on whether confirmatory research was a good idea, Iâd take other (counter-) arguments addressed more generally than this to be in scope.
1) Generally my probability mass is skewed to the lower ends of the intervals Iâm noting. Thus the 70% band is more with multiple caveats rather than just one (e.g. a bit likeâas Scott describes itâKetamine: only really useful for depression, and even then generally modest effects even as second line therapy). Likewise the 3% is mostly âaround SSRIs and maybe slightly betterâ, with subpercentile mass as the dramatic breakthrough I think you have in mind.
2) Re. updates: There wasnât a huge update on reading the studies (not that I claim to have examined them closely), because I was at least dimly aware since medical school of psychedelics having some promise in mental health.
Although this was before I appreciated the importance of being quantitative, I imagine I would have given higher estimates back then, with the difference mainly accounted for by my appreciation of how treacherous replication has proven in both medicine and psychology.
Seeing that at least some of the studies were conducted reasonably given their limitations has attenuated this hit, but I had mostly priced this in as I expected to see this (i.e. I wasnât expecting to see the body of psychedelic work was obviously junk science etc.).
3) Aside: Givewellâs view doesnât appear to be â1-2% that deworming effects are realâ, but:
I.e. Their central estimate prices across a range of âno effectâ âmodest effectâ âas good as the index study advertisedâ, but weighted towards the lower end.
One could argue whether, if applied to psychedelics, whether the discount factor they suggest should be higher or lower than this (multiple studies would probably push to a more generous discount factors, but an emphasis on quality might point to more pessimistic ones, as the Kremer index study has I think a stronger methodologyâand a lot more vettingâthan the work noted here). But even something like a discount of ~0.1 would make a lot of the results noted above considerably less exciting (e.g. The Calhart-Harris effect size drops to d~0.3, which is good but puts it back into the ranges seen with existing interventions like CBD).
VoI is distinct from this best guess (analogously, a further deworming RCT to reduce uncertainty may have higher or lower value than âexploitingâ based on current uncertainty), but Iâd return to my prior remarks to suggest the likelihood of ending up with something â(roughly) as good as initial results advertiseâ is low/ânegligible enough not to make it a good EA buy.
4) Further aside: Given the OP was about psychedelics generally (inc advocacy and research) rather than the particular points on whether confirmatory research was a good idea, Iâd take other (counter-) arguments addressed more generally than this to be in scope.