Because the competition ends tomorrow, I’m curious: Did any of these arguments change your views? Did people say what you expected them to say? Did you get what you wanted out of this exercise? What do you think the next steps are?
Thanks, I intend to write a follow-up post that goes into some detail on questions like these. (Probably will publish in a few weeks as I’m booked up until then.)
Briefly: it looks like only Gregory_Lewis & Carl_Shulman made arguments specifically against EA funding more psychedelic research. (Many people made arguments against psychedelics being an EA cause area in general, but not about funding more research in particular.)
Gregory didn’t close out his argument except to say that he thinks EA shouldn’t fund most kinds of research, including confirmatory research about psychedelics. (In his initial post, he pointed to some reasons why he thinks the results of the initial studies won’t hold up under further scrutiny, but he doesn’t think funding more scrutiny should be an EA priority, and I don’t follow why not.)
Carl pattern-matched psychedelic research to interventions like cold fusion, psychic powers, some parenting interventions, some nutritional / diet interventions, and a few other things. (Interventions which have initial promising results that fail to hold up under more scrutiny.)
Our crux here seems to be that the only way to figure out whether a promising early-stage result is real or not is to do confirmatory research. My prior suggests that funding confirmatory research for psychedelics would be a good use of EA funds, and Carl’s prior is probably that funding research like this wouldn’t be. We haven’t yet sorted out our difference here.
So I still hold the view that funding more psychedelic research would be a good use of EA funds.
Gregory didn’t close out his argument except to say that he thinks EA shouldn’t fund most kinds of research, including confirmatory research about psychedelics. (In his initial post, he pointed to some reasons why he thinks the results of the initial studies won’t hold up under further scrutiny, but he doesn’t think funding more scrutiny should be an EA priority, and I don’t follow why not.)
My views are the same as Carl’s, hence I didn’t make a further reply. (i.e. Low enough base rates imply the yield on chasing replications does not reach the—high—bar for EA).
Not sure how helpful percentages are given effect sizes and interventions are varied.
Per my OP, I’d benchmark getting results similar or better to SSRIs (i.e. modestly effective for a few mental illnesses) to be the top 3% ish of what I’d expect research to confirm. I’d give 25% for essentially nothing replicating and it going the way of power poses, priming, or other psych dead ends Scott mentions.
The remaining 70% is smeared across much less impressive results (and worth noting SSRIs are hardly a miracle cure): maybe sort-of helpful for one condition, maybe helpful but only for a subset of motivated individuals, etc. etc.
Do you feel like you updated after reading the studies I point to?
e.g. were you initially like “there’s literally a 0% chance this is real” and now you’re like “well, maybe there’s a 3% chance that psychedelics are an effective treatment & 70% that psychedelics do something but aren’t more efficacious than SSRIs” ?
Do you assign a non-negligible chance to psychedelics meaningfully outperforming current treatments like SSRIs? (“3% for similar to or better than SSRIs” blurs together the case where psychedelics are just as efficacious as SSRIs and the case where they are massively more efficacious.)
This is important because if there’s a small, non-negligible chance of a large effect over that of current treatment, investment could still be warranted.
1) Generally my probability mass is skewed to the lower ends of the intervals I’m noting. Thus the 70% band is more with multiple caveats rather than just one (e.g. a bit like—as Scott describes it—Ketamine: only really useful for depression, and even then generally modest effects even as second line therapy). Likewise the 3% is mostly ‘around SSRIs and maybe slightly better’, with subpercentile mass as the dramatic breakthrough I think you have in mind.
2) Re. updates: There wasn’t a huge update on reading the studies (not that I claim to have examined them closely), because I was at least dimly aware since medical school of psychedelics having some promise in mental health.
Although this was before I appreciated the importance of being quantitative, I imagine I would have given higher estimates back then, with the difference mainly accounted for by my appreciation of how treacherous replication has proven in both medicine and psychology.
Seeing that at least some of the studies were conducted reasonably given their limitations has attenuated this hit, but I had mostly priced this in as I expected to see this (i.e. I wasn’t expecting to see the body of psychedelic work was obviously junk science etc.).
3) Aside: Givewell’s view doesn’t appear to be “1-2% that deworming effects are real”, but:
The “1-2% chance” doesn’t mean that we think that there’s a 98-99% chance that deworming programs have no effect at all, but that we think it’s appropriate to use a 1-2% multiplier compared to the impact found in the original trials – this could be thought of as assigning some chance that deworming programs have no impact, and some chance that the impact exists but will be smaller than was measured in those trials.
I.e. Their central estimate prices across a range of ‘no effect’ ‘modest effect’ ‘as good as the index study advertised’, but weighted towards the lower end.
One could argue whether, if applied to psychedelics, whether the discount factor they suggest should be higher or lower than this (multiple studies would probably push to a more generous discount factors, but an emphasis on quality might point to more pessimistic ones, as the Kremer index study has I think a stronger methodology—and a lot more vetting—than the work noted here). But even something like a discount of ~0.1 would make a lot of the results noted above considerably less exciting (e.g. The Calhart-Harris effect size drops to d~0.3, which is good but puts it back into the ranges seen with existing interventions like CBD).
VoI is distinct from this best guess (analogously, a further deworming RCT to reduce uncertainty may have higher or lower value than ‘exploiting’ based on current uncertainty), but I’d return to my prior remarks to suggest the likelihood of ending up with something ‘(roughly) as good as initial results advertise’ is low/negligible enough not to make it a good EA buy.
4) Further aside: Given the OP was about psychedelics generally (inc advocacy and research) rather than the particular points on whether confirmatory research was a good idea, I’d take other (counter-) arguments addressed more generally than this to be in scope.
Because the competition ends tomorrow, I’m curious: Did any of these arguments change your views? Did people say what you expected them to say? Did you get what you wanted out of this exercise? What do you think the next steps are?
Thanks, I intend to write a follow-up post that goes into some detail on questions like these. (Probably will publish in a few weeks as I’m booked up until then.)
Briefly: it looks like only Gregory_Lewis & Carl_Shulman made arguments specifically against EA funding more psychedelic research. (Many people made arguments against psychedelics being an EA cause area in general, but not about funding more research in particular.)
Gregory didn’t close out his argument except to say that he thinks EA shouldn’t fund most kinds of research, including confirmatory research about psychedelics. (In his initial post, he pointed to some reasons why he thinks the results of the initial studies won’t hold up under further scrutiny, but he doesn’t think funding more scrutiny should be an EA priority, and I don’t follow why not.)
Carl pattern-matched psychedelic research to interventions like cold fusion, psychic powers, some parenting interventions, some nutritional / diet interventions, and a few other things. (Interventions which have initial promising results that fail to hold up under more scrutiny.)
Our crux here seems to be that the only way to figure out whether a promising early-stage result is real or not is to do confirmatory research. My prior suggests that funding confirmatory research for psychedelics would be a good use of EA funds, and Carl’s prior is probably that funding research like this wouldn’t be. We haven’t yet sorted out our difference here.
So I still hold the view that funding more psychedelic research would be a good use of EA funds.
My views are the same as Carl’s, hence I didn’t make a further reply. (i.e. Low enough base rates imply the yield on chasing replications does not reach the—high—bar for EA).
Got it. This seems like our crux, in that case.
I think it’s about 30% − 40% that the psychedelic results found to date are real (i.e. that they replicate).
What’s your estimate of how likely the results are to replicate?
Not sure how helpful percentages are given effect sizes and interventions are varied.
Per my OP, I’d benchmark getting results similar or better to SSRIs (i.e. modestly effective for a few mental illnesses) to be the top 3% ish of what I’d expect research to confirm. I’d give 25% for essentially nothing replicating and it going the way of power poses, priming, or other psych dead ends Scott mentions.
The remaining 70% is smeared across much less impressive results (and worth noting SSRIs are hardly a miracle cure): maybe sort-of helpful for one condition, maybe helpful but only for a subset of motivated individuals, etc. etc.
Do you feel like you updated after reading the studies I point to?
e.g. were you initially like “there’s literally a 0% chance this is real” and now you’re like “well, maybe there’s a 3% chance that psychedelics are an effective treatment & 70% that psychedelics do something but aren’t more efficacious than SSRIs” ?
I see, thanks.
Do you assign a non-negligible chance to psychedelics meaningfully outperforming current treatments like SSRIs? (“3% for similar to or better than SSRIs” blurs together the case where psychedelics are just as efficacious as SSRIs and the case where they are massively more efficacious.)
This is important because if there’s a small, non-negligible chance of a large effect over that of current treatment, investment could still be warranted.
Comparison point: GiveWell has directed tens of millions USD to deworming programs, even though most GiveWell staffers think there’s only a 1-2% chance that deworming effects are real.
1) Generally my probability mass is skewed to the lower ends of the intervals I’m noting. Thus the 70% band is more with multiple caveats rather than just one (e.g. a bit like—as Scott describes it—Ketamine: only really useful for depression, and even then generally modest effects even as second line therapy). Likewise the 3% is mostly ‘around SSRIs and maybe slightly better’, with subpercentile mass as the dramatic breakthrough I think you have in mind.
2) Re. updates: There wasn’t a huge update on reading the studies (not that I claim to have examined them closely), because I was at least dimly aware since medical school of psychedelics having some promise in mental health.
Although this was before I appreciated the importance of being quantitative, I imagine I would have given higher estimates back then, with the difference mainly accounted for by my appreciation of how treacherous replication has proven in both medicine and psychology.
Seeing that at least some of the studies were conducted reasonably given their limitations has attenuated this hit, but I had mostly priced this in as I expected to see this (i.e. I wasn’t expecting to see the body of psychedelic work was obviously junk science etc.).
3) Aside: Givewell’s view doesn’t appear to be “1-2% that deworming effects are real”, but:
I.e. Their central estimate prices across a range of ‘no effect’ ‘modest effect’ ‘as good as the index study advertised’, but weighted towards the lower end.
One could argue whether, if applied to psychedelics, whether the discount factor they suggest should be higher or lower than this (multiple studies would probably push to a more generous discount factors, but an emphasis on quality might point to more pessimistic ones, as the Kremer index study has I think a stronger methodology—and a lot more vetting—than the work noted here). But even something like a discount of ~0.1 would make a lot of the results noted above considerably less exciting (e.g. The Calhart-Harris effect size drops to d~0.3, which is good but puts it back into the ranges seen with existing interventions like CBD).
VoI is distinct from this best guess (analogously, a further deworming RCT to reduce uncertainty may have higher or lower value than ‘exploiting’ based on current uncertainty), but I’d return to my prior remarks to suggest the likelihood of ending up with something ‘(roughly) as good as initial results advertise’ is low/negligible enough not to make it a good EA buy.
4) Further aside: Given the OP was about psychedelics generally (inc advocacy and research) rather than the particular points on whether confirmatory research was a good idea, I’d take other (counter-) arguments addressed more generally than this to be in scope.