Went in to reading this with a lot of skepticism because RCTs exist for a reason (anecdotal experiences are heavily prone to bias, so theyâre not good for determining effectiveness or side effect profiles)
I was disappointed. You justify this analysis of Reddit and Amazon reviews by saying:
âIf we only accept RCT evidence, but wonât conduct RCTs without prior evidence, how do new treatments ever get studied? There has to be a gradientâa sort of ramp up.â
This is a fair point but then at the end you donât just recommend high quality trials but you encourage people to take Chanca piedra without them. So rather than really thinking of this as a bootstrap to trials youâve actually just bypassed trials and are now recommending a herbal supplement based on Reddit and Amazon comments.
Feels like a Chestertonâs fence situation where youâve forgotten why RCTs are not just nice but necessary to determine effectiveness and side effect profiles.
Weâd absolutely love there to be RCTs and definitely encourage larger studies to be run (as mentioned in the Final Recommendations section). It seems like such low-hanging fruit. And one of the main motivations behind this project is that it becomes exactly that incentive needed to get those studies running.
At the same time, it could be years until a large RCT is run. Given the rising incidence of kidney stones worldwide, it seems like asking people to drink more water and eat low-oxalate foods isnât cutting it.
We wrote âCautiously consider taking chanca piedra yourselfâ as our very last recommendation. We wanted to strike a balance between recommending something for which large RCTs are still missing and acknowledging that the suffering involved can be so severe that some action could be warranted (especially given that, as we emphasize, chanca piedra seems to cause fewer side effects than e.g. melatonin and ashwagandha).
Reviews are good for short-term common side effects, not as good for rare or delayed severe ones. Would an analysis of Reddit and Amazon reviews of thalidomide have revealed its negative effects? Bex powder? Diethylstilbestrol? Betel nut?
Thanks, Henry. Your care, as a doctor, for peopleâs wellbeing shines through! Weâd also hate to learn that this herb causes e.g. delayed, severe side effects. We hope this question can be settled for good asap given the scope and severity of the suffering involved here!
We start our post acknowledging that online reviews are not usually a reliable source of information concerning medical matters. So we asked ourselves: if there was a statistically significant signal, how could we possibly find it? This is what motivated us to:
Compare reviews across three different platforms (WebMD, Amazon, Reddit).
Compare reviews of chanca piedra to reviews of other kidney stone treatments within those same platforms.
Compare reviews of other popular supplements within those same platforms (e.g., ashwagandha, melatonin).
Do sub-analyses of the data, e.g. looking only at the most credible reviews and reading closely any reviews mentioning any severe or rare side effects (which we report).
Look into whatever literature we could find about chanca piedra to see if mechanistic explanations were compatible with our findings.
Would an analysis of Reddit and Amazon reviews of thalidomide have revealed its negative effects? Bex powder? Diethylstilbestrol? Betel nut?
Not necessarily, no. But if we had access to thousands of anecdotes and reviews of these medications dating back to 2008, including by people who have taken them for years, the chances would go up significantly. I guess thatâs our main âselling pointâ.
In the end, our goal is to reduce as much extreme suffering as possible. If it turns out that taking chanca piedra actually increases extreme suffering, weâd want to be the first ones to know! But from what we can tell, that hypothesis is looking very unlikely.
Went in to reading this with a lot of skepticism because RCTs exist for a reason (anecdotal experiences are heavily prone to bias, so theyâre not good for determining effectiveness or side effect profiles)
I was disappointed. You justify this analysis of Reddit and Amazon reviews by saying:
This is a fair point but then at the end you donât just recommend high quality trials but you encourage people to take Chanca piedra without them. So rather than really thinking of this as a bootstrap to trials youâve actually just bypassed trials and are now recommending a herbal supplement based on Reddit and Amazon comments.
Feels like a Chestertonâs fence situation where youâve forgotten why RCTs are not just nice but necessary to determine effectiveness and side effect profiles.
Thank you for your feedback, Henry! :)
Weâd absolutely love there to be RCTs and definitely encourage larger studies to be run (as mentioned in the Final Recommendations section). It seems like such low-hanging fruit. And one of the main motivations behind this project is that it becomes exactly that incentive needed to get those studies running.
At the same time, it could be years until a large RCT is run. Given the rising incidence of kidney stones worldwide, it seems like asking people to drink more water and eat low-oxalate foods isnât cutting it.
We wrote âCautiously consider taking chanca piedra yourselfâ as our very last recommendation. We wanted to strike a balance between recommending something for which large RCTs are still missing and acknowledging that the suffering involved can be so severe that some action could be warranted (especially given that, as we emphasize, chanca piedra seems to cause fewer side effects than e.g. melatonin and ashwagandha).
Am reminded of this XKCD:
Reviews are good for short-term common side effects, not as good for rare or delayed severe ones.
Would an analysis of Reddit and Amazon reviews of thalidomide have revealed its negative effects? Bex powder? Diethylstilbestrol? Betel nut?
Thanks, Henry. Your care, as a doctor, for peopleâs wellbeing shines through! Weâd also hate to learn that this herb causes e.g. delayed, severe side effects. We hope this question can be settled for good asap given the scope and severity of the suffering involved here!
We start our post acknowledging that online reviews are not usually a reliable source of information concerning medical matters. So we asked ourselves: if there was a statistically significant signal, how could we possibly find it? This is what motivated us to:
Compare reviews across three different platforms (WebMD, Amazon, Reddit).
Compare reviews of chanca piedra to reviews of other kidney stone treatments within those same platforms.
Compare reviews of other popular supplements within those same platforms (e.g., ashwagandha, melatonin).
Do sub-analyses of the data, e.g. looking only at the most credible reviews and reading closely any reviews mentioning any severe or rare side effects (which we report).
Look into whatever literature we could find about chanca piedra to see if mechanistic explanations were compatible with our findings.
Not necessarily, no. But if we had access to thousands of anecdotes and reviews of these medications dating back to 2008, including by people who have taken them for years, the chances would go up significantly. I guess thatâs our main âselling pointâ.
In the end, our goal is to reduce as much extreme suffering as possible. If it turns out that taking chanca piedra actually increases extreme suffering, weâd want to be the first ones to know! But from what we can tell, that hypothesis is looking very unlikely.