By confidence level I mean something like “how confident are you in the methodology of this study”, which could be operationalized as “how much would this change your mind if you didn’t already agree with it?”. I’m specifically not asking how confident you are in the study’s conclusion: bad studies demonstrate true things all the time.
I ask because checking studies is a lot of work, and it’s frustrating to spend several hours reading a paper, only to have a person dismiss fatal flaws by citing a different paper (that is equally flawed, but will also take hours to nail down). No data is perfect so I don’t have the luxury of throwing flawed studies out, but asking for a declaration of flaws felt like a good compromise to me.
For example, in addition to the issues you list with the study you cited, I see several more just from the abstract:
The study population was selected to have health issues exacerbated by Standard Shitty American Diet.
As I said in the post, I have zero interest in defending SSAD, and believe most people consume too much meat to the detriment of their health.
This is the equivalent of taking anemic vegans and putting them on a well-administered high-meat diet. It probably wil raise the iron levels of study participants, and that may represent a net improvement in their health, but it wouldn’t prove naive high-meat diets superior to veganism in the general population.
the difference in change in cholesterol levels between plant-based and control was not statistically significant.
The difference in BMI change is quite statistically significant, and large (4 BMI points).
BMI and cholesterol were the only end points, not mortality or cardiac events
The dropout rate was pretty bad (25%), although ~equal between treatment and control group so not nearly as distorting as it could have been.
The control group was on SSAD, rather than an equally effortful meat-containing diet.
The intervention group received B12 and pretty extensive nutrition education, which is the exact thing I complain about EA vegans not doing.
The videos you link to are comparing veganism to keto or very high-meat diets- which I already agreed are probably bad for most people. I wish you had declared that as well.
I fully agree large ag companies have been waging a multi-decade war on nutrition epistemics. That’s one reason I discount nutritional research, and I’m confused why you take the war as a reason to trust it.
Thanks for your comment and clarifying, I appreciate the effort, I realise it is difficult to converse about these things whilst retaining respect and sanity.
This RCT came to mind not because it meets all your criteria, but because I thought it was relevant (especially non-monetary costs, i.e. quality of life) to the criteria you set out, modulo multiple months rather than years. It also I think alleviates some concerns about unfeasibility you think vegan diets.
“ideal study is a longitudinal RCT where diet is randomly assigned, cost (across all dimensions, not just money) is held constant, and participants are studied over multiple years to track cumulative effects.”
I didn’t mean to suggest this study implies that a WFPB diet is healthier than (for example) WFPB+(Fish 2/week). I don’t think that sort of question is resolvable on the current evidence either way.
I am sorry I was not clearer about what the video links were saying—I didn’t mean to imply about high meat vs vegan etc. I linked the first one to signpost a scientist who does tightly controlled experimentals setups (perhaps too tersely) but not specifically on WFPB vs WFPB+(Fish 2/week).
The next 3 were about epidemiological evidence that reducing meat consumption follows a strong dose response relationship. While we can’t rule out there could be an uptick in negative outcomes at eliminating, agnosticism seems like the right call.
I don’t take the war as reason to blanket trust nutrition research (or blanket doubt) either, but to be suspicious of conclusions that favour industry and look to a few well done studies and well-respected researchers that have stood the test of time.
Thank you for writing this and if it’s any help, your article is what prompted me to eventually find Plant Chompers and soften to your position to my current agnosticism about the optimality of WFPB vs WFPB+(Fish 2/week). While I do have concerns about some of your baselines assumptions (see initial comment), I think it’s admirable to spend hours on each paper. You might find it beneficial to reach out to Chris Macaskill and collaborate on this projects with him—I suspect he’s got much more time and knowledge than me!
For the sake of my time, this should hopefully be my last comment on this post. Apologies for misleading or inconveniencing you in any way.
Thank you, I really appreciate everything you said here. I respect your desire to bow out and hope it is okay for me to say a few things.
I think “meat seems dose-dependently bad, and it would be weird if that reversed at low levels” is one of the strongest arguments I’ve heard. I have a bunch of specifics I’d want to follow up on, but I think the argument is both well-formed and has considerable evidence behind it.
By confidence level I mean something like “how confident are you in the methodology of this study”, which could be operationalized as “how much would this change your mind if you didn’t already agree with it?”. I’m specifically not asking how confident you are in the study’s conclusion: bad studies demonstrate true things all the time.
I ask because checking studies is a lot of work, and it’s frustrating to spend several hours reading a paper, only to have a person dismiss fatal flaws by citing a different paper (that is equally flawed, but will also take hours to nail down). No data is perfect so I don’t have the luxury of throwing flawed studies out, but asking for a declaration of flaws felt like a good compromise to me.
For example, in addition to the issues you list with the study you cited, I see several more just from the abstract:
The study population was selected to have health issues exacerbated by Standard Shitty American Diet.
As I said in the post, I have zero interest in defending SSAD, and believe most people consume too much meat to the detriment of their health.
This is the equivalent of taking anemic vegans and putting them on a well-administered high-meat diet. It probably wil raise the iron levels of study participants, and that may represent a net improvement in their health, but it wouldn’t prove naive high-meat diets superior to veganism in the general population.
the difference in change in cholesterol levels between plant-based and control was not statistically significant.
The difference in BMI change is quite statistically significant, and large (4 BMI points).
BMI and cholesterol were the only end points, not mortality or cardiac events
The dropout rate was pretty bad (25%), although ~equal between treatment and control group so not nearly as distorting as it could have been.
The control group was on SSAD, rather than an equally effortful meat-containing diet.
The intervention group received B12 and pretty extensive nutrition education, which is the exact thing I complain about EA vegans not doing.
The videos you link to are comparing veganism to keto or very high-meat diets- which I already agreed are probably bad for most people. I wish you had declared that as well.
I fully agree large ag companies have been waging a multi-decade war on nutrition epistemics. That’s one reason I discount nutritional research, and I’m confused why you take the war as a reason to trust it.
Thanks for your comment and clarifying, I appreciate the effort, I realise it is difficult to converse about these things whilst retaining respect and sanity.
This RCT came to mind not because it meets all your criteria, but because I thought it was relevant (especially non-monetary costs, i.e. quality of life) to the criteria you set out, modulo multiple months rather than years. It also I think alleviates some concerns about unfeasibility you think vegan diets.
“ideal study is a longitudinal RCT where diet is randomly assigned, cost (across all dimensions, not just money) is held constant, and participants are studied over multiple years to track cumulative effects.”
I didn’t mean to suggest this study implies that a WFPB diet is healthier than (for example) WFPB+(Fish 2/week). I don’t think that sort of question is resolvable on the current evidence either way.
I am sorry I was not clearer about what the video links were saying—I didn’t mean to imply about high meat vs vegan etc. I linked the first one to signpost a scientist who does tightly controlled experimentals setups (perhaps too tersely) but not specifically on WFPB vs WFPB+(Fish 2/week).
The next 3 were about epidemiological evidence that reducing meat consumption follows a strong dose response relationship. While we can’t rule out there could be an uptick in negative outcomes at eliminating, agnosticism seems like the right call.
I don’t take the war as reason to blanket trust nutrition research (or blanket doubt) either, but to be suspicious of conclusions that favour industry and look to a few well done studies and well-respected researchers that have stood the test of time.
Thank you for writing this and if it’s any help, your article is what prompted me to eventually find Plant Chompers and soften to your position to my current agnosticism about the optimality of WFPB vs WFPB+(Fish 2/week). While I do have concerns about some of your baselines assumptions (see initial comment), I think it’s admirable to spend hours on each paper. You might find it beneficial to reach out to Chris Macaskill and collaborate on this projects with him—I suspect he’s got much more time and knowledge than me!
For the sake of my time, this should hopefully be my last comment on this post. Apologies for misleading or inconveniencing you in any way.
Thank you, I really appreciate everything you said here. I respect your desire to bow out and hope it is okay for me to say a few things.
I think “meat seems dose-dependently bad, and it would be weird if that reversed at low levels” is one of the strongest arguments I’ve heard. I have a bunch of specifics I’d want to follow up on, but I think the argument is both well-formed and has considerable evidence behind it.