Some research evaluations last over time! But Munger’s ‘temporal validity’ argument really stuck with me: the social world changes over time, so things that work in one place and time could fail in another for reasons that have nothing to do with rigor, but changing context.
More broadly, for me personally, the way forward is to incentivize, champion, and promote better and more robust scientific work. I find this motivating and encouraging, and an efficient antidote against cynicism creep. I find it intellectually rewarding because it is an effort that spans many areas including teaching science, doing science, and communicating science. And I find it socially rewarding because it is a teamwork effort embedded in a large group of (largely early career) scientists trying to improve our fields and build a more robust, cumulative science.
I mean, I guess that is sort of encouraging, if you personally are a scientist, since it suggests you can do good work yourself. But it doesn’t offer me much sense that I who am not a scientist will ever in fact be able to trust very much outside established theory in the hard sciences, unless you think better methodology is going to become used nearly always by the big reputable orgs and journals. (I mean I already mostly didn’t have trust, but I kind of hoped GiveWell were relying on the minority of actually solid stuff.)
Obviously, ‘don’t trust anything’ could just be the right conclusion, and people should say it if it’s true! But it’s hard not to get disheartened about giving, if the messages is “don’t trust any research before c.2015, or also a lot of it afterward, even from the most apparently reliable and skeptical sources, and also, even good research produced now often has little external validity, so probably don’t trust that the good current stuff tells you much about what will happen going forward, either”.
Some research evaluations last over time! But Munger’s ‘temporal validity’ argument really stuck with me: the social world changes over time, so things that work in one place and time could fail in another for reasons that have nothing to do with rigor, but changing context.
In general, null results should be our default expectation in behavioral research: https://www.bu.edu/bulawreview/files/2023/12/STEVENSON.pdf
However, per https://eiko-fried.com/antidotes-to-cynicism-creep/#6_Antidotes_to_cynicism_creep
I mean, I guess that is sort of encouraging, if you personally are a scientist, since it suggests you can do good work yourself. But it doesn’t offer me much sense that I who am not a scientist will ever in fact be able to trust very much outside established theory in the hard sciences, unless you think better methodology is going to become used nearly always by the big reputable orgs and journals. (I mean I already mostly didn’t have trust, but I kind of hoped GiveWell were relying on the minority of actually solid stuff.)
Obviously, ‘don’t trust anything’ could just be the right conclusion, and people should say it if it’s true! But it’s hard not to get disheartened about giving, if the messages is “don’t trust any research before c.2015, or also a lot of it afterward, even from the most apparently reliable and skeptical sources, and also, even good research produced now often has little external validity, so probably don’t trust that the good current stuff tells you much about what will happen going forward, either”.