Doing something to democratize randomized controlled trials (RCTs) - thereby reducing the risk involved in testing new ideas and interventions.
RCTs are a popular methodology in medicine and the social sciences. They create a safety net for the scientists (and consumers) to test that the drug works as intended and doesn’t turn people into mutants.
I think using this methodology in other fields would be a high-leverage intervention. For example startups, policy-making, education, etc. Being able to try out new ideas without facing a huge downside should be a feature of every field. Big institutions already conduct similar tests before they release something. But I’m wondering how useful it would be to allow small institutions, startups, and maybe even individuals to do this.
Plus, adding an RCT into the launch pipeline of any intervention/product allows us to see the unintended consequences before they’re out there. I think this would have at least been helpful for the social media companies.
Based on some googling, I’ve understood that RCTs are very costly. But if the reasoning makes sense, this is exactly the kind of thing others can’t try out that a megaproject should.
Here’s a paraphrased quote by Eliezer Yudkowsky, that is relevant in this context: If people could learn from their mistakes without dying from them, well actually, that in itself would tend to fix a whole lot of problems over time. [source]
P.S. I’m thinking on working on this idea full-time in 2022. It would be very helpful to hear whatever criticism/thoughts you have—It’ll help me make sure my time is effectively spent.
I think you should write this up as a full post or at least as a question.
I don’t think people will see this and you deserve reasonable attention if its a full time project.
Note that my knee jerk reaction is caution. The value of RCTs is well known and they are coveted. Then, in the mental models I use, I would discount the idea that it could be readily distributed.
For example, something like the following logic might apply:
An RCT, or something that looks like it, with many of the characteristics/quality you want, will cost more than the seed grant or early funding for the new org doing the actual intervention.
Most smaller projects start with a pilot that gives credible information about effectiveness (by design, often much cheaper than an RCT).
Then “democratizing RCTs”, as you frame it, will basically boil down to funding/subsidizing smaller projects than bigger ones.
I’m happy for this reasoning to be thoroughly destroyed and RCTs available for all!
Doing something to democratize randomized controlled trials (RCTs) - thereby reducing the risk involved in testing new ideas and interventions.
RCTs are a popular methodology in medicine and the social sciences. They create a safety net for the scientists (and consumers) to test that the drug works as intended and doesn’t turn people into mutants.
I think using this methodology in other fields would be a high-leverage intervention. For example startups, policy-making, education, etc. Being able to try out new ideas without facing a huge downside should be a feature of every field. Big institutions already conduct similar tests before they release something. But I’m wondering how useful it would be to allow small institutions, startups, and maybe even individuals to do this.
Plus, adding an RCT into the launch pipeline of any intervention/product allows us to see the unintended consequences before they’re out there. I think this would have at least been helpful for the social media companies.
Based on some googling, I’ve understood that RCTs are very costly. But if the reasoning makes sense, this is exactly the kind of thing others can’t try out that a megaproject should.
Here’s a paraphrased quote by Eliezer Yudkowsky, that is relevant in this context: If people could learn from their mistakes without dying from them, well actually, that in itself would tend to fix a whole lot of problems over time. [source]
P.S. I’m thinking on working on this idea full-time in 2022. It would be very helpful to hear whatever criticism/thoughts you have—It’ll help me make sure my time is effectively spent.
I think you should write this up as a full post or at least as a question.
I don’t think people will see this and you deserve reasonable attention if its a full time project.
Note that my knee jerk reaction is caution. The value of RCTs is well known and they are coveted. Then, in the mental models I use, I would discount the idea that it could be readily distributed.
For example, something like the following logic might apply:
An RCT, or something that looks like it, with many of the characteristics/quality you want, will cost more than the seed grant or early funding for the new org doing the actual intervention.
Most smaller projects start with a pilot that gives credible information about effectiveness (by design, often much cheaper than an RCT).
Then “democratizing RCTs”, as you frame it, will basically boil down to funding/subsidizing smaller projects than bigger ones.
I’m happy for this reasoning to be thoroughly destroyed and RCTs available for all!