Thanks so much for the detailed comments and feedback!
1) I’m aware of Oxford Prio’s research into this and had read their research. They looked more into Open Science which is related but slightly different from meta-research. There are many more funders funding transparancy, openness and data sharing than meta-research. I do not recall that Oxford prio looked Registered Reports specifically. I think it’s an interesting analysis but I think the funders they listed in their analysis are neglecting Registered Reports.
2) re: the literature review methods: we do have a section titled ‘Our Research process’ in one of our reports. I did a literature review and based on expertise in science created a shortlist of potentially very impactful projects within meta-research (some other projects I considered are listed in the report linked above). Then I interviewed about 5 people in the meta-research field and asked them about their opinions on these projects. Registered Reports seemed most promising of all of these projects. How assess the quality of the papers I cite: as a general rule I look into the studies corresponding to how much weight I place on the point I’m trying to make. So if the study is crucial to the point I’m trying to I look into it more. I also try to find evidence from many different sources so that no argument rests on a single source (also see Sequence vs. Cluster Thinking by Givewell), and generally see whether there are many counter arguments and cite those. Thanks also for flagging the conflict of interest in the cancer study I cite, I absolutely see where you’re coming from and have added a footnote to the analysis. In this particular case, I disagree that the study should be completely dismissed and has little value, only because this study was partially funded by a pharma cooperation. I believe that the results can broadly be informative and give a general idea of the impact of new drugs, though it might be an a biased overestimate. This because the study was more descriptive as opposed to an experimental, is not advocating for a particular drug, was peer-reviewed, pushlished and partially funded by the impartial National Institute on Aging. It also does fits with my priors that new cancer drugs often do work quite well.
Evidence Frameworks such as GRADE are interesting and I think they deserve to be funded more. I think they are in practise applied much more in clinical science and would not have such a generalized impact as Registered Reports.
re: EU funding for meta-research: It’s encouraging that they have funded this specific doctoral programme for meta-research, and that space should definitely be watched, but I can’t see that there is a big general pot of funding available for meta-research, that would hint at this area not stillbeing relatively neglected.
In the future we might add explicit numerical estimates of the degrees of confidence in particular claims for now we opted for careful language and hedging where appropriate. It’s a great idea to add a section on what would ‘falsify’ the case for this grant, we might do this in the future (from the top of my mind: if one could provide evidence that RR advocacy would be bad in the sense that getting people to donate to this over conspicious consumption would be net bad for the world, or if there’d be much better high-risk, high-reward funding opportunity that smaller donors would give to that we’re distracting donors from then I would definitely change my mind).
Thanks also for the website suggestions! Generally your feedback is very much appreciated.
Thanks so much for the detailed comments and feedback!
1) I’m aware of Oxford Prio’s research into this and had read their research. They looked more into Open Science which is related but slightly different from meta-research. There are many more funders funding transparancy, openness and data sharing than meta-research. I do not recall that Oxford prio looked Registered Reports specifically. I think it’s an interesting analysis but I think the funders they listed in their analysis are neglecting Registered Reports.
2) re: the literature review methods: we do have a section titled ‘Our Research process’ in one of our reports. I did a literature review and based on expertise in science created a shortlist of potentially very impactful projects within meta-research (some other projects I considered are listed in the report linked above). Then I interviewed about 5 people in the meta-research field and asked them about their opinions on these projects. Registered Reports seemed most promising of all of these projects. How assess the quality of the papers I cite: as a general rule I look into the studies corresponding to how much weight I place on the point I’m trying to make. So if the study is crucial to the point I’m trying to I look into it more. I also try to find evidence from many different sources so that no argument rests on a single source (also see Sequence vs. Cluster Thinking by Givewell), and generally see whether there are many counter arguments and cite those. Thanks also for flagging the conflict of interest in the cancer study I cite, I absolutely see where you’re coming from and have added a footnote to the analysis. In this particular case, I disagree that the study should be completely dismissed and has little value, only because this study was partially funded by a pharma cooperation. I believe that the results can broadly be informative and give a general idea of the impact of new drugs, though it might be an a biased overestimate. This because the study was more descriptive as opposed to an experimental, is not advocating for a particular drug, was peer-reviewed, pushlished and partially funded by the impartial National Institute on Aging. It also does fits with my priors that new cancer drugs often do work quite well.
Evidence Frameworks such as GRADE are interesting and I think they deserve to be funded more. I think they are in practise applied much more in clinical science and would not have such a generalized impact as Registered Reports.
re: EU funding for meta-research: It’s encouraging that they have funded this specific doctoral programme for meta-research, and that space should definitely be watched, but I can’t see that there is a big general pot of funding available for meta-research, that would hint at this area not stillbeing relatively neglected.
In the future we might add explicit numerical estimates of the degrees of confidence in particular claims for now we opted for careful language and hedging where appropriate. It’s a great idea to add a section on what would ‘falsify’ the case for this grant, we might do this in the future (from the top of my mind: if one could provide evidence that RR advocacy would be bad in the sense that getting people to donate to this over conspicious consumption would be net bad for the world, or if there’d be much better high-risk, high-reward funding opportunity that smaller donors would give to that we’re distracting donors from then I would definitely change my mind).
Thanks also for the website suggestions! Generally your feedback is very much appreciated.