Hi, I really like this (shared this in my social media channels). I am working on another new research methods project and great to see others working in this area. ( we ( regivolution.org ) presented our work in EA Global London last month, and we hope to make forum post in here soon as well about us as well as our rigorous ultra-rapid review methods ) We might even use this as an inspiration for some materials we will generate in future (We will of course cite/link to this resource)!
General feedback about the project and content:
Some improvement suggestions—Note: In general I want to say that I really like the quality and I didn’t see almost any red flags in the quality of your work, you have gone in to detail, and cite pretty much similar material I would cite myself as well and have good material I wasn’t even aware of before. Also, these comments are at general level, I did not go through individual studies, except that I noticed one likely quite a problematic study in the page why fund research. I added this comment straight to the google docs document
Are you aware of the Oxford prioritisation project’s short assessment of open science also posted to EA forums? They pretty much stated that there would be in general no room for funding for open science including registered reports. However, they mentioned things that could change their mind and I think your case falls to these categories. I think this could be worth mentioning.
See here: https://forum.effectivealtruism.org/posts/x87EKumG8gS4aH7xP/daniel-may-open-science-little-room-for-more-funding
Transparency of literature review methods: I would like to hear more about the procedure you had for choosing this meta-research cause over other causes. Now it seems to me that you were interested about the possibilities of meta-research did some preliminary, very fast review of 4 options, then you chose that this researcher and pre-registration would be the best bet and then wrote detailed analyses only about this topic and nothing about others. It would be interesting to get a better impression of your “protocol/pre-notes” (:D ) what you planned, and what post-hoc changes you did during the way. I think this would be important, but this would be particularly important as you are advocating for pre-registration policies. (of course, I would not expect full-pre-registration but just some even very rough description of your process along the way, no matter how chaotic it was). One other thing you should probably be quite transparent is how you assess the quality of your references, if you do not assess each one of them carefully this is fine (based on this one quite clearly problematic study I noticed, this is probably likely), but in my opinion you should be transparent about this.
One alternative funding option (note: I have intellectual conflicts of interest as I am involved in both initiatives mentioned, also I think that it is not necessarily better than this very carefully chosen the one you have): Advocating for the use of evidence frameworks/systems. My opinion currently is that in the end for reaching superb scientific research impact we need to advocate for evidence frameworks/systems. The rationale is similar to why we should advocate for meta-science. Meta-science has potential to improve multiple research areas simultaneously. Evidence frameworks have potential to improve multiple methodological aspects simultaneously in a flexible way that can easily and transparently be improved, (e.g. when new meta-research evidence comes up) these can also be applicable to multiple research areas, improve the level of collaboration and improve the transparency of research findings. GRADE in evidence-based medicine is one quite successful example of this although it was still a lot of issues. Curate science is a database and framework in development that specifically designed to assess the credibility of replication studies (particularly in psychology)
European Union has also funded meta-science quite aggressively recently (see this quite large Mirror, research project)
Things I am less certain about and you might have considered them already
What would make you change your mind: I think this is more concrete and actionable than just listing out what are problems.
The degree of confidence: I personally like the way to present ones confidence to claims e.g. 80% certain.
Functionality of the website
Small thing—feature where you can see a footnote when you hover a mouse over number would be good, also confirmation to move another website was slightly irritating, when you wanted to look what the references actually are in laptop fast, but I think these may be more preference issues.
As I mentioned, in general, I think your work is good, and I didn’t see many red flags in the quality of your work, you have done detailed work, and cite pretty much similar material I would cite myself as well. Even though I have read quite a bit of research in this area there was even new interesting stuff for me.
Thanks so much for the detailed comments and feedback!
1) I’m aware of Oxford Prio’s research into this and had read their research. They looked more into Open Science which is related but slightly different from meta-research. There are many more funders funding transparancy, openness and data sharing than meta-research. I do not recall that Oxford prio looked Registered Reports specifically. I think it’s an interesting analysis but I think the funders they listed in their analysis are neglecting Registered Reports.
2) re: the literature review methods: we do have a section titled ‘Our Research process’ in one of our reports. I did a literature review and based on expertise in science created a shortlist of potentially very impactful projects within meta-research (some other projects I considered are listed in the report linked above). Then I interviewed about 5 people in the meta-research field and asked them about their opinions on these projects. Registered Reports seemed most promising of all of these projects. How assess the quality of the papers I cite: as a general rule I look into the studies corresponding to how much weight I place on the point I’m trying to make. So if the study is crucial to the point I’m trying to I look into it more. I also try to find evidence from many different sources so that no argument rests on a single source (also see Sequence vs. Cluster Thinking by Givewell), and generally see whether there are many counter arguments and cite those. Thanks also for flagging the conflict of interest in the cancer study I cite, I absolutely see where you’re coming from and have added a footnote to the analysis. In this particular case, I disagree that the study should be completely dismissed and has little value, only because this study was partially funded by a pharma cooperation. I believe that the results can broadly be informative and give a general idea of the impact of new drugs, though it might be an a biased overestimate. This because the study was more descriptive as opposed to an experimental, is not advocating for a particular drug, was peer-reviewed, pushlished and partially funded by the impartial National Institute on Aging. It also does fits with my priors that new cancer drugs often do work quite well.
Evidence Frameworks such as GRADE are interesting and I think they deserve to be funded more. I think they are in practise applied much more in clinical science and would not have such a generalized impact as Registered Reports.
re: EU funding for meta-research: It’s encouraging that they have funded this specific doctoral programme for meta-research, and that space should definitely be watched, but I can’t see that there is a big general pot of funding available for meta-research, that would hint at this area not stillbeing relatively neglected.
In the future we might add explicit numerical estimates of the degrees of confidence in particular claims for now we opted for careful language and hedging where appropriate. It’s a great idea to add a section on what would ‘falsify’ the case for this grant, we might do this in the future (from the top of my mind: if one could provide evidence that RR advocacy would be bad in the sense that getting people to donate to this over conspicious consumption would be net bad for the world, or if there’d be much better high-risk, high-reward funding opportunity that smaller donors would give to that we’re distracting donors from then I would definitely change my mind).
Thanks also for the website suggestions! Generally your feedback is very much appreciated.