Hi, I really like this (shared this in my social media channels). I am working on another new research methods project and great to see others working in this area. ( we ( regivolution.org ) presented our work in EA Global London last month, and we hope to make forum post in here soon as well about us as well as our rigorous ultra-rapid review methods ) We might even use this as an inspiration for some materials we will generate in future (We will of course cite/link to this resource)!
General feedback about the project and content:
Some improvement suggestions—Note: In general I want to say that I really like the qualityand I didn’t see almost any red flags in the quality of your work, you have gone in to detail, and cite pretty much similar material I would cite myself as well and have good material I wasn’t even aware of before. Also, these comments are at general level, I did not go through individual studies, except that I noticed one likely quite a problematic study in the page why fund research. I added this comment straight to the google docs document
Are you aware of the Oxfordprioritisation project’s short assessment of open science also posted to EA forums? They pretty much stated that there would be in general no room for funding for open science including registered reports. However, they mentioned things that could change their mind and I think your case falls to these categories. I think this could be worth mentioning.
Transparency of literature review methods: I would like to hear more about the procedure you had for choosing this meta-research cause over other causes. Now it seems to me that you were interested about the possibilities of meta-research did some preliminary, very fast review of 4 options, then you chose that this researcher and pre-registration would be the best bet and then wrote detailed analyses only about this topic and nothing about others. It would be interesting to get a better impression of your “protocol/pre-notes” (:D ) what you planned, and what post-hoc changes you did during the way. I think this would be important, but this would be particularly important as you are advocating for pre-registration policies. (of course, I would not expect full-pre-registration but just some even very rough description of your process along the way, no matter how chaotic it was). One other thing you should probably be quite transparent is how you assess the quality of your references, if you do not assess each one of them carefully this is fine (based on this one quite clearly problematic study I noticed, this is probably likely), but in my opinion you should be transparent about this.
One alternative funding option (note: I have intellectual conflicts of interest as I am involved in both initiatives mentioned, also I think that it is not necessarily better than this very carefully chosen the one you have): Advocating for the use of evidence frameworks/systems. My opinion currently is that in the end for reaching superb scientific research impact we need to advocate for evidence frameworks/systems. The rationale is similar to why we should advocate for meta-science. Meta-science has potential to improve multiple research areas simultaneously. Evidence frameworks have potential to improve multiple methodological aspects simultaneously in a flexible way that can easily and transparently be improved, (e.g. when new meta-research evidence comes up) these can also be applicable to multiple research areas, improve the level of collaboration and improve the transparency of research findings. GRADE in evidence-based medicine is one quite successful example of this although it was still a lot of issues. Curate science is a database and framework in development that specifically designed to assess the credibility of replication studies (particularly in psychology)
European Union has also funded meta-science quite aggressively recently (see this quite large Mirror, research project)
Things I am less certain about and you might have considered them already
What would make you change your mind: I think this is more concrete and actionable than just listing out what are problems.
The degree of confidence: I personally like the way to present ones confidence to claims e.g. 80% certain.
Functionality of the website
Small thing—feature where you can see a footnote when you hover a mouse over number would be good, also confirmation to move another website was slightly irritating, when you wanted to look what the references actually are in laptop fast, but I think these may be more preference issues.
As I mentioned, in general, I think your work is good, and I didn’t see many red flags in the quality of your work, you have done detailed work, and cite pretty much similar material I would cite myself as well. Even though I have read quite a bit of research in this area there was even new interesting stuff for me.
Hi, I really like this (shared this in my social media channels). I am working on another new research methods project and great to see others working in this area. ( we ( regivolution.org ) presented our work in EA Global London last month, and we hope to make forum post in here soon as well about us as well as our rigorous ultra-rapid review methods ) We might even use this as an inspiration for some materials we will generate in future (We will of course cite/link to this resource)!
General feedback about the project and content:
Some improvement suggestions—Note: In general I want to say that I really like the quality and I didn’t see almost any red flags in the quality of your work, you have gone in to detail, and cite pretty much similar material I would cite myself as well and have good material I wasn’t even aware of before. Also, these comments are at general level, I did not go through individual studies, except that I noticed one likely quite a problematic study in the page why fund research. I added this comment straight to the google docs document
Are you aware of the Oxford prioritisation project’s short assessment of open science also posted to EA forums? They pretty much stated that there would be in general no room for funding for open science including registered reports. However, they mentioned things that could change their mind and I think your case falls to these categories. I think this could be worth mentioning.
See here: https://forum.effectivealtruism.org/posts/x87EKumG8gS4aH7xP/daniel-may-open-science-little-room-for-more-funding
Transparency of literature review methods: I would like to hear more about the procedure you had for choosing this meta-research cause over other causes. Now it seems to me that you were interested about the possibilities of meta-research did some preliminary, very fast review of 4 options, then you chose that this researcher and pre-registration would be the best bet and then wrote detailed analyses only about this topic and nothing about others. It would be interesting to get a better impression of your “protocol/pre-notes” (:D ) what you planned, and what post-hoc changes you did during the way. I think this would be important, but this would be particularly important as you are advocating for pre-registration policies. (of course, I would not expect full-pre-registration but just some even very rough description of your process along the way, no matter how chaotic it was). One other thing you should probably be quite transparent is how you assess the quality of your references, if you do not assess each one of them carefully this is fine (based on this one quite clearly problematic study I noticed, this is probably likely), but in my opinion you should be transparent about this.
One alternative funding option (note: I have intellectual conflicts of interest as I am involved in both initiatives mentioned, also I think that it is not necessarily better than this very carefully chosen the one you have): Advocating for the use of evidence frameworks/systems. My opinion currently is that in the end for reaching superb scientific research impact we need to advocate for evidence frameworks/systems. The rationale is similar to why we should advocate for meta-science. Meta-science has potential to improve multiple research areas simultaneously. Evidence frameworks have potential to improve multiple methodological aspects simultaneously in a flexible way that can easily and transparently be improved, (e.g. when new meta-research evidence comes up) these can also be applicable to multiple research areas, improve the level of collaboration and improve the transparency of research findings. GRADE in evidence-based medicine is one quite successful example of this although it was still a lot of issues. Curate science is a database and framework in development that specifically designed to assess the credibility of replication studies (particularly in psychology)
European Union has also funded meta-science quite aggressively recently (see this quite large Mirror, research project)
http://miror-ejd.eu/ec-documents/
Things I am less certain about and you might have considered them already
What would make you change your mind: I think this is more concrete and actionable than just listing out what are problems.
The degree of confidence: I personally like the way to present ones confidence to claims e.g. 80% certain.
Functionality of the website
Small thing—feature where you can see a footnote when you hover a mouse over number would be good, also confirmation to move another website was slightly irritating, when you wanted to look what the references actually are in laptop fast, but I think these may be more preference issues.
As I mentioned, in general, I think your work is good, and I didn’t see many red flags in the quality of your work, you have done detailed work, and cite pretty much similar material I would cite myself as well. Even though I have read quite a bit of research in this area there was even new interesting stuff for me.