Seeking important GH or IDEV working papers to evaluate
I am a Global Health and Development editor at The Unjournal. In brief, our mission is to organize and fund public, journal-independent evaluation and rating of working papers. Right now we are focusing on quantitative work informing global priorities, and my area focuses on global health and international development.
I am writing because I want people to send me working papers (pre-publication) that are decision-relevant to organizations working in global health and international development. [1] Anyone can submit paper suggestions to rbriggs@uoguelph.ca. If you prefer, authors can submit work through our quick submission form HERE, and others can suggest work using this form.
I think two especially useful cases are:
Authors sending their own papers so that they can get high quality feedback before submission. This feedback will be public, but we are open to discussing delaying or embargoing feedback on a case-by-case basis.
Grant-making staff in orgs like GiveWell or OpenPhil sending in papers that are relevant to their work so that they can read expert reviews before they use the results to inform decisions.
The rest of this post explains in more detail: what The Unjournal is, our review process, why we’re doing this, what we’re looking for in a paper, and who can submit.
The Unjournal
The Unjournal (evaluations are here) is the brainchild of David Reinstein. Its mission is to offer clear, public pre-publication expert review of working papers. There are two motivations behind this project. The first is that researchers can do better work if they are able to more easily get high quality feedback on their work. The second is that the entire research ecosystem can benefit from these reviews being public. We pay expert reviewers for their time and we give awards for especially high quality research that we evaluate.
The initial focus of The Unjournal is on quantitative work that informs global priorities, especially in economics, policy, and social science. I’m an editor for global health and international development.
The process
While our process runs similarly to an academic journal, we do not publish articles. Instead, we link to working papers and publish reviews (which we call ‘evaluations’) and author responses.
The process starts with editors screening lists of working papers created from submissions or based on our knowledge of the research area. Once a working paper is selected, we contact the authors to ask if they would like to be involved in the process. Author involvement allows authors to converse with reviews and/or respond to reviews. It is not mandatory, but we prefer if authors are involved.
We then aim to find two reviewers per paper. Reviewers are experts in the research area and are paid for their reviews. Reviewers have the choice to remain anonymous or not. They read the work and fill out standardized review forms.
We then publish the reviews of the work on our webpage (see here) alongside brief comments from the editor and responses from the authors if they so wish.
Why do this?
If the journal system already exists, then why have something that looks kind of like a journal and offers sort of similar evaluations?
We think The Unjournal adds value to the existing journal system in a number of important ways. For example:
Unlike most academic journals, we have no bias against null results. This means that we’re just as likely to review important work that “fails” to find an effect as important work that rejects the null.
We’re transparent. We publish reviews online so that anyone can see not only the working paper but also what expert reviewers thought of it. Most social science journals do not do this.
We’re experimental. Journals move very slowly, but we’re willing to try new things. For example, we’re exploring ways to have work with “living” research projects in dynamic formats. Evaluations can can be updated as the work progresses, and as we learn more about a research question.
We’re fast. It can take over 5 years to publish a paper in economics. We aim to have published reviews of working papers in a period of months.
We collect standardized metrics across all papers. We aim to synthesize and analyze these in useful ways, as well as making this quantitative data accessible to meta-science researchers, meta-analysts, and tool-builders.
Ideal papers and who can submit
Ideal papers for me are those that are likely to be decision-relevant to the work of organizations in the international development or global health space. Papers should be quantitative and pre-publication. Anyone can submit a paper.
The easiest way to submit a paper is simply to email me about it at rbriggs@uoguelph.ca
- ^
Different disciplines have different names for these. We’re interested in working papers, pre-prints, papers hosted on places like arxiv, but not work that is published in academic journals or under review. We’re aiming to come into the process before submission to, or publication in, an academic journal.
This paper looks like the most promising intervention to improve air quality that I have seen. The authors even have an explicit cost-effectiveness analysis. I would love to see it critically evaluated. (I took a stab at doing so here.)
Brilliant! Thank you!
Thanks for this!
I just wanted to add that we can evaluate work that is under review at a traditional journal. Also, having your work evaluated by The Unjournal does not prevent you from submitting it to a traditional journal (before, during, or after). The Unjournal gives you more feedback and a quantified evaluation sooner; it doesn’t slow you down or limit your options.
But, as Ryan suggests, at this point in our trajectory we prefer to evaluate work early in the process, rather than work that is very close to being accepted in a peer-reviewed journal.
(Also, see our guidelines for evaluators for a view of the feedback and ratings/predictions we ask for.)
Great to see this moving forward!