1. Make academic research evaluation better, more efficient, and more informative (but focusing on the ‘impactful’ part of academic research)
2. Bring more academic attention and rigor to impactful research
3. Have academics focus on more impactful topics, and report their work in a way that makes it more impactful
For this to have the biggest impact, changing the systems and leveraging this… we need academics and academic reward systems to buy into it. It needs to be seen as rigorous, serious, ambitious, and practical. We need powerful academics and institutions to back it. But even for more modest goal of getting academic experts to (publicly) evaluate niche EA-relevant work, it’s still important to be seen as serious, rigorous, credible, etc. That’s why we’re aiming for the ‘rigor stuff’ for now, and will probably want to continue this, at least as a flagship tier/stream into the future
But it seems like a great amount of EA work is shallower, or more weirdly formatted than a working paper. I.e., Happier Lives Insitute reports are probably a bit below that level of depth (and we spend a lot more time than many others) and GiveWell’s CEAs have no dedicated write-ups. Would either of these research projects be suitable for the Unjournal?
Would need to look into specific cases. My guess is that your work largely would be, at least 1. If and when we launch the second stream and 2. For the more in-depth stuff that you might think “I could submit this to a conventional journal but it’s too much hassle”.
GiveWell’s CEAs have no dedicated write-ups.
I think they should have more dedicated writeups (or other presentation formats), and perhaps more transparent formats, with clear reasoning transparent justifications for their choices, and robustness calculations, etc. Their recent contest goes in the right direction, though.
In terms of ‘weird formats’ it depends what you mean by weird. We are eager to evaluate work that is not the typical ‘frozen pdf prison’ but is presented (e.g.) as a web site offering foldable explanations for reasoning transparency. And in particular, open-science friendly dynamic documents where the code (or calculations) producing each results can be clearly unfolded, with a clear data pipeline, and where all results can be replicated. This would an improvement over the current journal formats: less prone to error, easier to check, easier to follow, easier to re-use, etc.
I take the theory of change here, as you say to be “make rigorous work more impactful”. But that seems to rely on getting institutional buy in from academia, which sounds quite uphill.
I agree that it’s a challenge, but I think that this is a change whose time has come. Most academics I’ve talk to individually think public evaluation/rating would be better than the dominant (and inefficient) traditional journal system, but everyone thinks “I can’t go outside of this system on my own”. I outline how I think we might be able to crack this (collective action problem and inertia) HERE. (I should probably expand on this.). And I think that we (non-university linked EA researchers and funders) might be in a unique position to help solve this problem. And I think there would be considerable rewards and influence to ‘being the ones who changed this’. But still …
An alternative path is “to make impactful (i.e., EA) work more rigorous”. I’d guess there’s already a large appetite for this in EA. How do you see the tradeoffs here?
I agree this would be valuable, and might be an easier path to pursue. And there may indeed be some tradeoffs (time/attention). I want to continue to consider, discuss, and respond to this in more detail.
For now, some off-the-cuff justifications for the current path:
We can try the high-rigor path first and then could pivot or branch to the ‘bring rigorous evaluation to niche and less formal EA-policy stuff’ later. But I think the reverse ordering would be more difficult (first impressions what what).
IMO a lot of academic work is highly impactful, or could be made highly impactful with a bit of TLC.[1]
Related to that, I’m fairly sympathetic to the points made HERE about valuing expertise and rigor, and not always trying to reinvent the wheel. I think we could do a lot to connect academic/non-EA expertise with EA-aligned goals.
My background/ideas and the setup of The Unjournal might be more suited to doing the ‘make rigorous research more impactful’ (and improving research evaluation) part?
(I’ll respond to your final point in another comment, as it’s fairly distinct)
In the first stage, that is the idea. In the second stage, I propose to expand into other tracks.
Background: The Unjournal is trying to do a few things, and I think there are synergies: (see the Theory of Change sketch here)
1. Make academic research evaluation better, more efficient, and more informative (but focusing on the ‘impactful’ part of academic research)
2. Bring more academic attention and rigor to impactful research
3. Have academics focus on more impactful topics, and report their work in a way that makes it more impactful
For this to have the biggest impact, changing the systems and leveraging this… we need academics and academic reward systems to buy into it. It needs to be seen as rigorous, serious, ambitious, and practical. We need powerful academics and institutions to back it. But even for more modest goal of getting academic experts to (publicly) evaluate niche EA-relevant work, it’s still important to be seen as serious, rigorous, credible, etc. That’s why we’re aiming for the ‘rigor stuff’ for now, and will probably want to continue this, at least as a flagship tier/stream into the future
But it seems like a great amount of EA work is shallower, or more weirdly formatted than a working paper. I.e., Happier Lives Insitute reports are probably a bit below that level of depth (and we spend a lot more time than many others) and GiveWell’s CEAs have no dedicated write-ups. Would either of these research projects be suitable for the Unjournal?
Would need to look into specific cases. My guess is that your work largely would be, at least 1. If and when we launch the second stream and 2. For the more in-depth stuff that you might think “I could submit this to a conventional journal but it’s too much hassle”.
GiveWell’s CEAs have no dedicated write-ups.
I think they should have more dedicated writeups (or other presentation formats), and perhaps more transparent formats, with clear reasoning transparent justifications for their choices, and robustness calculations, etc. Their recent contest goes in the right direction, though.
In terms of ‘weird formats’ it depends what you mean by weird. We are eager to evaluate work that is not the typical ‘frozen pdf prison’ but is presented (e.g.) as a web site offering foldable explanations for reasoning transparency. And in particular, open-science friendly dynamic documents where the code (or calculations) producing each results can be clearly unfolded, with a clear data pipeline, and where all results can be replicated. This would an improvement over the current journal formats: less prone to error, easier to check, easier to follow, easier to re-use, etc.
I agree that it’s a challenge, but I think that this is a change whose time has come. Most academics I’ve talk to individually think public evaluation/rating would be better than the dominant (and inefficient) traditional journal system, but everyone thinks “I can’t go outside of this system on my own”. I outline how I think we might be able to crack this (collective action problem and inertia) HERE. (I should probably expand on this.). And I think that we (non-university linked EA researchers and funders) might be in a unique position to help solve this problem. And I think there would be considerable rewards and influence to ‘being the ones who changed this’. But still …
I agree this would be valuable, and might be an easier path to pursue. And there may indeed be some tradeoffs (time/attention). I want to continue to consider, discuss, and respond to this in more detail.
For now, some off-the-cuff justifications for the current path:
We can try the high-rigor path first and then could pivot or branch to the ‘bring rigorous evaluation to niche and less formal EA-policy stuff’ later. But I think the reverse ordering would be more difficult (first impressions what what).
IMO a lot of academic work is highly impactful, or could be made highly impactful with a bit of TLC.[1]
Related to that, I’m fairly sympathetic to the points made HERE about valuing expertise and rigor, and not always trying to reinvent the wheel. I think we could do a lot to connect academic/non-EA expertise with EA-aligned goals.
My background/ideas and the setup of The Unjournal might be more suited to doing the ‘make rigorous research more impactful’ (and improving research evaluation) part?
(I’ll respond to your final point in another comment, as it’s fairly distinct)
Reporting the right outcomes and CEAs, more reasoning-transparency, offering tools/sharing data and models, etc.
Thanks, great reply!