I think that inviting submissions from research in preprints or written up in EA forum posts is a good idea.
Definitely the former, but which ones? As to EA forum posts, I guess they mainly (with some exceptions) don’t have the sort of rigor that would permit the kind of review we want… And that would help unjournal evaluations become a replacement for academic journal publications?
A submission would simply be giving you permission to publish/host the original document and reviews in the unjournal. Post review, authors could have the option to provide a link to their revised output, or to add a comment.
Actually, I don’t propose to host or publish anything. Just linking a page with a DOI … and review it based on this, no?
I think I should go through this carefully, for sure.
One that comes to mind now is “How valuable is movement growth?”
I think reviewing EA forum posts is very valuable, but this is a separate thing from what I’m trying to do with Unjournal. If we include this in the ‘same batch of things’ it would probably drive away the academics and very serious researchers, no?
At least for me, The Awareness/Inclination Model (AIM) in this seemed, for a while, to be a popular theory influencing how EA people thought about movement building.
A review of it this would help with understanding how confident we should be (or have been) in the empirical data presented for these arguments made, and might also throw up some ideas to build on it or test it.
Again, I don’t think it has been written up in a way that is aiming at rigorous peer review, is it?
Finally, I think that treating the first few rounds of doing this as being an experiment is probably a good idea. It might be the case that only a certain type of paper/output works, or that reviews are more useful than you imagined even for relatively low level research. Probably hard to tell until you do a few rounds of the process.
I think you might be right. I should dive in and be less delicate with this. Partial excuse for slowness so far: I’m waiting for a grant to come through that should give me some more hours to work on this (fingers crossed)
I think that inviting submissions from research in preprints or written up in EA forum posts is a good idea.
Definitely the former, but which ones?
>PS> Yeah, the only easy options I can suggest now are to consider some of items in the BS newsletter
As to EA forum posts, I guess they mainly (with some exceptions) don’t have the sort of rigor that would permit the kind of review we want… And that would help unjournal evaluations become a replacement for academic journal publications?
>PS> This is probably a bigger discussion, but this makes me realise that that one difference between us is that I probably want the unjournal (and social science in general) to accept a lower level of rigor than most journal (perhaps somewhere between a very detailed forum/blog post and a short journal or conference article).
One reason is that I personally think that most social science journal articles sacrifice too much speed for better quality, given heterogeneity etc. I’d prefer maybe 10x as much research, at .1x the quality. To be clear, I am keen on keeping the key parts (e.g., a good method and explanation of theory and findings), but not having so much of the fluff (e.g., summarising much prior or future potential research etc).
A second is that I expect a lot more submissions near the level of conference work or a detailed forum post level, than will be journal level. There are probably 100x more forum posts and reports produced than journal articles. Additionally, there is a lot of competition for journal level submissions. If you expect an article to get accepted at a journal then you will probably submit it to one. On the other hand, if you wrote up a report pretty close to journal level in some regards and have nowhere to put it, or no patience with the demand of a journal or uncertainty, then the unjournal is relatively attractive given the lack of alternatives.
A submission would simply be giving you permission to publish/host the original document and reviews in the unjournal. Post review, authors could have the option to provide a link to their revised output, or to add a comment.
Actually, I don’t propose to host or publish anything. Just linking a page with a DOI … and review it based on this, no?
I think I should go through this carefully, for sure.
One that comes to mind now is “How valuable is movement growth?”
I think reviewing EA forum posts is very valuable, but this is a separate thing from what I’m trying to do with Unjournal. If we include this in the ‘same batch of things’ it would probably drive away the academics and very serious researchers, no?
>PS> Yeah, so that’s a good point. I think that it gets into the points above. Perhaps you can have different types of submissions (e.g., work in progress, opinion etc?) .You could treat it like some other journals have and scale up expectations over time once it starts getting known?
At least for me, The Awareness/Inclination Model (AIM) in this seemed, for a while, to be a popular theory influencing how EA people thought about movement building.
A review of it this would help with understanding how confident we should be (or have been) in the empirical data presented for these arguments made, and might also throw up some ideas to build on it or test it.
Again, I don’t think it has been written up in a way that is aiming at rigorous peer review, is it?
>PS> Perhaps not. Maybe that’s something that authors need to answer. Regardless, I think that there would be a lot of value in these sort of reports getting peer reviewed by academics/experts, especially where they are influential in the EA community.
Finally, I think that treating the first few rounds of doing this as being an experiment is probably a good idea. It might be the case that only a certain type of paper/output works, or that reviews are more useful than you imagined even for relatively low level research. Probably hard to tell until you do a few rounds of the process.
I think you might be right. I should dive in and be less delicate with this. Partial excuse for slowness so far: I’m waiting for a grant to come through that should give me some more hours to work on this (fingers crossed)
>PS> I think you are doing a good job and I am not sure I am giving good advice! However, it could be the case that you want to use this process to test some assumptions and processes (e.g., about how many people will submit, what sorts of articles you will get, how long things will take, how best to show outputs) etc.
Regardless, I think that there would be a lot of value in these sort of reports getting peer reviewed by academics/experts, especially where they are influential in the EA community.
I agree, but I don’t think this is what the Unjournal should handle right now. It should be done, but maybe with a different vehicle and approach.
I’d prefer maybe 10x as much research, at .1x the quality.
I tend to disagree with this. My concern is that most/much research is often ‘vomited out’ to satisfy tenure requirements and other needs to “publish something”. There is just so much research and writing out their to wade through.
I typically question...
Are the empirical results trustworthy? Can we have confidence in their validity and generalizability … and the interpretation the authors give?
Is anyone reading these papers (and if so, are they understanding them or just skimming and getting the wrong impressions)?
Are these being combined with other work, and with replication, to give a general picture of ‘what we know and with what confidence’? Is it entering into a general building of our knowledge and ability to learn more and use the work?
When I say I’d prefer maybe 10x as much research, at .1x the quality, I don’t want to miss out on quality overall. Instead, I’d like more small scale incremental and iterative research, where the rigour and the length, increase in proportion to the expected ROI. For instance, this could involve a range of small studies that increase in quality as they show evidence, followed by a rigorous review and replication process.
I also think that the reason for a lot of the current research vomit is that we don’t let people publish short and simple articles. I think that if you took most articles and pulled out their method, results and conclusion, you would give the reader about 95% of the value of the article in maybe 1/10th the space/words of the full article.
If a researcher just had to write these sections and a wrapper rather than plan and coordinate a whole document, they might produce and disseminate their insights in 2-5% of the time that it currently takes.
Definitely the former, but which ones? As to EA forum posts, I guess they mainly (with some exceptions) don’t have the sort of rigor that would permit the kind of review we want… And that would help unjournal evaluations become a replacement for academic journal publications?
Actually, I don’t propose to host or publish anything. Just linking a page with a DOI … and review it based on this, no?
I think I should go through this carefully, for sure.
I think reviewing EA forum posts is very valuable, but this is a separate thing from what I’m trying to do with Unjournal. If we include this in the ‘same batch of things’ it would probably drive away the academics and very serious researchers, no?
Again, I don’t think it has been written up in a way that is aiming at rigorous peer review, is it?
I think you might be right. I should dive in and be less delicate with this. Partial excuse for slowness so far: I’m waiting for a grant to come through that should give me some more hours to work on this (fingers crossed)
Thanks for the responses!
See >PS>
Definitely the former, but which ones?
>PS> Yeah, the only easy options I can suggest now are to consider some of items in the BS newsletter
As to EA forum posts, I guess they mainly (with some exceptions) don’t have the sort of rigor that would permit the kind of review we want… And that would help unjournal evaluations become a replacement for academic journal publications?
>PS> This is probably a bigger discussion, but this makes me realise that that one difference between us is that I probably want the unjournal (and social science in general) to accept a lower level of rigor than most journal (perhaps somewhere between a very detailed forum/blog post and a short journal or conference article).
One reason is that I personally think that most social science journal articles sacrifice too much speed for better quality, given heterogeneity etc. I’d prefer maybe 10x as much research, at .1x the quality. To be clear, I am keen on keeping the key parts (e.g., a good method and explanation of theory and findings), but not having so much of the fluff (e.g., summarising much prior or future potential research etc).
A second is that I expect a lot more submissions near the level of conference work or a detailed forum post level, than will be journal level. There are probably 100x more forum posts and reports produced than journal articles. Additionally, there is a lot of competition for journal level submissions. If you expect an article to get accepted at a journal then you will probably submit it to one. On the other hand, if you wrote up a report pretty close to journal level in some regards and have nowhere to put it, or no patience with the demand of a journal or uncertainty, then the unjournal is relatively attractive given the lack of alternatives.
Actually, I don’t propose to host or publish anything. Just linking a page with a DOI … and review it based on this, no?
>PS> Yeah, sounds good.
I think I should go through this carefully, for sure.
I think reviewing EA forum posts is very valuable, but this is a separate thing from what I’m trying to do with Unjournal. If we include this in the ‘same batch of things’ it would probably drive away the academics and very serious researchers, no?
>PS> Yeah, so that’s a good point. I think that it gets into the points above. Perhaps you can have different types of submissions (e.g., work in progress, opinion etc?) .You could treat it like some other journals have and scale up expectations over time once it starts getting known?
Again, I don’t think it has been written up in a way that is aiming at rigorous peer review, is it?
>PS> Perhaps not. Maybe that’s something that authors need to answer. Regardless, I think that there would be a lot of value in these sort of reports getting peer reviewed by academics/experts, especially where they are influential in the EA community.
I think you might be right. I should dive in and be less delicate with this. Partial excuse for slowness so far: I’m waiting for a grant to come through that should give me some more hours to work on this (fingers crossed)
>PS> I think you are doing a good job and I am not sure I am giving good advice! However, it could be the case that you want to use this process to test some assumptions and processes (e.g., about how many people will submit, what sorts of articles you will get, how long things will take, how best to show outputs) etc.
Thanks for your work on it!
I agree on most of your counts.
I agree, but I don’t think this is what the Unjournal should handle right now. It should be done, but maybe with a different vehicle and approach.
I tend to disagree with this. My concern is that most/much research is often ‘vomited out’ to satisfy tenure requirements and other needs to “publish something”. There is just so much research and writing out their to wade through.
I typically question...
Are the empirical results trustworthy? Can we have confidence in their validity and generalizability … and the interpretation the authors give?
Is anyone reading these papers (and if so, are they understanding them or just skimming and getting the wrong impressions)?
Are these being combined with other work, and with replication, to give a general picture of ‘what we know and with what confidence’? Is it entering into a general building of our knowledge and ability to learn more and use the work?
Thanks for replying.
When I say I’d prefer maybe 10x as much research, at .1x the quality, I don’t want to miss out on quality overall. Instead, I’d like more small scale incremental and iterative research, where the rigour and the length, increase in proportion to the expected ROI. For instance, this could involve a range of small studies that increase in quality as they show evidence, followed by a rigorous review and replication process.
I also think that the reason for a lot of the current research vomit is that we don’t let people publish short and simple articles. I think that if you took most articles and pulled out their method, results and conclusion, you would give the reader about 95% of the value of the article in maybe 1/10th the space/words of the full article.
If a researcher just had to write these sections and a wrapper rather than plan and coordinate a whole document, they might produce and disseminate their insights in 2-5% of the time that it currently takes.