Yeah, I have thought that it would probably be nice to find a field where it would be valuable (based on how much the field is struggling with these issues X the importance of the research), but I’ve also wondered if it might be best to first look for a field that has a fitting/acceptive ethos—i.e., a field where a lot of researchers are open to trying the idea. (Of course, that would raise questions about whether it could see similar buy-in when applied to different fields, but the purpose of such an early test would be to identify “how useful is this when there is buy-in?”)
At the same time, I have also recognized that it would probably be difficult… although I do wonder just how difficult it would be—or at least, why exactly it might be difficult. Especially if the problem is mainly about buy-in, I have thought that it would probably be helpful to look at similar movements like the shift towards peer-reviewing as well as the push for open data/data transparency: how did they convince journals/researchers to be more transparent and collaborative? If this system actually proved useful and feasible, I feel like it might have a decent chance of eventually getting traction (even if it may go slow).
The main concern I’ve had with the broader pipe dream I hinted at has been “who does the mapping/manages the systems?” Are the maps run by centralized authorities like journals or scientific associations (e.g., the APA), or is it mostly decentralized in that objects in the literature (individual studies, datasets, regressions, findings) have centrally-defined IDs but all of the connections (e.g., “X finding depends on Y dataset”, “X finding conflicts with Z finding”) are defined by packages/layers that researchers can contribute to and download from, like a library/buffet. (The latter option could allow “curation” by journals, scientific associations, or anyone else.) However, I think the narrower system I initially described would not suffer from this problem to the same extent—at least, the problems would not be more significant than those incurred with peer-review (since it is mainly just asking “1) does your research criticize another study? 2) What studies and datasets does your research rely on?”)
But I would definitely be interested to hear you elaborate on potential problems you see with the system. I have been interested in a project of this sort for years: I even did a small project last year to try the literature mapping (which had mixed-positive results in that it seemed potentially feasible/useful but I couldn’t find a great existing software platform to do both visually-nice mapping + rudimentary logic operations). I just can’t shake the desire to continue trying this/looking for research or commentary on the idea, but so far I really haven’t found all that much… which in some ways just makes me more interested in pursuing the idea (since that could suggest it’s a neglected idea… although it could also suggest that it’s been deemed impractical)
Yeah, I have thought that it would probably be nice to find a field where it would be valuable (based on how much the field is struggling with these issues X the importance of the research), but I’ve also wondered if it might be best to first look for a field that has a fitting/acceptive ethos—i.e., a field where a lot of researchers are open to trying the idea. (Of course, that would raise questions about whether it could see similar buy-in when applied to different fields, but the purpose of such an early test would be to identify “how useful is this when there is buy-in?”)
At the same time, I have also recognized that it would probably be difficult… although I do wonder just how difficult it would be—or at least, why exactly it might be difficult. Especially if the problem is mainly about buy-in, I have thought that it would probably be helpful to look at similar movements like the shift towards peer-reviewing as well as the push for open data/data transparency: how did they convince journals/researchers to be more transparent and collaborative? If this system actually proved useful and feasible, I feel like it might have a decent chance of eventually getting traction (even if it may go slow).
The main concern I’ve had with the broader pipe dream I hinted at has been “who does the mapping/manages the systems?” Are the maps run by centralized authorities like journals or scientific associations (e.g., the APA), or is it mostly decentralized in that objects in the literature (individual studies, datasets, regressions, findings) have centrally-defined IDs but all of the connections (e.g., “X finding depends on Y dataset”, “X finding conflicts with Z finding”) are defined by packages/layers that researchers can contribute to and download from, like a library/buffet. (The latter option could allow “curation” by journals, scientific associations, or anyone else.) However, I think the narrower system I initially described would not suffer from this problem to the same extent—at least, the problems would not be more significant than those incurred with peer-review (since it is mainly just asking “1) does your research criticize another study? 2) What studies and datasets does your research rely on?”)
But I would definitely be interested to hear you elaborate on potential problems you see with the system. I have been interested in a project of this sort for years: I even did a small project last year to try the literature mapping (which had mixed-positive results in that it seemed potentially feasible/useful but I couldn’t find a great existing software platform to do both visually-nice mapping + rudimentary logic operations). I just can’t shake the desire to continue trying this/looking for research or commentary on the idea, but so far I really haven’t found all that much… which in some ways just makes me more interested in pursuing the idea (since that could suggest it’s a neglected idea… although it could also suggest that it’s been deemed impractical)