1) I agree that replication work is also vital. It seems to me that a better equilibrium compared to what we have now would both incentivize novelty and replication more than they do currently. Perhaps this entails two different metrics, which would both form part of an overall “scorecard” (the sabermetrics approach favoured by the cited paper). 2) This is likely to be one of the constraints yes. Would this also apply to secondary citations?
As for your comment—indeed,starting with a specific field would be a reasonable first step. Any idea which EA-relevant field would suffer the most from a lack of novel research? Biorisks perhaps? (I would assume AI research suffers less from h-index incentives due to economic incentives, though I may be wrong).
I think secondary citations would be easier like you say. And you wouldn’t have to stop there—once you have the citation data, you could probably do a lot of creative things analysing the resulting graphs (graphs in the mathematical sense). I expect it’s where the input data is harder to reach and scrape (like whole text) that logistical worries enter.
Yeah I don’t know! I’m sure there some folks who have thought about meta-science/improving science etc. that might have good ideas.
Good questions!
1) I agree that replication work is also vital. It seems to me that a better equilibrium compared to what we have now would both incentivize novelty and replication more than they do currently. Perhaps this entails two different metrics, which would both form part of an overall “scorecard” (the sabermetrics approach favoured by the cited paper).
2) This is likely to be one of the constraints yes. Would this also apply to secondary citations?
As for your comment—indeed,starting with a specific field would be a reasonable first step. Any idea which EA-relevant field would suffer the most from a lack of novel research? Biorisks perhaps? (I would assume AI research suffers less from h-index incentives due to economic incentives, though I may be wrong).
I think secondary citations would be easier like you say. And you wouldn’t have to stop there—once you have the citation data, you could probably do a lot of creative things analysing the resulting graphs (graphs in the mathematical sense). I expect it’s where the input data is harder to reach and scrape (like whole text) that logistical worries enter.
Yeah I don’t know! I’m sure there some folks who have thought about meta-science/improving science etc. that might have good ideas.