Thanks, this is interesting! 2 questions and a comment:
1) Would a novelty-focused metric trade off against replication work?
2) Would resource constraints matter for choice of metric? I’m thinking that some metrics are computationally/logistically easier to gather and maintain (e.g. pre-existing citation databases), and the cost/bother of performing textual analysis to some depth of the volumes of relevant literature might be non-negligible.
Comment: My impression from reading some Wikipedia articles (https://en.wikipedia.org/wiki/H-index , https://en.wikipedia.org/wiki/Citation_impact , https://en.wikipedia.org/wiki/Citation_analysis ) is that there are lots of proposals for different metrics, but a common theme of criticism is the difficulty of comparing between disciplines, where field-dependent factors are critical to a metric being meaningful/useful. If this is the case, maybe a smaller version of this project would be to pick a particularly important field to EAs, and see if targeted analysis/work can propose a more relevant metric for it.
1) I agree that replication work is also vital. It seems to me that a better equilibrium compared to what we have now would both incentivize novelty and replication more than they do currently. Perhaps this entails two different metrics, which would both form part of an overall “scorecard” (the sabermetrics approach favoured by the cited paper). 2) This is likely to be one of the constraints yes. Would this also apply to secondary citations?
As for your comment—indeed,starting with a specific field would be a reasonable first step. Any idea which EA-relevant field would suffer the most from a lack of novel research? Biorisks perhaps? (I would assume AI research suffers less from h-index incentives due to economic incentives, though I may be wrong).
I think secondary citations would be easier like you say. And you wouldn’t have to stop there—once you have the citation data, you could probably do a lot of creative things analysing the resulting graphs (graphs in the mathematical sense). I expect it’s where the input data is harder to reach and scrape (like whole text) that logistical worries enter.
Yeah I don’t know! I’m sure there some folks who have thought about meta-science/improving science etc. that might have good ideas.
Thanks, this is interesting! 2 questions and a comment:
1) Would a novelty-focused metric trade off against replication work?
2) Would resource constraints matter for choice of metric? I’m thinking that some metrics are computationally/logistically easier to gather and maintain (e.g. pre-existing citation databases), and the cost/bother of performing textual analysis to some depth of the volumes of relevant literature might be non-negligible.
Comment:
My impression from reading some Wikipedia articles (https://en.wikipedia.org/wiki/H-index , https://en.wikipedia.org/wiki/Citation_impact , https://en.wikipedia.org/wiki/Citation_analysis ) is that there are lots of proposals for different metrics, but a common theme of criticism is the difficulty of comparing between disciplines, where field-dependent factors are critical to a metric being meaningful/useful. If this is the case, maybe a smaller version of this project would be to pick a particularly important field to EAs, and see if targeted analysis/work can propose a more relevant metric for it.
Good questions!
1) I agree that replication work is also vital. It seems to me that a better equilibrium compared to what we have now would both incentivize novelty and replication more than they do currently. Perhaps this entails two different metrics, which would both form part of an overall “scorecard” (the sabermetrics approach favoured by the cited paper).
2) This is likely to be one of the constraints yes. Would this also apply to secondary citations?
As for your comment—indeed,starting with a specific field would be a reasonable first step. Any idea which EA-relevant field would suffer the most from a lack of novel research? Biorisks perhaps? (I would assume AI research suffers less from h-index incentives due to economic incentives, though I may be wrong).
I think secondary citations would be easier like you say. And you wouldn’t have to stop there—once you have the citation data, you could probably do a lot of creative things analysing the resulting graphs (graphs in the mathematical sense). I expect it’s where the input data is harder to reach and scrape (like whole text) that logistical worries enter.
Yeah I don’t know! I’m sure there some folks who have thought about meta-science/improving science etc. that might have good ideas.