How can you estimate the value of research output? You could use pairwise comparisons, e.g., to ask specialists how much more valuable Darwin’s The Original of Species is than Dembski’s Intelligent Design. Then you can use these relative valuations to estimate absolute valuations.
Summary
Estimating values is hard. One way to elicit value estimates is ask researchers to compare two different items A and B, asking how much better A is than B. This makes the problem more concrete than just asking “what is the value of A?”. The Quantified Uncertainty Institute has made an app for doing this kind of thing, described here.
Nuño Sempere had a post about eliciting comparisons of research value from 6 effective altruism researchers. This is a more recent post about AI risk, but it uses distributions instead of point estimates.
This post proposes some technical solutions to problems introduced to me in Nuño’s post. In particular, it includes principled ways to
estimate subjective values,
measure consistency in pairwise value judgments,
measure agreement between the raters,
aggregate subjective values.
I also propose to use weighted least squares when the raters supply distributions instead of numbers. It is not clear to me it is worth it to ask for distributions in these kinds of questions though, as your uncertainty level can be modelled implicitly by comparing different pairwise comparisons.
Estimating value from pairwise comparisons
Link post
How can you estimate the value of research output? You could use pairwise comparisons, e.g., to ask specialists how much more valuable Darwin’s The Original of Species is than Dembski’s Intelligent Design. Then you can use these relative valuations to estimate absolute valuations.
Summary
Estimating values is hard. One way to elicit value estimates is ask researchers to compare two different items A and B, asking how much better A is than B. This makes the problem more concrete than just asking “what is the value of A?”. The Quantified Uncertainty Institute has made an app for doing this kind of thing, described here.
Nuño Sempere had a post about eliciting comparisons of research value from 6 effective altruism researchers. This is a more recent post about AI risk, but it uses distributions instead of point estimates.
This post proposes some technical solutions to problems introduced to me in Nuño’s post. In particular, it includes principled ways to
estimate subjective values,
measure consistency in pairwise value judgments,
measure agreement between the raters,
aggregate subjective values.
I also propose to use weighted least squares when the raters supply distributions instead of numbers. It is not clear to me it is worth it to ask for distributions in these kinds of questions though, as your uncertainty level can be modelled implicitly by comparing different pairwise comparisons.
I use these methods on the data from the 6 researchers post.
I’m assuming you have read the 6 researchers post recently. I think this post will be hard to read if you haven’t.