There’s a lot here, and it will take me some time to think about. It seems like you’re coming at this from the lens of the pairwise comparison literature. I was coming at this from the lens of (what I think is) simpler expected value maximization foundations.
I’ve spent some time trying to understand the pairwise comparison literature, but haven’t gotten very fair. What I’ve seen has been focused very much on (what seems to me) like narrow elicitation procedures. As you stated, I’m more focused on representation.
“Table of value ratios” are meant to be a natural extension of “big lists of expected values”.
You could definitely understand a “list of expected value estimates” to be a function that helps convey certain preferences, but it’s a bit of an unusual bridge, outside the pairwise comparison literature.
On Contexts
You spend a while expressing the importance of clear contexts. I agree that precise contexts are important. It’s possible that the $1 example I used was a bit misleading—the point I was trying to make is that many value ratios will be less sensitive to changes context, then absolute values (the typical alternative, in expected value theory) would be.
Valuing V($5)/V($1) should give fairly precise results, for people of many different income levels. This wouldn’t be the case if you tried converting dollars to a common unit of QALYs or something first, before dividing.
Now, I could definitely see people from the discrete choice literature saying, “of course you shouldn’t first convert to QALYs, instead you should use better mathematical abstractions to represent direct preferences”. In that case I’d agree, there’s just a somewhat pragmatic set of choices about which abstractions give a good fit of practicality and specificity. I would be very curious if people from this background would suggest other approaches to large-scale, collaborative, estimation, as I’m trying to achieve here.
I would expect that with Relative Value estimation, as with EV estimation, we’d generally want precise definitions of things, especially if they were meant as forecasting questions. But “precise definitions” could mean “a precise set of different contexts”. Like, “What is the expected value of $1, as judged by 5 random EA Forum readers, for themselves?”
The only piece of literature I had in mind was von Neumann and Morgenstern’s representation theorem. It says: if you have a set of probability distributions over a set of outcomes and for each pair of distributions you have a preference (one is better than the other, or they are equal) and if this relation satisfies the additional requirements of transitivity, continuity and independence from alternatives, then you can represent the preferences with a utility function unique up to affine transformation.
Given that this is a foundational result for expected utility theory, I don’t think it is unusual to think of a utility function as a representation of a preference relation.
Do you envision your value ratio table to be underwritten by a unique utility function? That is, could we assign a single number V(x) to every outcome x such that the table cell corresponding to three outcomes pair (x,y) is always equal to V(x)/V(y)? These utilities could be treated as noisy estimates, which allows for correlations between V(x) and V(y) for some pairs.
My remarks concern what a value ratio table might be if it is more than just a “visualisation” of a utility function.
The value ratio table, as shown, is a presentation/visualization of the utility function (assuming you have joint distributions).
The key question is how to store the information within the utility function.
It’s really messy to try to store meaningful joint distributions in regular ways, especially if you want to approximate said distributions using multiple pieces. It’s especially to do this with multiple people, because then they would need to coordinate to ensure they are using the right scales.
The value ratio functions are basically one specific way to store/organize and think about this information. I think this is feasible to work with, in order to approximate large utility functions without too many trade-offs.
“Joint distributions on values where the scales are arbitrary” seem difficult to intuit/understand, so I think that typically representing them as ratios is a useful practice.
So constructing a value ratio table means estimating a joint distribution of values from a subset of pairwise comparisons, then sampling from the distribution to fill out the table?
In that case, I think estimating the distribution is the hard part. Your example is straightforward because it features independent estimates, or simple functional relationships.
Estimation is actually pretty easy (using linear regression), and is essentially a solved problem since 1952. Scheffé, H. (1952). An Analysis of Variance for Paired Comparisons. Journal of the American Statistical Association, 47(259), 381–400. https://doi.org/10.1080/01621459.1952.10501179
I wrote about the methodology (before finding Scheffé′s paper) here.
I can see how this gets you E(valuei|comparisons) for each each item i, but not P((valuei)i∈items|comparisons). One of the advantages Ozzie raises is the possibility to keep track of correlations in value estimates, which requires more than the marginal expectations.
I’m not sure what you mean. I’m thinking about pairwise comparisons in the following way.
(a) Every pair of items i,j have a true ratio of expectations E(Xi)/E(Xj)=μij. I hope this is uncontroversial.
(b) We observe the variables Rij according to logRij∼logμij+ϵij for some some normally distributed ϵij. Error terms might be dependent, but that complicates the analysis. (And is most likely not worth it.) This step could be more controversial, as there are other possible models to use.
Note that you will get a distribution over every E(Xi) too with this approach, but that would be in the Bayesian sense, i.e., p(E(Xi)∣comparisons), when we have a prior over E(Xi).
There’s a lot here, and it will take me some time to think about. It seems like you’re coming at this from the lens of the pairwise comparison literature. I was coming at this from the lens of (what I think is) simpler expected value maximization foundations.
I’ve spent some time trying to understand the pairwise comparison literature, but haven’t gotten very fair. What I’ve seen has been focused very much on (what seems to me) like narrow elicitation procedures. As you stated, I’m more focused on representation.
“Table of value ratios” are meant to be a natural extension of “big lists of expected values”.
You could definitely understand a “list of expected value estimates” to be a function that helps convey certain preferences, but it’s a bit of an unusual bridge, outside the pairwise comparison literature.
On Contexts
You spend a while expressing the importance of clear contexts. I agree that precise contexts are important. It’s possible that the $1 example I used was a bit misleading—the point I was trying to make is that many value ratios will be less sensitive to changes context, then absolute values (the typical alternative, in expected value theory) would be.
Valuing V($5)/V($1) should give fairly precise results, for people of many different income levels. This wouldn’t be the case if you tried converting dollars to a common unit of QALYs or something first, before dividing.
Now, I could definitely see people from the discrete choice literature saying, “of course you shouldn’t first convert to QALYs, instead you should use better mathematical abstractions to represent direct preferences”. In that case I’d agree, there’s just a somewhat pragmatic set of choices about which abstractions give a good fit of practicality and specificity. I would be very curious if people from this background would suggest other approaches to large-scale, collaborative, estimation, as I’m trying to achieve here.
I would expect that with Relative Value estimation, as with EV estimation, we’d generally want precise definitions of things, especially if they were meant as forecasting questions. But “precise definitions” could mean “a precise set of different contexts”. Like, “What is the expected value of $1, as judged by 5 random EA Forum readers, for themselves?”
The only piece of literature I had in mind was von Neumann and Morgenstern’s representation theorem. It says: if you have a set of probability distributions over a set of outcomes and for each pair of distributions you have a preference (one is better than the other, or they are equal) and if this relation satisfies the additional requirements of transitivity, continuity and independence from alternatives, then you can represent the preferences with a utility function unique up to affine transformation.
Given that this is a foundational result for expected utility theory, I don’t think it is unusual to think of a utility function as a representation of a preference relation.
Do you envision your value ratio table to be underwritten by a unique utility function? That is, could we assign a single number V(x) to every outcome x such that the table cell corresponding to three outcomes pair (x,y) is always equal to V(x)/V(y)? These utilities could be treated as noisy estimates, which allows for correlations between V(x) and V(y) for some pairs.
My remarks concern what a value ratio table might be if it is more than just a “visualisation” of a utility function.
The value ratio table, as shown, is a presentation/visualization of the utility function (assuming you have joint distributions).
The key question is how to store the information within the utility function.
It’s really messy to try to store meaningful joint distributions in regular ways, especially if you want to approximate said distributions using multiple pieces. It’s especially to do this with multiple people, because then they would need to coordinate to ensure they are using the right scales.
The value ratio functions are basically one specific way to store/organize and think about this information. I think this is feasible to work with, in order to approximate large utility functions without too many trade-offs.
“Joint distributions on values where the scales are arbitrary” seem difficult to intuit/understand, so I think that typically representing them as ratios is a useful practice.
So constructing a value ratio table means estimating a joint distribution of values from a subset of pairwise comparisons, then sampling from the distribution to fill out the table?
In that case, I think estimating the distribution is the hard part. Your example is straightforward because it features independent estimates, or simple functional relationships.
Estimation is actually pretty easy (using linear regression), and is essentially a solved problem since 1952. Scheffé, H. (1952). An Analysis of Variance for Paired Comparisons. Journal of the American Statistical Association, 47(259), 381–400. https://doi.org/10.1080/01621459.1952.10501179
I wrote about the methodology (before finding Scheffé′s paper) here.
I can see how this gets you E(valuei|comparisons) for each each item i, but not P((valuei)i∈items|comparisons). One of the advantages Ozzie raises is the possibility to keep track of correlations in value estimates, which requires more than the marginal expectations.
I’m not sure what you mean. I’m thinking about pairwise comparisons in the following way.
(a) Every pair of items i,j have a true ratio of expectations E(Xi)/E(Xj)=μij. I hope this is uncontroversial. (b) We observe the variables Rij according to logRij∼logμij+ϵij for some some normally distributed ϵij. Error terms might be dependent, but that complicates the analysis. (And is most likely not worth it.) This step could be more controversial, as there are other possible models to use.
Note that you will get a distribution over every E(Xi) too with this approach, but that would be in the Bayesian sense, i.e., p(E(Xi)∣comparisons), when we have a prior over E(Xi).