I’ve been thinking about this sort of preference aggregation problem for a few years. I think the best way to do it, that we have right now, is to form a graph with edges weighted by the comparison strength, then do pagerank, or something like it. Rank entries by their pagerank scores.
But I’ve been working towards something more precise, and this might be novel: Parallel and serial reducer functions (and another one, a “crosslink” reducer, I believe), sort of like the reducer functions you’d use to make judgements about electronic circuit graphs, which, given two nodes in the graph, work together to reduce the edges of the graph to a single edge between them, and then you know how those edges compare.
I have a really firm and consistent and unambiguous sense of how the reducers should behave, but I’d need to spend some time with a mathematician and a whiteboard to come up with formalizations. I’m pretty confident we’d produce something legit, if that were set up, though!
In case you’re wondering how it handles cycles: If the cycle is even it resolves that each option in it is equal. If the cycle is a little bit lopsided then it creates a ranking, but with high controversy. If it’s extremely lopsided then it creates a ranking with low controversy. If you put a gun to its head, it’ll always be able to tell you which nodes are at the top of the ranking, but it can also tell you when there’s a lot of uncertainty between them.
My sense is that the mathematized version would be much more valuable (for instance, I could incorporate it into my tooling), but also harder to obtain than you might realize.
I dunno if it’s that hard. Comparisons are an old and very well-developed area of statistics, if only for use in tournaments, and you can find a ton of papers and code for pairwise comparisons. I have some & a R utility in a similar spirit on my Resorter page. Compared (ahem) to many problems, it’s pretty easy to get started with some Elo or Bradley-Terry-esque system and then work on nailing down your ordinal rankings into more cardinal stuff. This is something where the hard part is the UX/UI and tailoring to use-cases, and too much attention to the statistics may be wankery.
Comparisons are an old and very well-developed area of statistics
Yeah, but it’s not clear to me that discrete choice is a good fit for the kind of thing that I’m trying to do (though I’ve downloaded a few textbooks, and I’ll find out). I agree that UX is important.
I’ve been thinking about this sort of preference aggregation problem for a few years. I think the best way to do it, that we have right now, is to form a graph with edges weighted by the comparison strength, then do pagerank, or something like it. Rank entries by their pagerank scores.
But I’ve been working towards something more precise, and this might be novel: Parallel and serial reducer functions (and another one, a “crosslink” reducer, I believe), sort of like the reducer functions you’d use to make judgements about electronic circuit graphs, which, given two nodes in the graph, work together to reduce the edges of the graph to a single edge between them, and then you know how those edges compare.
I have a really firm and consistent and unambiguous sense of how the reducers should behave, but I’d need to spend some time with a mathematician and a whiteboard to come up with formalizations. I’m pretty confident we’d produce something legit, if that were set up, though!
In case you’re wondering how it handles cycles: If the cycle is even it resolves that each option in it is equal. If the cycle is a little bit lopsided then it creates a ranking, but with high controversy. If it’s extremely lopsided then it creates a ranking with low controversy.
If you put a gun to its head, it’ll always be able to tell you which nodes are at the top of the ranking, but it can also tell you when there’s a lot of uncertainty between them.
My sense is that the mathematized version would be much more valuable (for instance, I could incorporate it into my tooling), but also harder to obtain than you might realize.
I dunno if it’s that hard. Comparisons are an old and very well-developed area of statistics, if only for use in tournaments, and you can find a ton of papers and code for pairwise comparisons. I have some & a R utility in a similar spirit on my Resorter page. Compared (ahem) to many problems, it’s pretty easy to get started with some Elo or Bradley-Terry-esque system and then work on nailing down your ordinal rankings into more cardinal stuff. This is something where the hard part is the UX/UI and tailoring to use-cases, and too much attention to the statistics may be wankery.
Yeah, but it’s not clear to me that discrete choice is a good fit for the kind of thing that I’m trying to do (though I’ve downloaded a few textbooks, and I’ll find out). I agree that UX is important.