Thanks for your interest and also for raising this with us before you posted so I could post this response quickly!
Thanks for sharing the 1st version of your answer too, which prompted me to add a little more detail about what I was asking in the post.
If some of the additional notes below seem like they should be included thatād be helpful to hear.
I think it would be valuable to include all the additional notes which are not on your website. As a minimum viable product, you may want to link to your comment.
To reply to your specific question about aggregating peopleās personal rankings: no, we donāt do any formal sort of āvotingā system like that. The problems and paths rankings are informed by the views of the staff at 80,000 Hours and external advisors via surveys where I elicit peopleās personal rankings, and lots of ongoing internal discussion, but I am the āpoint personā for ultimately deciding how to combine this information into a ranking. In practice, this means my views can be expected to have an outsized influence, but I put a lot of emphasis on takes from others and aim for the lists to be something 80,000 Hours as an organisation can stand behind.
Thanks for sharing! The approach you are following seems to be analogous to what happens in the broader society, where there is often one single person responsible for informally aggregating various views. Using a formal aggregation method is the norm in forecasting circles. However, there are often many forecasts to be aggregated, so informal aggregation would hardly be feasible for most cases. On the other hand, Samotsvety, āa group of forecasters with a great track recordā, alsouses formal aggregation methods. I am not aware of research comparing informal to formal aggregation of a few forecasts, so there might not be a strong case either way. In any case, I encourage you to try formal aggregation to see if you arrive to meaningfully different results.
Another big factor is what the lists were before, which I tend to view as a prior to update from, and which were informed by the research we did in the past and the views of people like like Ben Todd, Howie Lempel, and Rob Wiblin.
Makes sense.
The rankings are not nearly as formal or quantitative as, e.g. the cost-effectiveness analyses that GiveWell performs of its top charities. Though previous versions of the site have included numerical weightings to something like the problem profiles list, weāve moved away from that practice. We didnāt think the BOTECs and estimations that generated these kinds of numbers were actually driving our views, and the numbers they produced seemed like they suggested a misleading sense of precision.
Your previous quantitative framework was equivalent to a weighted-factor model (WFM) with the logarithms of importance, tractability and neglectedness as factors with the same weight, such that the sum respects the logarithm of the cost-effectiveness. Have you considered trying a WFM with the factors that actually drive your views?
?I think it would be valuable to include all the additional notes which are not on your website. As a minimum viable product, you may want to link to your comment.
Thanks for your feedback here!
Your previous quantitative framework was equivalent to a weighted-factor model (WFM) with the logarithms of importance, tractability and neglectedness as factors with the same weight, such the sum respects the logarithm of the cost-effectiveness. Have you considered trying a WFM with the factors that actually drive your views?
I feel unsure about whether we should be trying to do another WFM at some point. There are a lot of ways we can improve our advice, and Iām not sure this should be at the top of our list but perhaps if/āwhen we have more research capacity. Iād also guess it would still have the problem of giving a misleading sense of precision, so itās not clear how much of an improvement it would be. But it is certainly true that the ITN framework substantially drives our views.
Thanks for the comprehensive reply, Arden!
Thanks for sharing the 1st version of your answer too, which prompted me to add a little more detail about what I was asking in the post.
I think it would be valuable to include all the additional notes which are not on your website. As a minimum viable product, you may want to link to your comment.
Thanks for sharing! The approach you are following seems to be analogous to what happens in the broader society, where there is often one single person responsible for informally aggregating various views. Using a formal aggregation method is the norm in forecasting circles. However, there are often many forecasts to be aggregated, so informal aggregation would hardly be feasible for most cases. On the other hand, Samotsvety, āa group of forecasters with a great track recordā, also uses formal aggregation methods. I am not aware of research comparing informal to formal aggregation of a few forecasts, so there might not be a strong case either way. In any case, I encourage you to try formal aggregation to see if you arrive to meaningfully different results.
Makes sense.
Your previous quantitative framework was equivalent to a weighted-factor model (WFM) with the logarithms of importance, tractability and neglectedness as factors with the same weight, such that the sum respects the logarithm of the cost-effectiveness. Have you considered trying a WFM with the factors that actually drive your views?
Thanks for your feedback here!
I feel unsure about whether we should be trying to do another WFM at some point. There are a lot of ways we can improve our advice, and Iām not sure this should be at the top of our list but perhaps if/āwhen we have more research capacity. Iād also guess it would still have the problem of giving a misleading sense of precision, so itās not clear how much of an improvement it would be. But it is certainly true that the ITN framework substantially drives our views.