Doing deep evaluation of some of this without getting more empirical data (for instance, surveying people to see which ones might have taken this advice, or having probing conversations with Guesstimate users) seems necessary to get a decent picture.
(I assume you mean something like âand getting more empirical dataâ, not âwithoutâ?)
I think itâd indeed be interesting and useful to combine the sort of intuitive estimation approach Nuño adopted here with some gathering of empirical data. Nuño (or whoever) could perhaps randomly select a subset of posts/âoutputs to gather empirical data on, to reduce how time-consuming/âcostly the data collection is.
Two potential methods of data collection that come to mind are:
E.g., Rethink do âstructured interviews with key decision-makers and leaders at EA organizations[, and seek] intervieweesâ feedback on the general importance of our work for them and for the community, what they have and have not found helpful in what weâve done, what we can do in the future that would be useful for them, and ways we can improve.â
In fact, one possibility would be to use the intuitive estimation approach on the work of one of the orgs/âpeople who already have a bunch of this sort of data relevant to that work (after checking that the org/âpeople are happy to have their work used for this process), and then look at the empirical data, and see how they compare.
(I recently started working for Rethink, but the views in this comment are just my personal views.)
In fact, one possibility would be to use the intuitive estimation approach on the work of one of the orgs/âpeople who already have a bunch of this sort of data relevant to that work (after checking that the org/âpeople are happy to have their work used for this process), and then look at the empirical data, and see how they compare.
This seems like a neat idea to me. Weâll investigate it.
(I assume you mean something like âand getting more empirical dataâ, not âwithoutâ?)
I think itâd indeed be interesting and useful to combine the sort of intuitive estimation approach Nuño adopted here with some gathering of empirical data. Nuño (or whoever) could perhaps randomly select a subset of posts/âoutputs to gather empirical data on, to reduce how time-consuming/âcostly the data collection is.
Two potential methods of data collection that come to mind are:
Surveys
E.g., Rethink Prioritiesâ âimpact surveyâ, my survey which was inspired by Rethinkâs, 80kâs annual survey, and a recent Happier Lives Institute survey
Some extra discussion here
Interviews
E.g., Rethink do âstructured interviews with key decision-makers and leaders at EA organizations[, and seek] intervieweesâ feedback on the general importance of our work for them and for the community, what they have and have not found helpful in what weâve done, what we can do in the future that would be useful for them, and ways we can improve.â
In fact, one possibility would be to use the intuitive estimation approach on the work of one of the orgs/âpeople who already have a bunch of this sort of data relevant to that work (after checking that the org/âpeople are happy to have their work used for this process), and then look at the empirical data, and see how they compare.
(I recently started working for Rethink, but the views in this comment are just my personal views.)
Thatâs quite useful, thanks
This seems like a neat idea to me. Weâll investigate it.