Doing deep evaluation of some of this without getting more empirical data (for instance, surveying people to see which ones might have taken this advice, or having probing conversations with Guesstimate users) seems necessary to get a decent picture.
(I assume you mean something like “and getting more empirical data”, not “without”?)
I think it’d indeed be interesting and useful to combine the sort of intuitive estimation approach Nuño adopted here with some gathering of empirical data. Nuño (or whoever) could perhaps randomly select a subset of posts/outputs to gather empirical data on, to reduce how time-consuming/costly the data collection is.
Two potential methods of data collection that come to mind are:
E.g., Rethink do “structured interviews with key decision-makers and leaders at EA organizations[, and seek] interviewees’ feedback on the general importance of our work for them and for the community, what they have and have not found helpful in what we’ve done, what we can do in the future that would be useful for them, and ways we can improve.”
In fact, one possibility would be to use the intuitive estimation approach on the work of one of the orgs/people who already have a bunch of this sort of data relevant to that work (after checking that the org/people are happy to have their work used for this process), and then look at the empirical data, and see how they compare.
(I recently started working for Rethink, but the views in this comment are just my personal views.)
In fact, one possibility would be to use the intuitive estimation approach on the work of one of the orgs/people who already have a bunch of this sort of data relevant to that work (after checking that the org/people are happy to have their work used for this process), and then look at the empirical data, and see how they compare.
This seems like a neat idea to me. We’ll investigate it.
(I assume you mean something like “and getting more empirical data”, not “without”?)
I think it’d indeed be interesting and useful to combine the sort of intuitive estimation approach Nuño adopted here with some gathering of empirical data. Nuño (or whoever) could perhaps randomly select a subset of posts/outputs to gather empirical data on, to reduce how time-consuming/costly the data collection is.
Two potential methods of data collection that come to mind are:
Surveys
E.g., Rethink Priorities’ “impact survey”, my survey which was inspired by Rethink’s, 80k’s annual survey, and a recent Happier Lives Institute survey
Some extra discussion here
Interviews
E.g., Rethink do “structured interviews with key decision-makers and leaders at EA organizations[, and seek] interviewees’ feedback on the general importance of our work for them and for the community, what they have and have not found helpful in what we’ve done, what we can do in the future that would be useful for them, and ways we can improve.”
In fact, one possibility would be to use the intuitive estimation approach on the work of one of the orgs/people who already have a bunch of this sort of data relevant to that work (after checking that the org/people are happy to have their work used for this process), and then look at the empirical data, and see how they compare.
(I recently started working for Rethink, but the views in this comment are just my personal views.)
That’s quite useful, thanks
This seems like a neat idea to me. We’ll investigate it.