Thanks for clarifying your thinking! It’s great to see you being responsive to community feedback in this way; for instance, the discussion about low acceptance rates should help give job seekers a more accurate picture of the landscape.
I’ve got a few thoughts to share in the spirit of offering more constructive criticism that will hopefully driving further improvements.
1. After a lot of time spent by 80K and the EA community trying to interpret the results of this question, we’re at a place where:
we do not think the precise numbers are a reliable answer to decision-relevant questions for job seekers, funders, or potential employers. We think it’s likely that mistakes are driving up these estimates. Even ignoring the high probability of mistakes, the implications of the data depend heavily on exactly what is driving the results. We are very uncertain about the magnitude of various considerations, so we recommend against leaning on these numbers when making career decisions.
So why not just ditch this question in favor of something more helpful, instead of continuing to pour more resources into it? If you look at things from a cost-benefit perspective, this question clearly has a high cost due to all the clarification its required. Is the benefit high enough to warrant that? Even if you worked out all the kinks and knew all the respondents were thinking about the question the same way, and investing the time to produce an answer they trusted, what do we really learn by asking: “For a typical recent Senior/Junior hire, how much financial compensation would you need to receive today, to make you indifferent about that person having to stop working for you or anyone for the next 3 years?” Which parties would take which different actions on the margins because of that information?
2. I suspect one of the reasons why people are having trouble interpreting this question is because it doesn’t correspond to a real world decision people have to make. Confusing hypothetical questions are likely to produce non-actionable results. I suggest thinking carefully about exactly what you’re trying to capture (the value of an employee? The urgency and/or difficulty of hiring?) and looking at whether you can measure that dynamic via a clearly labeled Likert scale (e.g. a 1-7 scale rating with values like “employee is transformative in allowing us to do important new things”, “employee lets us do what we currently do, but a tiny bit better”, “it would be very easy to replace an employee of this type”, etc.) I’ve posted some specific suggestions in a separate comment.
And relatedly, I’d recommend as strongly as possible not using this question (or any question involving genies). I think its very likely to lead to the same sorts of interpretation problems.
Below is a very rough draft of one version of the question we are considering asking. We hope it would be more decision-relevant than the question used above, but we haven’t yet had time to pilot it or vet it for any issues:
Imagine that sometime in the next year you are about to hire your next junior (senior) hire. A genie appears and offers you the following choice. You can have one of the following:
1. The genie will create a person and applicant for the job from thin air. They will be as much more productive (in % terms) than the next best applicant in the pool, as your last junior (senior) hire appeared to be at the point when you were evaluating whether to hire them. This person will live out the rest of their life like any other staff member, and may well go on to do other useful work outside of your organisation later on. You should consider the benefits of that for the world as well.
2. The genie will distribute $X among whichever organisations or people you nominate – which can include you and your organisation – to be used to improve the world as much as possible. Consider all the benefits for the world this would generate.
At what value of X would you be indifferent between these two options?