Specific reactions to the evaluation of Takeaways from EAF’s Hiring Round
The way to estimate impact here would be something like: “Counterfactual impact of the best hire(s) in the organizations it influenced, as opposed to the impact of the hires who would otherwise have been chosen”
I think that’s a substantial part of the impact, but that there may be other substantial parts too, such as:
Time saved by employees who have to design application processes (since it’s usually easier to do things when one has a good writeup as guidance)
Causing orgs to hire sooner, since they’re more confident they can do it well and without a huge time investment
Something along the lines of “health of the organisation”; if that post reduces the chance of making a hire which isn’t a good fit, it reduces the chance of frictions and someone ending up being fired or quitting, which I imagine are negative for the culture of the organisation
Something along the lines of “health of the community”; I imagine a better hiring round will mean better applicant experiences, which could reduce rates of value drift or burnout or the like
But these are just quick thoughts, and I haven’t run application rounds myself, and I think that those things overlap somewhat such that there’s a risk of double-counting.
As an upper bound: Let’s say it influenced 1 to 5 hiring rounds, and advice in the post allowed advice-takers to hire 0 to 3 people per organization who were 1 to 10% more effective, and who stayed with the organization for 0.5 to 3 years
FWIW, I think the upper bound of my 80% confidence interval would be above 10% more effective and 3 years staying at the org, and definitely above 1% more effect and 0.5 years staying there.
I’m also not sure how to interpret your upper bound itself having a range? (Caveat that I haven’t looked at your Guesstimate model.)
I also think that one other effect perhaps worth modelling is that better hiring rounds might mean hires stay at the org for longer (since better and more fitting hires are chosen). This could either be modelled as more output by those employees, or as less cost/output-reduction by employees involved in later hiring rounds, or maybe both.
There are also cases in which an org just doesn’t hire anyone at all at a given time if they don’t find a good enough fit, and presumably better hiring rounds somewhat reduces the odds of that.
Ballpark: 0 to three excellent EA forum posts.
FWIW, intuitively, that seems like a pretty low upper bound for the value of improving other orgs’ hiring rounds. I guess this is just for the reasons noted above. (And obviously it’d be better if I actually myself provided specific alternative numbers and models—I’m just taking the quicker option, sorry!)
but sometimes, like in the case of a hiring round, I have the intuition that one might want to penalize the hiring organization for the lost opportunity cost of applicants, even though that’s not what Shapley values recommends
Many of the other projects were stated to have an impact by increasing the funding certain organisations received, thereby helping them hire more people, thereby resulting in more useful output. So by that logic, shouldn’t those projects also be penalised for the lost opportunity cost of applicants involved in the hiring rounds run by the orgs which received extra funding due to the project?
Or am I misunderstanding the reasoning or the modelling approach? (That’s very possible; I didn’t actually look at any of your Guesstimate models.)
I think that’s a substantial part of the impact, but that there may be other substantial parts too, such as...
Yes, those seem like at least somewhat important pathways to impact that I’ve neglected, particularly the first two points. I imagine that could easily lead to a 2x to 3x error (but probably not to a 10x error)
FWIW, I think the upper bound of my 80% confidence interval would be above 10% more effective and 3 years staying at the org, and definitely above 1% more effect and 0.5 years staying there.
Yeah, I disagree with this. I’d expect most interventions to have a small effect, and in particular I expect it to just be hard to change people’s actions by writing words. In particular, I’d be much higher if I was thinking about the difference between a completely terrible hiring round and an excellent one, but I don’t know that people start off all that terrible or that this particular post brings people up all that much.
That seems reasonable. I think my intuitions would still differ from yours, but I don’t have that much reason to expect my intuitions are well-calibrated here, nor have I thought about this carefully and explicitly.
Specific reactions to the evaluation of Takeaways from EAF’s Hiring Round
I think that’s a substantial part of the impact, but that there may be other substantial parts too, such as:
Time saved by employees who have to design application processes (since it’s usually easier to do things when one has a good writeup as guidance)
Causing orgs to hire sooner, since they’re more confident they can do it well and without a huge time investment
Something along the lines of “health of the organisation”; if that post reduces the chance of making a hire which isn’t a good fit, it reduces the chance of frictions and someone ending up being fired or quitting, which I imagine are negative for the culture of the organisation
Something along the lines of “health of the community”; I imagine a better hiring round will mean better applicant experiences, which could reduce rates of value drift or burnout or the like
But these are just quick thoughts, and I haven’t run application rounds myself, and I think that those things overlap somewhat such that there’s a risk of double-counting.
FWIW, I think the upper bound of my 80% confidence interval would be above 10% more effective and 3 years staying at the org, and definitely above 1% more effect and 0.5 years staying there.
I’m also not sure how to interpret your upper bound itself having a range? (Caveat that I haven’t looked at your Guesstimate model.)
I also think that one other effect perhaps worth modelling is that better hiring rounds might mean hires stay at the org for longer (since better and more fitting hires are chosen). This could either be modelled as more output by those employees, or as less cost/output-reduction by employees involved in later hiring rounds, or maybe both.
There are also cases in which an org just doesn’t hire anyone at all at a given time if they don’t find a good enough fit, and presumably better hiring rounds somewhat reduces the odds of that.
FWIW, intuitively, that seems like a pretty low upper bound for the value of improving other orgs’ hiring rounds. I guess this is just for the reasons noted above. (And obviously it’d be better if I actually myself provided specific alternative numbers and models—I’m just taking the quicker option, sorry!)
Many of the other projects were stated to have an impact by increasing the funding certain organisations received, thereby helping them hire more people, thereby resulting in more useful output. So by that logic, shouldn’t those projects also be penalised for the lost opportunity cost of applicants involved in the hiring rounds run by the orgs which received extra funding due to the project?
Or am I misunderstanding the reasoning or the modelling approach? (That’s very possible; I didn’t actually look at any of your Guesstimate models.)
Yes, those seem like at least somewhat important pathways to impact that I’ve neglected, particularly the first two points. I imagine that could easily lead to a 2x to 3x error (but probably not to a 10x error)
To answer this specifically:
Yeah, I disagree with this. I’d expect most interventions to have a small effect, and in particular I expect it to just be hard to change people’s actions by writing words. In particular, I’d be much higher if I was thinking about the difference between a completely terrible hiring round and an excellent one, but I don’t know that people start off all that terrible or that this particular post brings people up all that much.
That seems reasonable. I think my intuitions would still differ from yours, but I don’t have that much reason to expect my intuitions are well-calibrated here, nor have I thought about this carefully and explicitly.
Upper bound being a range is a mistake, fixed now.