I think IASPCs handle these things well, and think there’s some misinterpretation going on. What makes a strong plan change under this metric is determined by whatever 80,000 Hours thinks is most important, and currently this includes academic, industry, EA org and government roles. These priorities also change in response to new information and needs. The problem Sebastian is worried about seems more of a big deal: maybe some orgs / local groups are defining their metrics mostly in terms of one category, or that it’s easy to naively optimise for one category at the expense of the others.
The part about counting impact from skill-building and direct work differently simply seems like correct accounting: EA orgs should credit themselves with substantially more impact for a plan change which has led to impact as one which might do so in the future, most obviously because the latter has a <100% probability of turning into the former.
I also think the metric works fine with Sebastian’s point that quant trading can be competitive with other priority paths. You seem to imply that the use of IASPCs contradicts his advice, but you point to a non-priority rating for ‘earn to give in a medium income career’, which is not quant trading!† 80,000 Hours explicitly list quant trading as a priority path (as Seb pointed out in the post), so if an org uses IASPCs as one of their metrics they should be excited to see people with those particular skills go down that route. (If any readers land quant jobs in London, please do say hi :) )
I agree that misapplication of this or similar metrics is dangerous, and that if e.g. some local groups are just optimising for EA-branded orgs instead of at least the full swathe of priority paths, there’s a big opportunity to improve. All the normal caveats about using metrics sensibly continue to apply.
All views my own.
†As a former trader, I felt the need to put an exclamation mark somewhere in this paragraph.
Ultimately the more you ground the metric in “what some sensible people thing is important and makes sense right now”, the more nuance it has, and the more is it tracking reality. The text in my quote is verbatim copy from the page describing the metric from 2015, so I think it’s highly relevant for understanding how IASPCs were understood. I agree that 80k career guides as a whole actually has much more nuance, and suggests approach like “figure out what will be needed in future and prepare for that”.
The whole accounting still seems wrong: per definition, what’s counted is … caused them to change the career path they intend to pursue, … ; this is still several steps away from impact: if someone changes their intentions to pursue jobs in EA orgs, it is counted as impact, even if the fraction of the people making such plans who will succeed is low.
For specificity, would you agree that someone who was 2 years away from graduation in 2016, deciding to change career plan to pursuing a job in CEA, would have been counted as impact 10, while someone switching from a plan going to industry to pursuing PhD in econ would have been counted as 1, and someone deciding to stay in, let’s say, cognitive neuroscience, would have been counted as 0?
I just wrote a bit more about how we measure IASPCs in another comment on this thread. We don’t use a precise formula and the details are important so I can’t say exactly how we’d rate a particular change at this level of generality.
That said, we take into account someone’s degree of follow through when we score their plan change, such that very few of our highest rated plan changes (rated-100 or more) are from people who are not currently doing impactful work.
30% have changed their plans but not yet passed a major “milestone” in their shift. Most of these people have applied to a new graduate programme but not yet received an offer.
30% have reached a milestone, but are still building career capital (e.g. entered graduate school, or taken a high-earning job but not yet donated much).
40% have already started having an impact (e.g. have published research, taken a non-profit job).
I agree if we didn’t take follow through into account it would lead to some scores that were far removed from expected impact such as the hypothetical you’ve described.
thanks very much for the clarifying discussion. The fact that there is this discussion (also looking at the high number of votes for the comments) illustrates that there is at least some confusion around rating EA org vs. non-EA org careers, which is a bit concerning in itself.
FWIW my original claim was not that people (neither 80k nor community members) get the rational analysis part wrong. And a career path where actual impact is a few years off should totally get a reduced expected value & rating. (My claim in the initial post is that many of the other paths are still competitive with EA org roles.) There is little actual disagreement that quant trading is a great career.
My worry is that many soft factors may cause people to develop preferences that are not in line with the EV reasoning, and that may reduce motivation and/or lead to people overly focused on jobs at explicit EA employers.
Also, you lack a ‘stamp of approval’ from 80k when you pursue some of these careers that you kind of don’t need when doing a ‘standard’ path like working at CEA/FHI/80k/OPP or do a top ML PhD, even if all of them were rated 10. (In coaching days this was better, because you could just tell your doubting student group leader that this is what 80k wants you to do :) )
I’m sympathetic to your comment. The fact that (I think) 80k is not making this particular mistake in its IASPC system does not imply that there’s nothing to be concerned about. I think your post as well as some of the comments in other threads do a good job of laying out many of the factors pushing people toward jobs at explicitly EA orgs.
I think IASPCs handle these things well, and think there’s some misinterpretation going on. What makes a strong plan change under this metric is determined by whatever 80,000 Hours thinks is most important, and currently this includes academic, industry, EA org and government roles. These priorities also change in response to new information and needs. The problem Sebastian is worried about seems more of a big deal: maybe some orgs / local groups are defining their metrics mostly in terms of one category, or that it’s easy to naively optimise for one category at the expense of the others.
The part about counting impact from skill-building and direct work differently simply seems like correct accounting: EA orgs should credit themselves with substantially more impact for a plan change which has led to impact as one which might do so in the future, most obviously because the latter has a <100% probability of turning into the former.
I also think the metric works fine with Sebastian’s point that quant trading can be competitive with other priority paths. You seem to imply that the use of IASPCs contradicts his advice, but you point to a non-priority rating for ‘earn to give in a medium income career’, which is not quant trading!† 80,000 Hours explicitly list quant trading as a priority path (as Seb pointed out in the post), so if an org uses IASPCs as one of their metrics they should be excited to see people with those particular skills go down that route. (If any readers land quant jobs in London, please do say hi :) )
I agree that misapplication of this or similar metrics is dangerous, and that if e.g. some local groups are just optimising for EA-branded orgs instead of at least the full swathe of priority paths, there’s a big opportunity to improve. All the normal caveats about using metrics sensibly continue to apply.
All views my own.
†As a former trader, I felt the need to put an exclamation mark somewhere in this paragraph.
Ultimately the more you ground the metric in “what some sensible people thing is important and makes sense right now”, the more nuance it has, and the more is it tracking reality. The text in my quote is verbatim copy from the page describing the metric from 2015, so I think it’s highly relevant for understanding how IASPCs were understood. I agree that 80k career guides as a whole actually has much more nuance, and suggests approach like “figure out what will be needed in future and prepare for that”.
The whole accounting still seems wrong: per definition, what’s counted is … caused them to change the career path they intend to pursue, … ; this is still several steps away from impact: if someone changes their intentions to pursue jobs in EA orgs, it is counted as impact, even if the fraction of the people making such plans who will succeed is low.
For specificity, would you agree that someone who was 2 years away from graduation in 2016, deciding to change career plan to pursuing a job in CEA, would have been counted as impact 10, while someone switching from a plan going to industry to pursuing PhD in econ would have been counted as 1, and someone deciding to stay in, let’s say, cognitive neuroscience, would have been counted as 0?
Hi Jan,
I just wrote a bit more about how we measure IASPCs in another comment on this thread. We don’t use a precise formula and the details are important so I can’t say exactly how we’d rate a particular change at this level of generality.
That said, we take into account someone’s degree of follow through when we score their plan change, such that very few of our highest rated plan changes (rated-100 or more) are from people who are not currently doing impactful work.
Of the rated 10’s, our analysis in 2017 found:
30% have changed their plans but not yet passed a major “milestone” in their shift. Most of these people have applied to a new graduate programme but not yet received an offer.
30% have reached a milestone, but are still building career capital (e.g. entered graduate school, or taken a high-earning job but not yet donated much).
40% have already started having an impact (e.g. have published research, taken a non-profit job).
I agree if we didn’t take follow through into account it would lead to some scores that were far removed from expected impact such as the hypothetical you’ve described.
Hope this clarifies things.
Hey Jan and Howie,
thanks very much for the clarifying discussion. The fact that there is this discussion (also looking at the high number of votes for the comments) illustrates that there is at least some confusion around rating EA org vs. non-EA org careers, which is a bit concerning in itself.
FWIW my original claim was not that people (neither 80k nor community members) get the rational analysis part wrong. And a career path where actual impact is a few years off should totally get a reduced expected value & rating. (My claim in the initial post is that many of the other paths are still competitive with EA org roles.) There is little actual disagreement that quant trading is a great career.
My worry is that many soft factors may cause people to develop preferences that are not in line with the EV reasoning, and that may reduce motivation and/or lead to people overly focused on jobs at explicit EA employers.
Also, you lack a ‘stamp of approval’ from 80k when you pursue some of these careers that you kind of don’t need when doing a ‘standard’ path like working at CEA/FHI/80k/OPP or do a top ML PhD, even if all of them were rated 10. (In coaching days this was better, because you could just tell your doubting student group leader that this is what 80k wants you to do :) )
Hey Sebastian,
I’m sympathetic to your comment. The fact that (I think) 80k is not making this particular mistake in its IASPC system does not imply that there’s nothing to be concerned about. I think your post as well as some of the comments in other threads do a good job of laying out many of the factors pushing people toward jobs at explicitly EA orgs.