I’m looking at different career choices and in particular, academic research projects. I’ve tried to compare their impact by using some Fermi calculations, including working out:
the magnitude of the problem the research will attempt to solve
the likely value of the research if it is successful
the likelihood it will be successful
my marginal contribution to the research if I get involved with it
My calculations seem like they could be easily out by a couple of orders of magnitude. And it makes a difference—one less order of magnitude and the project is not more than the value of my marginal career impact if I simply maximized income and earned-to-give. Of course...who knows...I might have the order of magnitude the other way around, and perhaps the research project could be even more impactful.
Honestly, it seems like a bad idea to make any kind of decision based on this process, but if I want to know which career path has the most impact, I don’t know any other way to do it!
So I have lots of questions, but the biggest one is: if you don’t use this awfully flawed method to decide what the impact of a career path will be, what else would you use?
This makes me think of the contrast between systems analysis and net assessment/strategy. Yes, Fermi calculations are a valuable input into the discussion, but the nature of the problem is likely too complex to give sole weight to that calculation (in most circumstances—I think your example of comparing different research papers is a relatively simple/constrained environment).
In net assessment/strategy, the nature of the choice and your assessment of that nature determine the best methods of analysis. In systems analysis, a one-size-fits-all approach of measurement and analysis is taken. In certain circumstances that method of measurement is hugely valuable (why it became so popular in the US DoD), but in many other circumstances (like the Vietnam War) that method is deeply flawed.
It takes a solid assessment of the problem space to identify the key aspects of the competition. From there you can assess which methods and measures are most useful to judge the decision upon. And from there you can search for the data available to match your assessment.
Although choosing a career isn’t a competition against a competitor, your search to do the most good is a clearer objective than most adversaries can articulate regarding their objectives for long-term competition. The same meta-methods of analysis (to determine the best methods of analysis to fit the problem) apply. And then you can go on to finding the right data to fit those methods you’ve deemed most effective (maybe a Fermi calculation, but maybe 5 expert opinions in your career field or maybe your own gut feeling). I also think that in a career decision (unless it’s super major and clearly between two choices), the data you are going to be looking for is closer to a startup’s lean experiment than a longitudinal study. As you mentioned, Fermi calculations being readily available and usable doesn’t mean that is an effective tool for your decision.
Looking back at my answer, this didn’t really answer your question. The most effective technique I have found for deciding career choices though is to talk to as many career-related people as you can—because 1. they’ll be able to help you better understand the nature of the decision; and 2. they’ll open up doors for future opportunities, potentially even avoiding the need to make your current decision.
Firstly, Monte Carlo simulations (such as on https://www.getguesstimate.com/ ) is likely more precise/useful than pure Fermi estimates, as they make uncertainty, including your consideration of the value possibly being off by factor 10, explicit and thus you can have greater confidence in the results. One advantage Fermi estimates definitely have is that they force you to think about the different components of the problem, or in this case how different careers contribute to your impact. But they are generally speaking primarily helpful to estimate orders of magnitude, and are thus not all that useful in comparing different options unless they lie very far apart.
Secondly, I agree with shawzach that it makes a lot of sense to talk your career considerations through with other people. I used EAGx Virtual for that purpose and looked for all the people who might have something to contribute to my career considerations. But then again, it differed from your situation in that I didn’t have several alternatives but rather one preexisting plan and wanted to figure out whether it was “good” or I should look further for alternatives. Still, people might be able to add a few crucial considerations or just arguments you hadn’t thought of before that affect your estimates.
I recently made a big career change, and I am planning to write a detailed post on this soon. In particular, it will touch this point.
I did use use Fermi calculation to estimate my impact in my career options.
In some areas it was fairly straightforward (the problem is well defined, it is possible to meaningfully estimate the percentage of problem expected to be solved, etc.). However, in other areas I am clueless as to how to really estimate this (the problem is huge and it isn’t clear where I will fit in, my part in the problem is not very clear, there are too many other factors and actors, etc.).
In my case, I had 2 leading options, one of which was reasonable to amenable to these kind of estimates, and the other—not so much. The interesting thing was that in the first case, my potential impact turned out to be around the same order of magnitude as EtG, maybe a little bit more (though there is a big confidence interval).
All in all, I think this is a helpful method to gain some understanding of the things you can expect to achieve, though, as usual, these estimates shouldn’t be taken too seriously in my opinion.