5) Is the information that would change your mind worth the cost of gathering it? (This might be tricky, but see below.)
For the last question, usually the answer is obviously yes or no. Sometimes, however, it’s unclear, and you need to think a bit more quantitatively about the value of the information. If you want to see the math for how VoI is used in practice, here are some examples, and some more, of how to do the basic quantitative work.
Thanks for the post, but this seems like the tricky bit to me. Might you be able to give some rough rules of thumb people could apply to answer this question?
Trying to do actual VOI estimates gets pretty confusing, so what would be great is something simpler than that, but better than just going with your intuition.
I think there are probably some things to say, along the lines of things like “if you can spend under 10% of the time at stake in the decision, and you think it’s likely you’d change your mind (say with 50% chance), then probably investigate more”; or “if you’re early in your career, leans towards investigating because information is more valuable to you”; “people typically consider too few options, so it’s worth generating at least one alternative to your current options”.
This is an area I should probably write more about, but I have a harder time being pithy, and haven’t tried to distill my thoughts enough. But since you asked....
As a first approximation, you want to first consider the plausible value of the decision. If it’s choosing a career, for example, the difference between a good choice and a bad one is plausibly a couple million dollars. You almost certainly don’t want to spend more than a small fraction of that gathering information, but you do want to spend up to, say, 5% on thinking about the decision. (Yes, I’d say spending a year or two exploring the options before picking a career is worthwhile, if you’re really uncertain—but you shouldn’t need to be. See below.)
Once you have some idea of what the options are, you should pick what about the different options are good or bad—or uncertain. This should form the basis of at least a pro/con list—which is often enough by itself. (See my simulation here.) If you see that one option is winning on that type of list, you should probably just pick it—unless there are uncertainties that would change your mind.
Next, list those key uncertainties. In the career example, these might include: Will I enjoy doing the work?, How likely am I to be successful in the area?, How likely is the field to continue to be viable in the coming decades?, and How easy or hard is it to transition into/out of?
Notice that some of the uncertainties matter significantly, and others don’t. We have a tool that’s useful for this, which is the theoretical maximum of VoI, called Value of Perfect Information. This is the difference in value between knowing the answer with certainty, and the current decision. (Note: not knowing the future with certainty, but rather knowing the correct answer to the uncertainty. For example, knowing that you would have a 70% chance of being successful and making tons of money in finance.) Now ask yourself: If I knew the answer, would it change my decision? If the answer is no, drop it from the list of key uncertainties. If a relatively small probability of success would still leave finance as your top option, because of career capital and the potentially huge payoff, maybe this doesn’t matter. Alternatively, if even a 95% chance of success wouldn’t matter because you don’t know if you’d enjoy it, it still doesn’t matter—so move on to other questions.
If the answer is that knowing the answer would change your mind, you need to ask what information you could plausibly get about the question, and how expensive it is. For instance, you currently think there’s a 50% chance you’d enjoy working in finance. Spending a summer interning would make you sure one way or the other—but the cost in time is very high. It might be worth it, but there are other possibilities. Spending 15 minutes talking to someone in the field won’t make you certain, but will likely change your mind to think the odds are 90% or 10% - and in the former case, you can still decide to apply for a summer internship, and in the latter case, you can drop the idea now.
You should continue with this process of finding key things that would change your mind until you either think that you’re unlikely to change your mind further, or the cost of more certainty is high enough compared to the value of the decision that it’s not obviously worth the investment of time and money. (If it’s marginal or unclear, unless the decision is worth tens or hundreds of millions of dollars, significant further analysis is costly enough that you may not want to do it. If you’re unsure which way to decide at that point, you should flip a coin about the eventual decision—and if you’re uncertain enough to use a coin flip, then just do the riskier thing.)
Thanks for the post, but this seems like the tricky bit to me. Might you be able to give some rough rules of thumb people could apply to answer this question?
Trying to do actual VOI estimates gets pretty confusing, so what would be great is something simpler than that, but better than just going with your intuition.
I think there are probably some things to say, along the lines of things like “if you can spend under 10% of the time at stake in the decision, and you think it’s likely you’d change your mind (say with 50% chance), then probably investigate more”; or “if you’re early in your career, leans towards investigating because information is more valuable to you”; “people typically consider too few options, so it’s worth generating at least one alternative to your current options”.
This is an area I should probably write more about, but I have a harder time being pithy, and haven’t tried to distill my thoughts enough. But since you asked....
As a first approximation, you want to first consider the plausible value of the decision. If it’s choosing a career, for example, the difference between a good choice and a bad one is plausibly a couple million dollars. You almost certainly don’t want to spend more than a small fraction of that gathering information, but you do want to spend up to, say, 5% on thinking about the decision. (Yes, I’d say spending a year or two exploring the options before picking a career is worthwhile, if you’re really uncertain—but you shouldn’t need to be. See below.)
Once you have some idea of what the options are, you should pick what about the different options are good or bad—or uncertain. This should form the basis of at least a pro/con list—which is often enough by itself. (See my simulation here.) If you see that one option is winning on that type of list, you should probably just pick it—unless there are uncertainties that would change your mind.
Next, list those key uncertainties. In the career example, these might include: Will I enjoy doing the work?, How likely am I to be successful in the area?, How likely is the field to continue to be viable in the coming decades?, and How easy or hard is it to transition into/out of?
Notice that some of the uncertainties matter significantly, and others don’t. We have a tool that’s useful for this, which is the theoretical maximum of VoI, called Value of Perfect Information. This is the difference in value between knowing the answer with certainty, and the current decision. (Note: not knowing the future with certainty, but rather knowing the correct answer to the uncertainty. For example, knowing that you would have a 70% chance of being successful and making tons of money in finance.) Now ask yourself: If I knew the answer, would it change my decision? If the answer is no, drop it from the list of key uncertainties. If a relatively small probability of success would still leave finance as your top option, because of career capital and the potentially huge payoff, maybe this doesn’t matter. Alternatively, if even a 95% chance of success wouldn’t matter because you don’t know if you’d enjoy it, it still doesn’t matter—so move on to other questions.
If the answer is that knowing the answer would change your mind, you need to ask what information you could plausibly get about the question, and how expensive it is. For instance, you currently think there’s a 50% chance you’d enjoy working in finance. Spending a summer interning would make you sure one way or the other—but the cost in time is very high. It might be worth it, but there are other possibilities. Spending 15 minutes talking to someone in the field won’t make you certain, but will likely change your mind to think the odds are 90% or 10% - and in the former case, you can still decide to apply for a summer internship, and in the latter case, you can drop the idea now.
You should continue with this process of finding key things that would change your mind until you either think that you’re unlikely to change your mind further, or the cost of more certainty is high enough compared to the value of the decision that it’s not obviously worth the investment of time and money. (If it’s marginal or unclear, unless the decision is worth tens or hundreds of millions of dollars, significant further analysis is costly enough that you may not want to do it. If you’re unsure which way to decide at that point, you should flip a coin about the eventual decision—and if you’re uncertain enough to use a coin flip, then just do the riskier thing.)
Thanks—it’s useful to see your take on this!
+1
From my (limited) experience with VOI, it sounds great in theory, but in practice quickly turns into a hopeless pedantic muddle.
Yes, that was partially the conclusion of my dissertation—and see my response to the above comment.