When To Find More Information: A Short Explanation
What do you need to know to make a decision, and when should you try to find out more?
I think this is a really important topic, but when you’re making a decision, it’s one that deserves very little attention—it’s just rather important that it gets that bit of attention.
I wrote my dissertation on the topic, and would argue that basically all questions that matter should be approached using Value-of-Information (VoI). And there are complex quantitative tools that are useful for decisions about this. In doing my dissertation, I tried to actually use some of these methods. I came to the conclusion that you will all live far happier lives if you never actually use them, or even read a dissertation about them. Instead, there are some simple approaches and heuristics that get you almost the whole way there, so I’m writing this post.
Hopefully, this post can help get most of the value at a very low cost. But it will help even more if you use it as an exercise for a specific decision you are making. If possible, pick some personal decision you need to make now—even if it doesn’t help, it will be useful practice.
Ready?
VoI for Smart People
You have a decision to make. The fact that you are trying to make a decision implies that the answer isn’t obvious—you’re unsure about something. Given that, the first step is to spend 5 seconds thinking about whether getting more information would help you make the decision. At this point, you almost always should be able to think of something that you would want to know. (If the answer is that you don’t need any, you probably want to think again—or read the above linked post.)
If you think of information you can get easily and at very low cost, get it. Afterwards, if the decision isn’t made, come back to the post and start over—is there anything more?
If you discover there are uncertainties you’d like to understand, but it’s unclear what you need to know, where to find out more, or how hard it will be to get the information, you should now spend another five minutes, by the clock, really thinking about what you might want to know. But before you start, there are a few ideas to consider.
For that five minutes of thinking, you need to think about what you are trying to decide and how information could help. Here’s a set of questions to ask yourself during those five minutes:
1) What decision(s) are you making?
2) What do you not know, specifically, which is important for the decision?
3) If you knew more, would you change your decision? (If not, it’s not useful for the decision.)
4) Is there a way to become less uncertain? What could you find out that might help?
5) Is the information that would change your mind worth the cost of gathering it? (This might be tricky, but see below.)
For the last question, usually the answer is obviously yes or no. Sometimes, however, it’s unclear, and you need to think a bit more quantitatively about the value of the information. If you want to see the math for how VoI is used in practice, here are some examples, and some more, of how to do the basic quantitative work.
Sometimes, however, you will realize that you need to spend time thinking about the questions and consulting with others before you can start putting numbers together. That’s a bit more complex, and if you want to read about how I recommend doing this in great depth, feel free to read the last chapter of the dissertation.
(1) Justifying this would require a much more in-depth discussion than this post warrants, so feel free to ignore the claim.
(2) Specifically Bayesian Markov Chain Monte Carlo simulation, and Discrete Bayesian Networks. It’s all in the dissertation—which you shouldn’t read (see below.)
(3) If you want to put yourself to sleep, feel free to read anything other than the first 5 pages of the fifth chapter of the dissertation. (4)
(4) Really, only pages 159-164, here. That little bit will provide 90% of the value. That’s the part where I point out that doing the basics is a better use of your time than quantitative modeling. Trust me - the rest of the information in the dissertation is low-value, and for almost everyone, it’s not worth the cost of reading.
- Improving Institutional Decision-Making: Which Institutions? (A Framework) by (23 Aug 2021 2:26 UTC; 81 points)
- Introduction: A Primer for Politics, Policy and International Relations by (31 Dec 2019 19:27 UTC; 69 points)
- 's comment on doing more good vs. doing the most good possible by (3 Jan 2022 7:14 UTC; 13 points)
- Philosophy PhD Application: Advice on Written Submission by (19 Sep 2021 16:41 UTC; 4 points)
- 's comment on Improving Institutional Decision-Making: Which Institutions? (A Framework) by (23 Sep 2021 11:25 UTC; 4 points)
- 's comment on Option Value, an Introductory Guide by (1 Mar 2020 19:43 UTC; 2 points)
- 's comment on The Future Fund’s Project Ideas Competition by (10 Mar 2022 4:02 UTC; 1 point)
This was a really great Forum post! You expressed yourself really clearly, and I understood why you didn’t provide further information and what I should do next. Thanks so much!
Thanks for the post, but this seems like the tricky bit to me. Might you be able to give some rough rules of thumb people could apply to answer this question?
Trying to do actual VOI estimates gets pretty confusing, so what would be great is something simpler than that, but better than just going with your intuition.
I think there are probably some things to say, along the lines of things like “if you can spend under 10% of the time at stake in the decision, and you think it’s likely you’d change your mind (say with 50% chance), then probably investigate more”; or “if you’re early in your career, leans towards investigating because information is more valuable to you”; “people typically consider too few options, so it’s worth generating at least one alternative to your current options”.
This is an area I should probably write more about, but I have a harder time being pithy, and haven’t tried to distill my thoughts enough. But since you asked....
As a first approximation, you want to first consider the plausible value of the decision. If it’s choosing a career, for example, the difference between a good choice and a bad one is plausibly a couple million dollars. You almost certainly don’t want to spend more than a small fraction of that gathering information, but you do want to spend up to, say, 5% on thinking about the decision. (Yes, I’d say spending a year or two exploring the options before picking a career is worthwhile, if you’re really uncertain—but you shouldn’t need to be. See below.)
Once you have some idea of what the options are, you should pick what about the different options are good or bad—or uncertain. This should form the basis of at least a pro/con list—which is often enough by itself. (See my simulation here.) If you see that one option is winning on that type of list, you should probably just pick it—unless there are uncertainties that would change your mind.
Next, list those key uncertainties. In the career example, these might include: Will I enjoy doing the work?, How likely am I to be successful in the area?, How likely is the field to continue to be viable in the coming decades?, and How easy or hard is it to transition into/out of?
Notice that some of the uncertainties matter significantly, and others don’t. We have a tool that’s useful for this, which is the theoretical maximum of VoI, called Value of Perfect Information. This is the difference in value between knowing the answer with certainty, and the current decision. (Note: not knowing the future with certainty, but rather knowing the correct answer to the uncertainty. For example, knowing that you would have a 70% chance of being successful and making tons of money in finance.) Now ask yourself: If I knew the answer, would it change my decision? If the answer is no, drop it from the list of key uncertainties. If a relatively small probability of success would still leave finance as your top option, because of career capital and the potentially huge payoff, maybe this doesn’t matter. Alternatively, if even a 95% chance of success wouldn’t matter because you don’t know if you’d enjoy it, it still doesn’t matter—so move on to other questions.
If the answer is that knowing the answer would change your mind, you need to ask what information you could plausibly get about the question, and how expensive it is. For instance, you currently think there’s a 50% chance you’d enjoy working in finance. Spending a summer interning would make you sure one way or the other—but the cost in time is very high. It might be worth it, but there are other possibilities. Spending 15 minutes talking to someone in the field won’t make you certain, but will likely change your mind to think the odds are 90% or 10% - and in the former case, you can still decide to apply for a summer internship, and in the latter case, you can drop the idea now.
You should continue with this process of finding key things that would change your mind until you either think that you’re unlikely to change your mind further, or the cost of more certainty is high enough compared to the value of the decision that it’s not obviously worth the investment of time and money. (If it’s marginal or unclear, unless the decision is worth tens or hundreds of millions of dollars, significant further analysis is costly enough that you may not want to do it. If you’re unsure which way to decide at that point, you should flip a coin about the eventual decision—and if you’re uncertain enough to use a coin flip, then just do the riskier thing.)
Thanks—it’s useful to see your take on this!
+1
From my (limited) experience with VOI, it sounds great in theory, but in practice quickly turns into a hopeless pedantic muddle.
Yes, that was partially the conclusion of my dissertation—and see my response to the above comment.
I think that it’s generally useful to share a clear paradigm which is useful for non-experts, based on deep knowledge of a subject, and that’s what I tried to do here. In this case, I think that the concept and approach are a very generally useful point, and I would be very excited for more people to think about Value of Information more often, though as the post notes, mostly informally.
I may be falling prey to the curse of knowledge since I already knew about VoI when I read it but I thought How to Measure Anything (which has a chapter on VoI) was very pragmatic and accessible.
Also, not sure if it’s actually useful or falls into the “complex quantitative tools … [that you shouldn’t use]” category but I have a basic VoI calculator at: https://www.col-ex.org/posts/value-of-information-calculator-explained/.
Good writeup, and cool tool. I may use it and/or point to it in the future.
I agree that when everything is already quantified, and you can do this. The chapter in HtMA is also fantastic. But it’s fairly rare that people have already quantified all of the relevant variables and properly explored what the available decisions are or what they would affect—and not doing so can materially change the VoI, and are far more important to do anyways.
That said, no, basic VoI isn’t hard. It’s just that the use case is fairly narrow, and the conceptual approach is incredibly useful in the remainder of cases, even those cases where actually quantifying everything or doing to math is incredibly complex or even infeasible.
I read (or sort of skimmed) your dissertation (since I have a decision to make—actually (too) many) and thought this might provide some useful advice on how to do it.
I was surprised you discouraged people from reading it—maybe you are trying to hide the fun. I found it be one of the most exciting and racy PhD thesis’ i’ve ever read—it could be a best seller.
And it just doesn’t read like a pulp novel; like the best novels, it also hits peaks of scientific inspiration which i think rival the visions I’ve been told you get by reading great poets or theological works or einstein or Dirac or hawking in physics—takes you to a transcendent place.
In any event while i enjoyed the read, i didn’t get my problem solved—i.e. advice on what decision to make. (I ’might even have to decide first which of my problems i should try to solve first, and then do a 2nd iteration of the method and try to solve that particular problem. Or, the method might suggest I try to solve all the problems at once—eg if your house is burning down in a fire you both put it out and rebuild it at the same time.)
Since i found the procedure not exactly intuitive, it seemed to suggest you have to weigh off the cost of a ‘search procedure’ to get information about the possible value of alternative decisions, combined with the cost of making a decision. (eg for a burning house, you might spend some costly time consulting with some firefighters and homebuilders about the best course of action.)
A simple example might be shopping for an item. You could just go to a place you know—and you know the cost, and also the quality or value of the item. Or you could spend alot of time and money shopping around, to see if you can find something with a better value/cost ratio. There is always the risk that you may just waste alot of time and money and find nothing better.
I was wondering if the example above is the sort of decision making problem you are talking about.
(I think my main problem is finding a job—but I don’t really know where to look, and if there are any of the kind I want (I might be looking for something that does not exist) , and whether and how to research finding one. The cheapest default option might be just to wait and see if one comes to me, though it may lead to same result----nothing. One problem is I don’t know what the ‘prior’ is—any more than did the people from europe who explored the USA knew what they would find. They could try to get information from the indigneous people, but that takes time and effort,, and they didn’t know how reliable it would be.)
David might be cheating, using a neg: saying so much “don’t read my dissertation” that now I am actually very curious about it.
Seriously—start with the 5 pages I recommended, and that should give you enough information (VoI FTW!) to decide if you want to read Chapters 1 & 2 as well.
(But Chapters 3 and 4 really *are* irrelevant unless you happen to be designing a biosurveillance system or a terrorism threat early warning detection system that uses classified information.)
Sorry, man, you’re just making it sound even cooler.
OK, in retrospect, it definitely turns out that designing good biosurveillance systems was more important than I implied 2 years ago.