The cases for specific priorities or interventions that are commonly advocated based on a longtermist perspective (e.g. “work on technical AI safety”) are usually far from watertight. It could be valuable to improve them, by making them more “robust” or otherwise.
Expected-value calculations that are based on a single quantitative model have significant limitations. They can be useful as one of many inputs to a decision, but it would usually be bad to use them as one’s sole decision tool.
(I am actually a big fan of the GiveWell/Holden Karnofsky posts you link to. When I disagree with other people it often comes down to me favoring more “cluster thinking”. For instance, these days this happens a lot to me when talking to people about AI timelines, or other aspects of AI risk.)
However, I think I disagree with your characterization of the case for CL more broadly, at least for certain uses/meanings of CL.
Here is one version of CL which I believe is based on much more than just expected-value calculations within a single model: This is roughly the claim that (i) in our project of doing as much good as possible we should at the highestlevel be mostly guided by very long-run effects and (ii) this makes an actual difference for how we plan and prioritize at intermediate levels.
Here are I have a picture in mind that is roughly as follows:
Lowest level: Which among several available actions should I take right now?
Intermediate levels:
What are the “methods” and inputs (quantitative models, heuristics, intuitions, etc.) I should use when thinking about the lowest level?
What systems, structures, and incentives should we put in place to “optimize” which lowest-level decision situations I and other agents find ourselves in in the first place?
How do I in turn best think about which methods, systems, structures, etc. to use for answering these intermediate-level questions?
Etc.
Highest level: How should I ultimately evaluate the intermediate levels?
So the following would be one instance of part (i) of my favored CL claim: When deciding whether to use cluster thinking or sequence thinking for a decision, we should aim to choose whichever type of thinking best helps us find the option with most valuable long-run effects. For this it is not required that I make the choice between sequence thinking or cluster thinking by an expected-value calculation, or indeed any direct appeal to any long-run effects. But, ultimately, if I think that, say, cluster thinking is superior to sequence thinking for the matter at hand, then I do so because I think this will lead to the best long-run consequences.
And these would be an instances of part (ii): That often we should decide primarily based on the proxy of “what does most reduce existential risk?”; that it seems good to increase the “representation” of future generations in various political contexts; etc.
Regarding what the case for this version of CL rests on:
For part (i), I think it’s largely a matter of ethics/philosophy, plus some high-level empirical claims about the world (the future being big etc.). Overall very similar to the case for AL. I think the ethics part is less in need of “cluster thinking”, “robustness” etc. And that the empirical part is, in fact, quite “robustly” supported.
[This point made me most want to push back against your initial claim about CL:] For part (ii), I think there are several examples of proxy goals, methods, interventions, etc., that are commonly pursued by longtermists which have a somewhat robust case behind them that does not just rely on an expected value estimate based on a single quantitative model. For instance, avoiding extinction seems very important from a variety of moral perspectives as well as common sense, there are historical precedents of research and advocacy at least partly motivated by this goal (e.g. nuclear winter, asteroid detection, perhaps even significant parts of environmentalism), there is a robust case for several risks longtermists commonly worry about (including AI), etc. More broadly, conversations involving explicit expected value estimates, quantitative models, etc. are only a fraction of the longtermist conversations I’m seeing. (If anything I might think that longtermists, at least in some contexts, make too little use of these tools.) E.g. look at the frontpage of LessWrong, or their curated content. I’m certainly not among the biggest fans of LessWrong or the rationality community, but I think it would be fairly inaccurate to say that a lot of what is happening there is people making explicit expected value estimates. Ditto for longtermist content featured in the EA Newsletter, etc. etc. I struggle to think of any example I’ve seen where a longtermist has made an important decision based just on a single EV estimate.
Rereading your initial comment introducing AL and CL, I’m less sure if by CL you had in mind something similar to what I’m defending above. There certainly are other readings that seem to hinge more on explicit EV reasoning or that are just absurd, e.g. “CL = never explicitly reason about anything happening in the next 100 years”. However, I’m less interested in these versions since they to me would seem to be a poor description of how longtermists actually reason and act in practice.
Hi Sam, thank you for your thoughtful reply.
Here are some things we seem to agree on:
The cases for specific priorities or interventions that are commonly advocated based on a longtermist perspective (e.g. “work on technical AI safety”) are usually far from watertight. It could be valuable to improve them, by making them more “robust” or otherwise.
Expected-value calculations that are based on a single quantitative model have significant limitations. They can be useful as one of many inputs to a decision, but it would usually be bad to use them as one’s sole decision tool.
(I am actually a big fan of the GiveWell/Holden Karnofsky posts you link to. When I disagree with other people it often comes down to me favoring more “cluster thinking”. For instance, these days this happens a lot to me when talking to people about AI timelines, or other aspects of AI risk.)
However, I think I disagree with your characterization of the case for CL more broadly, at least for certain uses/meanings of CL.
Here is one version of CL which I believe is based on much more than just expected-value calculations within a single model: This is roughly the claim that (i) in our project of doing as much good as possible we should at the highest level be mostly guided by very long-run effects and (ii) this makes an actual difference for how we plan and prioritize at intermediate levels.
Here are I have a picture in mind that is roughly as follows:
Lowest level: Which among several available actions should I take right now?
Intermediate levels:
What are the “methods” and inputs (quantitative models, heuristics, intuitions, etc.) I should use when thinking about the lowest level?
What systems, structures, and incentives should we put in place to “optimize” which lowest-level decision situations I and other agents find ourselves in in the first place?
How do I in turn best think about which methods, systems, structures, etc. to use for answering these intermediate-level questions?
Etc.
Highest level: How should I ultimately evaluate the intermediate levels?
So the following would be one instance of part (i) of my favored CL claim: When deciding whether to use cluster thinking or sequence thinking for a decision, we should aim to choose whichever type of thinking best helps us find the option with most valuable long-run effects. For this it is not required that I make the choice between sequence thinking or cluster thinking by an expected-value calculation, or indeed any direct appeal to any long-run effects. But, ultimately, if I think that, say, cluster thinking is superior to sequence thinking for the matter at hand, then I do so because I think this will lead to the best long-run consequences.
And these would be an instances of part (ii): That often we should decide primarily based on the proxy of “what does most reduce existential risk?”; that it seems good to increase the “representation” of future generations in various political contexts; etc.
Regarding what the case for this version of CL rests on:
For part (i), I think it’s largely a matter of ethics/philosophy, plus some high-level empirical claims about the world (the future being big etc.). Overall very similar to the case for AL. I think the ethics part is less in need of “cluster thinking”, “robustness” etc. And that the empirical part is, in fact, quite “robustly” supported.
[This point made me most want to push back against your initial claim about CL:] For part (ii), I think there are several examples of proxy goals, methods, interventions, etc., that are commonly pursued by longtermists which have a somewhat robust case behind them that does not just rely on an expected value estimate based on a single quantitative model. For instance, avoiding extinction seems very important from a variety of moral perspectives as well as common sense, there are historical precedents of research and advocacy at least partly motivated by this goal (e.g. nuclear winter, asteroid detection, perhaps even significant parts of environmentalism), there is a robust case for several risks longtermists commonly worry about (including AI), etc. More broadly, conversations involving explicit expected value estimates, quantitative models, etc. are only a fraction of the longtermist conversations I’m seeing. (If anything I might think that longtermists, at least in some contexts, make too little use of these tools.) E.g. look at the frontpage of LessWrong, or their curated content. I’m certainly not among the biggest fans of LessWrong or the rationality community, but I think it would be fairly inaccurate to say that a lot of what is happening there is people making explicit expected value estimates. Ditto for longtermist content featured in the EA Newsletter, etc. etc. I struggle to think of any example I’ve seen where a longtermist has made an important decision based just on a single EV estimate.
Rereading your initial comment introducing AL and CL, I’m less sure if by CL you had in mind something similar to what I’m defending above. There certainly are other readings that seem to hinge more on explicit EV reasoning or that are just absurd, e.g. “CL = never explicitly reason about anything happening in the next 100 years”. However, I’m less interested in these versions since they to me would seem to be a poor description of how longtermists actually reason and act in practice.