Strong Longtermism, Irrefutability, and Moral Progress
The following critique is a lightly modified version of the one found here. It builds on the recent post A Case Against Strong Longtermism by Vaden Masrani, but can be read and understood independently. If you’re sick of our rants on the forum, you can also listen to a podcast episode in which Vaden and I cover similar territory—albeit more quickly and with a large helping of cheekiness. We promise to move onto other topics after this (although Vaden is now threatening a response to his post—God help us. I guess that’s why downvoting exists). Much love to the community and all its members.
Thanks to Daniel Hageman, Vaden Masrani, and Mauricio Baker for their continual feedback and criticism as this piece evolved, and to Luke Freeman, Mira Korb, Isis Kearney, Alex HT, Max Heitmann, Gavin Acquroff, and Maximilian Negele for their comments and suggestions on earlier drafts. All errors, misrepresentations, and harsh words are my own.
The first paragraph of the final section is stolen from an upcoming piece I wrote for GWWC. Whoops.
TL;DR: Focusing on the long-term destroys the means by which we make progress — moral and otherwise.
The new moral philosophy of longtermism has staggering implications if widely adopted. In The Case for Strong Longtermism, Hilary Greaves and Will MacAskill write
The idea, then, is that for the purposes of evaluating actions, we can in the first instance often simply ignore all the effects contained in the first 100 (or even 1000) years, focussing primarily on the further-future effects. Short-run effects act as little more than tie-breakers. (pg. 1; italics mine)
The idea energizing this philosophy is that most of our “moral value” lies thousands, millions, or even billions of years from now, because we can expect many more humans and animals to exist in the future than right now. In the words of Greaves and MacAskill: “If humanity’s saga were a novel we would still be on the very first page.” (pg. 1)
Longtermism is causing many to question why we should be at all concerned with the near term impact of our actions. Indeed, if you are convinced by this calculus, then all current injustice, death, and suffering are little more than rounding errors in our moral calculations. Why care about parasitic worms in Africa if we can secure utopia for future generations?
EA has yet to take irreversible action based on these ideas, but the philosophy is gaining traction and therefore deserves an equal amount of criticism. There have been millions donated to the cause of improving the long-term future: at the time of writing the Long-Term Future Fund has received just under $4.5 million USD in total, and the Open Philanthropy Project has dedicated a focus area to this cause in the form of “risks from advanced artificial intelligence.” While many millions more are still funneled through GiveWell, The Life You Can Save, and Animal Charity Evaluators, should Greaves and MacAskill prove sufficiently persuasive such “near-term” efforts could vanish: “If society came to adopt these views, much of what we would prioritise in the world today would change.” (pg. 3)
This post is a critique of longtermism as expounded in The Case for Strong Longtermism. Prior criticism of the idea has typically revolved around the intractability objection which, while agreeing that the long-term future should dominate our moral concerns, argues we can’t have any reliable effect on it. While correct, it lets longtermism off far too lightly. It does not criticize it as a moral ideal, but rather as something good but unrealizable.
The recent essay by Vaden Masrani does attempt to refute the two premises on which strong longtermism is founded. It argues that (i) the mathematics involved in the expected value calculations over possible futures are fundamentally flawed — indeed, meaningless — and (ii) that we should be biased towards the present because it is the only thing we know how to reliably affect. My criticisms will build on these.
I will focus on two aspects of strong longtermism, henceforth simply longtermism. First, the underlying arguments inoculate themselves from criticism by using arbitrary assumptions on the number of future generations. Second, ignoring short-term effects destroys the means by which we make progress — moral, scientific, artistic, and otherwise. In other words, longtermism is a dangerous moral ideal because it robs us of the ability to correct our mistakes.
Since the critique may come across as somewhat harsh, it’s worth spending a moment to frame it.
Motivation
My assailment of longtermism comes from a place of deep sympathy with and general support of the ideals of effective altruism. The community has both generated and advocated many great ideas, including evaluating philanthropic efforts based on impact rather than emotional valence, acknowledging that “doing good” is a difficult resource-allocation problem, and advocating an ethical system grounded in impartiality across all sentient beings capable of suffering. Calling attention to farmed animal welfare, rigorously evaluating charities, and encouraging the privileged among us to donate our wealth, have all been hugely important initiatives. Throughout its existence, EA has rightly rejected two forms of authority which have traditionally dominated the philanthropic space: emotional and religious authority.
It has, however, succumbed to a third — mathematical authority. Firmly grounded in Bayesian epistemology, the community is losing its ability to step away from the numbers when appropriate, and has forgotten that its favourite tools — expected value calculations, Bayes theorem, and mathematical models — are precisely that: tools. They are not in and of themselves a window onto truth, and they are not always applicable. Rather than respect the limit of their scope, however, EA seems to be adopting the dogma captured by the charming epithet shut up and multiply.
EA is now at risk of adopting a bad idea; one that if fully subscribed to, I fear will lead to severe and irreversible damage — not only to the movement, but to the many people and animals whose suffering would be willfully ignored. As will be elaborated on later, rejecting longtermism will not cause a substantial shift in current priorities; many of the prevailing causes will remain unaffected. If, however, longtermism is widely adopted and its logic taken seriously, many of EA’s current priorities would be replaced with vague and arbitrary interventions to improve the course of the long-term future.
Let’s begin by examining the kinds of reasoning used to defend the premises of longtermism.
Irrefutable Reasoning
“For the purposes of this article”, write Greaves and MacAskill,
we will generally make the quantitative assumption that there are, in expectation, at least 1 quadrillion (10^15) people to come — 100,000 times as many people in the future as are alive today. This we [sic] be true if, for example, we assign at least a 1% chance to civilization continuing until the Earth is no longer habitable, using an estimate of 1 billion years’ time for that event and assuming the same per-century population as today, of approximately 10 billion people per century. (pg. 5)
This paragraph illustrates one of the central pillars of longtermism. Without positing such large numbers of future people, the argument would not get off the ground. The assumptions, however, are tremendously easy to change on the fly. Consequently, they’re dangerously impermeable to reason. Just as the astrologer promises us that “struggle is in our future” and can therefore never be refuted, so too can the longtermist simply claim that there are a staggering number of people in the future, thus rendering any counter argument mute.
Such unfalsifiable claims lead to the following sorts of conclusions:
Suppose that $1bn of well-targeted grants could reduce the probability of existential catastrophe from artificial intelligence by 0.001%. . . . Then the expected good done by [someone] contributing $10,000 to AI [artificial intelligence] safety would be equivalent . . . to one hundred thousand lives saved. (pg. 14)
Of course, it is impossible to know whether $1bn of well-targeted grants could reduce the probability of existential risk, let alone by such a precise amount. The “probability” in this case thus refers to someone’s (entirely subjective) probability estimate — “credence” — a number with no basis in reality and based on some ad-hoc amalgamation of beliefs. Notice that if one shifted one’s credence from 0.001% to 0.00001%, donating to AI safety would still be more than twice as effective as donating to the Against Malaria Foundation (AMF) (using GiveWell’s 2020 estimates).
A reasonable retort here is that all estimates in this space necessarily include a certain amount of uncertainty. That, for example, the difference between GiveWell’s estimates and those for AI risk are a matter of degree, not of kind. This is correct — the differences are a matter of degree. But each of those degrees introduces more subjectivity and arbitrariness into the equation. Our incredulity and skepticism should rise in equal measure.
GiveWell’s estimates use real, tangible, collected data. Other studies may of course conflict with their findings, in which case we’d have work to do. Indeed, such criticism would be useful for it would force GiveWell to develop more robust estimates. Needless to say, this process is entirely different than assigning arbitrary numbers to events about which we are utterly ignorant. My credence could be that working on AI safety will reduce existential risk by 5% and yours could be 10^-19%, and there’s no way to discriminate between them. Appealing to the beliefs of experts in the field does not solve the problem. From which dataless, magical sources are their beliefs derived?
Moreover, should your credence be 10^-19% in the effectiveness of AI Safety interventions, then I can still make that intervention look arbitrarily good, simply by increasing the “expected number of humans” in the future. Indeed, in his book Superintelligence, Nick Bostrom has “estimated” that there could be 10^64 sentient beings in the future. By those lights, the expected number of lives, even with a credence of 10^-19%, is still positively astronomical.
As alluded to above, the philosophy validating the reliance on subjective probability estimates is called Bayesian epistemology. It frames the search for knowledge in terms of beliefs (which we quantify with numbers, and must update in accordance with Bayes rule, else risk rationality-apostasy!). It has imported valid statistical methods used in economics and computer science, and erroneously applied them to epistemology, the study of knowledge creation. It’s ill-defined, is based on confirmation as opposed to falsification, leads to paradoxes, and relies on the provably false probabilistic induction. In other words, it has been refuted, and yet, somehow manages to stick around (ironically, it’s precisely this aspect of Bayesianism which is so dubious: its inability to reject any hypothesis).
Bayesian epistemology unhelpfully borrows standard mathematical notation. Thus, subjective credences tend to be compared side-by-side with statistics derived from actual data, and treated as if they were equivalent. But prophecies about when AGI will take over the world — even when cloaked in advanced mathematics — are of an entirely different nature than, say, impact evaluations from randomized controlled trials. They should not be treated as equivalent.
Once one adopts Bayesianism and loses track of the different origins of various predictions, then the attempt to compare cause areas becomes a game of “who has the bigger number.” And longtermism will win this game. Every time. It becomes unavoidable because it abolishes the means by which one can disagree with its conclusion, because it can always simply use bigger numbers. But we must remind ourselves that the numbers used in longtermist calculations are not the same as those derived from actual data. We should remember that mathematics is not an oracle unto truth. It is a tool, and one that in this case is inappropriately used. There are insufficient constraints when reasoning based solely on beliefs and big numbers — it is not informative and is not in any way tethered to a real data set, or to reality. Just as we discard poor, unfalsifiable, justifications in other areas, so too should we dispense with them in moral reasoning.
The Antithesis of Moral Progress
If you wanted to implement a belief structure which justified unimaginable horrors, what sort of views would it espouse? A good starting point would be to disable our critical capacities from evaluating the consequences of our actions, most likely by appealing to some vague and distant glorious future lying in wait. And indeed, this tool has been used by many horrific ideologies in the past.
Definitely and beyond all doubt, our future or maximum program is to carry China forward to socialism and communism. Both the name of our Party and our Marxist world outlook unequivocally point to this supreme ideal of the future, a future of incomparable brightness and splendor.
- Mao Tse Tung, “On Coalition Government”. Selected Works, Vol. III, p. 282. (italics mine)
Of course, the parallel between longtermism and authoritarianism is a weak one, if only because longtermism has yet to be instantiated. I don’t doubt that longtermism is rooted in deep compassion for those deemed to be ignored by our current moral frameworks and political processes. Indeed, I know it is, because the EA community is filled with the most kind-hearted people I’ve ever met.
Inadvertently, however, longtermism is almost tailor-made to disable the mechanisms by which we make progress.
Progress entails solving problems and generating the knowledge to do so. Because humans are fallible and our ideas are prone to error, our solutions usually have unintended negative consequences. These, in turn, become new problems. We invent pain relief medications which facilitate an opioid epidemic. We create the internet which leads to social media addiction. We invent cars which lead to car accidents. This is not to say we would have been better off not solving problems (of course we wouldn’t), only that solutions beget new — typically less severe — problems. This is a good thing. It’s the sign of a dynamic, open society focused on implementing good ideas and correcting bad ones.
Moral progress is no different. Abstract reasoning from first principles can be useful, but it will only get you so far. No morality prior to the industrial revolution could have foreseen the need to introduce eight-hour workdays or labour laws. No one 1,000 years ago could have foreseen factory farming, child-pornography spread via the internet, or climate change. As society changes, it is crucial that we maintain the ability to constantly adapt and evolve our ethics in order to handle new situations.
The moral philosophy espoused by EA should be one focused on highlighting problems and solving them. On being open to changing our ideas for the better. On correcting our errors.
Longtermism is precisely the opposite. By “ignoring the effects contained in the first 100 (or even 1000) years,” we ignore problems with the status quo, and hamstring our efforts to create solutions. If longtermism had been adopted 100 years ago then problems like factory farming, HIV/AIDS, and Measles would have been ignored. Greaves and MacAskill argue that we should have no moral discount factor, i.e., a “zero rate of pure time preference”. I agree — but this is besides the point. While time is morally irrelevant, it is relevant for solving problems. Longtermism asks us to ignore problems now, and focus on what we believe will be the biggest problems many generations from now. Abiding by this logic would result in the stagnation of knowledge creation and progress.
It is certainly possible to accuse me of taking the phrase “ignoring the effects” too literally. Perhaps longtermists wouldn’t actually ignore the present and its problems, but their concern for it would be merely instrumental. In other words, longtermists may choose to focus on current problems, but the reason to do so is out of concern for the future.
My response is that attention is zero-sum. We are either solving current pressing problems, or wildly conjecturing what the world will look like in tens, hundreds, and thousands of years. If the focus is on current problems only, then what does the “longtermism” label mean? If, on the other hand, we’re not only focused on the present, then the critique holds to whatever extent we’re guessing about future problems and ignoring current ones. We cannot know what problems the future will hold, for they will depend on the solutions to our current problems which, by definition, have yet to be discovered. The best we can do is safe-guard our ability to make progress and to correct our mistakes.
In sum, given the need for a constantly evolving ethics, one of our most important jobs is to ensure that we can continue criticizing and correcting prevailing moral views. The focus on the long-term future, however, stops the means by which we can obtain feedback about our actions now — the only reliable way to improve our current moral theories. Moral principles, like all ideas, evolve over time according to the pressure exerted on them by criticism. The ability to criticize, then, is paramount to making progress. Disregarding current problems and suffering renders longtermism impermeable to error-correction. Thus, while the longtermist project may arise out of more compassion for sentient beings than many other dogmas, it has the same nullifying effect on our critical capacities.
What now?
We are at an unprecedented time in history: We can do something about the abundance of suffering around us. For most of the human story, our ability to eradicate poverty, cure disease, and save lives was devastatingly limited. We were hostages to our environments, our biology, and our traditions. Finally however, trusting in our creativity, we have developed powerful ideas on how to improve life. We now know of effective methods to prevent malaria, remove parasitic worms, prevent vitamin deficiencies, and provide surgery for fistula. We have the technology to produce clean-meat to reduce animal suffering. We constructed democratic institutions to protect the vulnerable and reduce conflict. These are all staggering feats of human ingenuity.
Longtermism would have us disavow this tradition of progress. We would stop solving the problems in front of us, only to focus on distant problems obscured by the impenetrable wall of time.
For what it’s worth, should the EA community abandon longtermism, I think many of its current priorities would remain unchanged; long-term causes do not yet dominate its portfolio. Causes such as helping the global poor and reducing suffering from factory farming would of course remain a priority. So too would interventions such as improving institutional decision making and reducing the threat of nuclear war and pandemics. Such causes are important because the problems exist and do not require arbitrary assumptions on the number of future people.
My goal is not necessarily to change the current focus of the EA community, but rather to criticize the beginnings of a philosophy which has the potential to upend the values which made it unique in the first place: the combination of compassion with evidence and reason. It is in danger of discarding the latter half of that equation.
We can look at their track record on other questions, and see how reliably (or otherwise) different people’s predictions track reality.
I agree that below a certain level (certainly by 10^-19, and possibly as high as 10^-3) direct calibration-in-practice becomes somewhat meaningless. But we should be pretty suspicious of people claiming extremely accurate probabilities at the 10^-10 mark if they aren’t even accurate at the 10^-1 mark.
In general I’m not a fan of this particular form of epistemic anarchy where people say that they can’t know anything with enough precision under uncertainty to give numbers, and then act as if their verbal non-numeric intuitions are enough to carry them through consistently making accurate decisions.
It’s easy to lie (including to yourself) with numbers, but it’s even easier to lie without them.
Hi Linch!
I’d rather not rely on the authority of past performance to gauge whether someone’s arguments are good. I think we should evaluate the arguments directly. If they are, they’ll stand on their own regardless of someone’s prior luck/circumstance/personality.
I would actually argue that it’s the opposite of epistemic anarchy. Admitting that we can’t know the unknowable changes our decision calculus: Instead of focusing on making the optimal decision, we recognize that all decisions will have unintended negative consequences which we’ll have to correct. Fostering an environment of criticism and error-correction becomes paramount.
I’d disagree. I think trying to place probabilities on inherently unknowable events lends us a false sense of security.
(All said with a smile of course :) )
You or other readers might find this post of mine from last year of interest: Potential downsides of using explicit probabilities.
The potential downsides I cover include causing overconfidence, underestimating the value of information, and anchoring, among other things that are less directly related to your point. That said, I ultimately conclude that:
Relatedly, I think it’s not at all obvious that putting numbers on things, forecasting, etc. would tend to get in the way of “Fostering an environment of criticism and error-correction becomes paramount”. (It definitely could get in the way sometimes; it depends on the details.) There are various reasons why putting numbers on things and making forecasts can be actively helpful in fostering such an environment (some of which I discuss in my post).
[Disclaimer that I haven’t actually read your post yet—sorry! - though I may do so soon :)]
I agree that we should often/usually evaluate arguments directly. But:
We have nowhere near enough time to properly evaluate all arguments relevant to our decisions. And in some cases, we also lack the relevant capabilities. So in effect, it’s often necessary and/or wise to base certain beliefs mostly on what certain other people seem to believe.
For example, I don’t actually know that much about how climate science works, and my object-level understanding of the arguments for climate change being real, substantial, and anthropogenic are too shallow for me to be confident—on that basis alone—that those conclusions are correct. (I think a clever person could’ve made false claims about climate science sound similarly believable to me, if they’d been motivated to do so and I’d only looked into it to the extent that I have.)
The same is more strongly true for people with less education and intellectual curiosity than me.
But it’s good for us to default to being fairly confident that things most relevant scientists agree are true are indeed true.
The same basic point is even more clearly true when it comes to things like the big bang or the fact that dinosaurs existed and when they did so
See also epistemic humility
We can both evaluate arguments directly and consider people’s track records
We could also evaluate the “meta argument” that “people who have been shown to be decent forecasters (or better forecasters than other people are) on relatively short time horizons will also be at least slightly ok forecasts (or at least slightly better forecasters than other people are) on relatively long time horizons”
Evaluating that argument directly, I think we should land on “This seems more likely to be true than not, though there’s still room for uncertainty”
See also How Feasible Is Long-range Forecasting?, and particularly footnote 17
Another way of making a perhaps similar point is that it very often makes sense to see past outcomes from some person/object/process or whatever as at least a weak indicator of what the future outcomes from that same thing will be
E.g., the more often a car has failed to start up properly in the past, the more often we should expect it to do so in future
E.g., the more a person has done well at a job in the past, the more we should expect them to do well at that job or similar jobs in future
It’s not clear why this would fail to be the case for forecasting
And indeed, there is empirical evidence that it is the case for forecasting
That said, there is the issue that we’re comparing forecasts over short time horizons to forecasts over long time horizons, and that does introduce some more room for doubt, as noted above
What Linch was talking about seems very unlikely to boil down to just “someone’s prior luck/circumstance/personality”.
Actual track records would definitely not be a result of personality except inasmuch as personality is actually relevant to better performance (e.g. via determination to work hard at forecasting).
They’re very likely partly due to luck, but the evidence shows that some forecasters tend to do better over a large enough set of questions that it can’t be just due to luck (I have in mind Tetlock’s work).
I appreciate this tangential to the main point of the post, but these asides strike me as (unintentionally) likely to leave the reader with a common-but-inaccurate impression, and I think it’s worth correcting this impression as it arises in the name of general integrity and transparency.
Specifically, I think a reader of the above without further context would assume that longtermism is very new (say <2 years old, perhaps the age of the 2019 working paper), is basically just getting off the ground in EA, and it receives a small-but-significant amount of funding/time to date but has the potential to absorb much more.
The actual state of the world as I understand it is as follows:
Open Philanthropy, as of Jan 2018, plan to donate more than half of their money to longtermist areas. In particular, I note that their grants to date in this area completely swamp the $4.5m given through the Long-Term Future Fund.
The EA Survey suggests that ‘Long Term Future’ causes, broadly defined, are the top priority of 41% of the survey population.
80000 hours has a (commendably explicit) longtermist outlook and prioritises career paths largely on this basis.
As of May 2018, CEA’s Current Thinking page stated they believed “the most effective opportunities to do good are aimed at helping the long-term future”. Note this page has a disclaimer saying this no longer reflects their current views; I do not know whether this section was part of what they no longer think. Still, the fact it was true at time of writing is relevant for assessing the age of this framework within the movement.
The overall picture this paints is of longtermism as the current dominant framework of the movement, at least within the core. I think it has been for at least three years. Questions of whether long-term causes dominate the portfolio, far from being settled, are very sensitive to whether we are counting only-money or money + time, and how broadly we define the EA community. The narrower the definition of ‘EA’ and the more you look at non-financial input, the more long-term causes dominate.
Thanks AGB, this is helpful.
I agree that longtermism is core part of the movement, and probably commands a larger share of adherents than I imply. However, I’m not sure to what extent strong longtermism is supported. My sense is that while most people agree with the general thrust of the philosophy, many would be uncomfortable with “ignoring the effects” of the near term, and remain focused on near-term problems. I didn’t want to claim that a majority of EAs supported longtermism broadly-defined, but then only criticize a subset of those views.
I hadn’t seen the results of the EA Survey—fascinating.
I know I’m late to the discussion, but…
I agree with AGB’s comment, but I would also like to add that strong longtermism seems like a moral perspective with much less “natural” appeal, and thus much less ultimate growth potential, than neartermist EA causes such as global poverty reduction or even animal welfare.
For example, I’m a Program Officer in the longtermist part of Open Philanthropy, but >80% of my grantmaking dollars go to people who are not longtermists (who are nevertheless doing work I think is helpful for certain longtermist goals). Why? Because there are almost no longtermists anywhere in the world, and even fewer who happen to have the skills and interests that make them a fit for my particular grantmaking remit. Meanwhile, Open Philanthropy makes far more grants in neartermist causes (though this might change in the future), in part because there are tons of people who are excited about doing cost-effective things to help humans and animals who are alive and visibly suffering today, and not so many people who are excited about trying to help hypothetical people living millions of years in the future.
Of course to some degree this is because longtermism is fairly new, though I would date it at least as far back as Bostrom’s “Astronomical Waste” paper from 2003.
I would also like to note that many people I speak to who identify (like me) as “primarily longtermist” have sympathy (like me) for something like “worldview diversification,” given the deep uncertainties involved in the quest to help others as much as possible. So e.g. while I spend most of my own time on longtermism-motivated efforts, I also help out with other EA causes in various ways (e.g. this giant project on animal sentience), and I link to or talk positively about GiveWell top charities a lot, and I mostly avoid eating non-AWA meat, and so on… rather than treating these non-longtermist priorities as a rounding error. Of course some longtermists take a different approach than I do, but I’m hardly alone in my approach.
Yikes… now I’m even more worried … :|
Thanks so much for writing this Ben! I think it’s great that strong longtermism is being properly scrutinised, and I loved your recent podcast episode on this (as well as Vaden’s piece).
I don’t have a view of my own yet; but I do have some questions about a few of your points, and I think I can guess at how a proponent of strong longtermism might respond to others.
For clarity, I’m understanding part of your argument as saying something like the following. First, “[E]xpected value calculations, Bayes theorem, and mathematical models” are tools — often useful, often totally innapropriate or inapplicable. Second, ‘Bayesian epistemology’ (BE) makes inviolable laws out of these tools, running into all kinds of paradoxes and failing to represent how scientific knowledge advances. This makes BE silly at best and downright ‘refuted’ at worst. Third, the case for strong longtermism relies essentially on BE, which is bad news for strong longtermism.
I can imagine that a fan of BE would just object that Bayesianism in particular is just not a tool which can be swapped out for something else when it’s convenient . This feels like an important but tangential argument — this LW post might be relevant. Also, briefly, I’m not 100% convinced by Popper’s argument against Bayesianism which you’re indirectly referencing, and I haven’t read the paper Vaden wrote but it looks interesting. In any case: declaring that BE “has been refuted” seems unfairly rash.
You suggest at a few points that longtermists are just pulling numbers out of nowhere in order to take an expectation over, for instance, the number of people who will live in the long-run future. In other words, I’m reading you as saying that these numbers are totally arbitrary. You also mention that they’re problematically unfalsifiable.
On the first point, it feels more accurate to say that these numbers are highly uncertain rather than totally arbitrary. I can imagine someone saying “I wouldn’t be surprised if my estimate were off by several orders of magnitude”; but not “I have literally no reason to believe that this estimate is any better than a wildly different one”. That’s because it is possible to begin reasoning about these numbers. For instance, I was reminded of Nick Beckstead’s preliminary review of the feasibility of space colonisation. If it turned out that space colonisation was practically impossible, the ceiling would fall down on estimates for the size of humanity’s future. So there’s some information to go on — just very little.
You make the same point in the context of estimating existential risks:
Really? If you’re a rationalist (in the broad Popperian sense and the internet-cult sense), and we share common knowledge of each other’s beliefs, then shouldn’t we be able to argue towards closer agreement? Not if our estimates were totally arbitrary — but clearly they’re not. Again, they’re just especially uncertain.
You can use bigger numbers in the sense that you can type extra zeroes on your keyboard, but you can’t use bigger numbers if you care about making sure your numbers fall reasonably in line with the available facts, right? I could try turning “donating to Fin’s retirement fund” into an EA cause area by just lying about its impact, but there are norms of honesty and criticism (and common sense) which would prevent the plot succeeding. Because I don’t think you’re suggesting that proponents of strong longtermism are being dishonest in this way, I’m confused about what you are suggesting.
Plus, as James Aung mentioned, I don’t think it works to criticise subjective probabilities (and estimates derived from them) as too precise. The response is presumably: “sure, this guess is hugely uncertain. But better to give some number rather than none, and any number I pick is going to seem too precise to you. Crucially, I’m trying to represent something about my own beliefs — not that I know something precise about the actual world.”
On the falsifiability point, estimates about the size of humanity’s future clearly are falsifiable — it’s just going to take a long time to find out. But plenty of sensible scientific claims are like this — e.g. predictions about the future of stars including our Sun. So the criticism can’t be that predictions about the size of humanity’s future are somehow unscientific because not immediately falsifiable.
I think this paragraph is key:
My reaction is something like this: even if other interpretations of probability are available, it seems at least harmless to form subjective credences about the effectiveness of, say, global health interventions backed by a bunch of RCTs. Where there’s lots of empirical evidence, there should be little daylight between your subjective credences and the probabilities that fall straight out of the ‘actual data’. In fact, using subjective credences begins to look positively useful when you venture into otherwise comparable but more speculative interventions. That’s because whether you might want to fund that intervention is going to depend on your best guess about its likely effects and what you might learn from them, and that guess should be sensitive to all kinds of information — a job Bayesian methods were built for. However, if you agree that subjective credences are applicable to innocuous ‘short-term’ situations with plenty of ‘data’, then you can imagine gradually pushing the time horizon (or some other source of uncertainty) all the way to questions about the very long-run future. At this extreme, you’ve said that there’s something qualitatively wrong with subjective credences about such murky questions. But I want to say: given that you can join up the two kinds of subject matter by a series of intermediate questions, and there wasn’t originally anything wrong with using credences and no qualitative or step-change, why think that the two ends of the scale end up being “of an entirely different nature”? I think this applies to Vaden’s point that the maths of taking an expectation over the long-run future is somehow literally unworkable, because you can’t have a measure over infinite possibilities (or something). Does that mean we can’t take an expectation over what happens next year? The next decade?
I hope that makes sense! Happy to say more.
My last worry is that you’re painting an unrealistically grim picture of what strong longtermism practically entails. For starters, you say “[l]ongtermism asks us to ignore problems now”, and Hilary and Will say we can “often” ignore short-term effects . Two points here: first, in situations where we can have a large effect on the present / immediate future without risking something comparably bad in the future, it’s presumably still just as good to do that thing. Second, it seems reasonable to expect considerable overlap between solving present problems and making the long-run future go best, for obvious reasons. For example, investing in renewables or clean meat R&D just seem robustly good from short-term and long-term perspectives.
I’m interested in the comparison to totalitarian regimes, and it reminded me of something Isaiah Berlin wrote:
However, my guess is that there are too few similarities for the comparison to be instructive. I would want to say that the totalitarian regimes of the past failed so horrendously not because they used expected utility theory or Bayesian epistemology correctly but innapropriately, but because they were just wrong — wrong that revolutionary violence and totalitarianism make the world remotely better in the short or long term. Also, note that a vein of longtermist thinking discusses reducing the likelihood of a great power conflict, improving instutional decision-making, and spreading good (viz. liberal) political norms in general — in other words, how to secure an open society for our descendants.
Isn’t it the case that strong longtermism makes knowledge creation and accelerating progress seem more valuable, if anything? And would the world really generate less knowledge, or progress at a slower rate, if the EA community shifted priorities in a longtermist direction?
Finally, a minor point: my impression is that ‘longtermism’ is generally taken to mean something a little less controversial than ‘strong longtermism’. I appreciate you make the distinction early on, but using the ‘longtermism’ shorthand seems borderline misleading when some of your arguments only apply to a specific version.
For what it’s worth, I’m most convinced by the practical problems with strong longtermism. I especially liked your point about longtermism being less permeable to error correction, and generally I’m curious to know more about reasons for thinking that influencing the long-run future is really tractable. Thanks again for starting this conversation along with Vaden!
I think there is an important point here. One of the assumptions in Aumann’s theorem is that both people have the same prior, and I think this is rarely true in the real world.
I roughly think of Bayesian reasoning as starting with a prior, and then adjusting the prior based on observed evidence. If there’s a ton of evidence, and your prior isn’t dumb, the prior doesn’t really matter. But the more speculative the problem, and the less available evidence, the more the prior starts to matter. And your prior bakes in a lot of your assumptions about the world, and I think it’s tricky to resolve disagreements about what your prior should be. At least not in ways that approach being objective.
I think you can make progress on this. Eg, ‘how likely is it that AI could get way better, really fast?’ is a difficult question to answer, and could be baked into a prior either way. And things like AI Impact’s study of discontinuous progress in other technologies can be helpful for getting closer to consensus. But I think choosing a good prior is still a really hard and important problem, and near impossible to be objective about
Hey Fin! Nice—lot’s here. I’ll respond to what I can. If I miss anything crucial just yell at me :) (BTW, also enjoying your podcast. Maybe we should have a podcast battle at some point … you can defend longtermism’s honour).
Yep, this is fair. I’m imagining myself in the position of some random stranger outside of a fancy EA-gala, and trying to get people’s attention. So yes—the language might be a little strong (although I do really think Bayesianism doesn’t stand up to scrutiny if you drill down on it).
Sure, guessing that there will be between 1 billion and 1000 quadrillion people in the future is probably a better estimate than 1000 people. But it still leaves open a discomfortingly huge ran. Greaves and MacAskill could easily have used half a quadrillion people, or 10 quadrillion people. Instead of trying to wrestle with this uncertainty, which is fruitless, we should just acknowledge that we can’t know and stop trying.
Bit of a nitpick here, but space colonization isn’t prohibited by the laws of physics, so it can only be “practically impossible” based on our current knowledge. It’s just a problem to be solved. So this particular example couldn’t bring down the curtains on our expected value calculations.
I don’t think so. There’s no data on the problem, so there’s nothing to adjudicate between our disagreements. We can honestly try this if you want. What’s your credence?
Now, even if we could converge on some number, what’s the reason for thinking that number captures any aspect of reality? Most academics were sympathetic to communism before it was tried; most physicists thought Einstein was wrong.
What are the available facts when it comes to the size of the future? There’s a reason these estimates are wildly different across papers: From 10^15 here, to 10^68 (or something) from Bostrom, and everything in between. I’m gonna add mine in: 10^124 + 3.
Agree that this is probably the response. But then we need to be clear that these estimates aren’t saying “anything precise about the actual world.” They should be treated completely differently than estimates based on actual data. But they’re not. When Greaves and MacAskill compare how many lives are saved by donating to AI safety versus the AMF, they compare these numbers as if they were equally as reliable and equally as capable of capturing something about reality.
There should be no daylight. Whatever daylight there is would have to be a result of purely subjective beliefs, and we shouldn’t lend this any credibility. It doesn’t belong alongside an actual statistical estimate.
I think the above also answers this? Subjective credences aren’t applicable to short term situations. (Again, when I say “subjective” there’s an implied “and based on no data”).
I’ve seen arguments to the contrary. Here for instance:
There’s also the quote by Toby Ord (I think?) that goes something like: “We’ve grown technologically mature without acquiring the commensurate wisdom.” I take the implication here to be that we should stop developing technology and wait for our wisdom to catch up. But this misses how wisdom is generated in the first place: by solving problems.
When you believe the fate of an untold number of future people is on the line, then you can justify almost anything in the present. This is what I find so disturbing about longtermism. I find many of the responses to my critique say things like: “Look, longtermism doesn’t mean we should throw out concern for the present, or be focused on problem-solving and knowledge creation, or continue improving our ethics”. But you can get those things without appealing to longtermism. What does longtermism buy you that other philosophies don’t, except for headaches when trying to deal with insanely big numbers? I see a lot of downsides, and no benefits that aren’t there in other philosophies. (Okay, harsh words to end, I know—but if anyone is still reading at this point I’m surprised ;) )
I don’t think this is true. Whenever Greaves and MacAskill carry out a longtermist EV calculation in the paper it seems clear to me that their aim is to illustrate a point rather than calculate a reliable EV of a longtermist intervention. Their world government EV calculation starts with the words “suppose that...”. They also go on to say:
This is the point they are trying to get across by doing the EV calculations.
Thanks for replying Ben, good stuff! Few thoughts.
I’ll concede that point!
I think a better response to the one I originally gave was to point out that the case for strong longtermism relies on establishing a sensible lower(ish) bound for total future population. Greaves and MacAskill want to convince you that (say) at least a quadrillion lives could plausibly lie in the future. I’m curious if you have an issue with that weaker claim?
I think your point about space exploration is absolutely right, and more than a nitpick. I would say two things: one is that I can imagine a world in which we could be confident that we would never colonise the stars (e.g. if the earth were more massive and we had 5 decades before the sun scorched us or something). Second, voicing support for the ‘anything permitted by physics can become practically possible’ camp indirectly supports an expectation of a large numbers of future lives, no?
Hmm — to my lights Greaves and MacAskill are fairly clear about the differences between the two kinds of estimate. If your reply is that doing any kind of (toy) EV calculation with both estimates just implies that they’re somehow “equally as capable of capturing something about reality”, then it feels like you’re begging the question.
I don’t understand what you mean here, which is partly my fault for being unclear in my original comment. Here’s what I had in mind: suppose you’ve run a small-scale experiment and collected your data. You can generate a bunch of statistical scores indicating e.g. the effect size, plus the chance of getting the results you got assuming the null hypothesis was true (p-value). Crucially (and unsurprisingly) none of those scores directly give you the likelihood of an effect (or the ‘true’ anything else). If you have reason to expect a bias in the direction of positive results (e.g. publication bias), then your guess about how likely it is that you’re picked up on a real effect may in fact be very different from any statistic, because it makes use of information from beyond those statistics (i.e. your prior). For instance, in certain social psych journals, you might pick a paper at random, see that p < 0.05, and nonetheless be fairly confident that you’re looking at a false positive. So subjective credences (incorporating info from beyond the raw stats) do seem useful here. My guess is that I’m misunderstanding you, yell at me if I am.
By ‘subjective credence’ I just mean degree of belief. It feels important that everyone’s on the same terminological page here, and I’m not sure any card-carrying Bayesians imply “based on no data” by “subjective”! Can you point me towards someone who has argued that subjective credences in this broader sense aren’t applicable even to straightforward ‘short-term’ situations?
Fair point about strong longtermism plausibly recommending slowing certain kinds of progress. I’m also not convinced — David Deutsch was an influence here (as I’m guessing he was for you). But the ‘wisdom outrunning technological capacity’ thing still rings true to me.
There’s two ways to close the gap, of course, and isn’t the obvious conclusion just to speed up the ‘wisdom’ side?
Which ties in to your last point. Correct me if I’m wrong, but I’m taking you as saying: to the extent that strong longtermism implies significant changes in global priorities, those changes are really worrying: the logic can justify almost any present sacrifices, there’s no closed feedback loop or error-correction mechanism, and it may imply a slowing down of technological progress in some cases. To the extent that strong longtermism doesn’t imply significant changes in global priorities, then it hardly adds any new or compelling reasons for existing priorities. So it’s either dangerous or useless or somewhere between the two.
I won’t stick up for strong longtermism, because I’m unsure about it, but I will stick up for semi-skimmed longtermism. My tentative response is that there are some recommendations that (i) are more-or-less uniquely recomended by this kind of longtermism, and (ii) not dangerous or silly in the ways you suggest. One example is establishing kinds of political representation for future generations. Or funding international bodies like the BWC, spreading long-term thinking through journalism, getting fair legislative frameworks in place for when transformative / general AI arrives, or indeed for space governance.
Anyway, a crossover podcast on this would be amazing! I’ll send you a message.
I share your concerns with using arbitrary numbers and skepticism of longtermism, but I wonder if your argument here proves too much. It seems like you’re acting as if you’re confident that the number of people in the future is not huge, or that the interventions are otherwise not so impactful (or they do more harm than good), but I’m not sure you actually believe this. Do you?
It sounds like you’re skeptical of AI safety work, but it also seems what you’re proposing is that we should be unwilling to commit to beliefs on some questions (like the number of people in the future), and then deprioritize longtermism as a result, but, again, doing so means acting as if we’re committed to beliefs that would make us pessimistic about longtermism.
I think it’s more fair to think that we don’t have enough reason to believe longtermist work does much good at all, or more good than harm (and generally be much more skeptical of causal effects with little evidence), than it is to be extremely confident that the future won’t be huge.
I think you do need to entertain arbitrary probabilities, even if you’re not a longtermist, although I don’t think you should commit to a single joint probability distribution. You can do a sensitivity analysis.
Here’s an example: how do we decide between human-focused charities and animal charities, given the pretty arbitrary nature of assigning consciousness probabilities to nonhuman animals and the very arbitrary nature of assigning intensities of suffering to nonhuman animals?
I think the analogous response to your rejection of longtermism here would be to ignore your effects on animals, not just with donations or your career, but in your everyday life, too. But, based on this conclusion, we could reverse engineer what kinds of credences you would have to commit to if you were a Bayesian to arrive at such a conclusion (and there could be multiple compatible joint distributions). And then it would turn out you’re acting as if you’re confident that factory farmed chickens suffer very little (assuming you’re confident in the causal effects of certain interventions/actions), and you’re suggesting everyone else should act as if factory farmed chickens suffer very little.
Hi Michael!
I have no idea about the number of future people. And I think this is the only defensible position. Which interventions do you mean? My argument is that longtermism enables reasoning that de-prioritizes current problems in lieu of possible, highly uncertain, future problems. Focusing on such problems prohibits us from making actual progress.
I’m not quite sure I’m following this criticism, but I think it can be paraphrased as: You refuse to commit to a belief about x, but commit to one about y and that’s inconsistent. (Happy to revise if this is unfair.) I don’t think I agree—would you commit to a belief about what Genghis Khan was thinking on his 17th birthday? Some things are unknowable, and that’s okay. Ignorance is par for the course. We don’t need to pretend otherwise. Instead, we need a philosophy which is robust to uncertainty which, as I’ve argued, is one that focuses on correcting mistakes and solving the problems in front of us.
… but they’d be arbitrary, so by definition don’t tell us anything about the world?
This is of course a difficult question. But I don’t think the answer is to assign arbitrary numbers to the consciousness of animals. We can’t pull knowledge out of a hat, even using the most complex maths possible. We have theories of neurophysiology, and while none of them conclusively tells us that animals definitely feel pain, I think that’s the best explanation of our current observations. So, acknowledging this, we are in a situation where billions of animals needlessly suffer every year according to our best theory. And that’s a massive, horrendous tragedy—one that we should be fighting hard to stop. Assigning credences to the consciousness of animals just so we can start comparing this to other cause areas is just pretending knowledge where we have none.
I would rephrase as “You say you refuse to commit to a belief about x, but seem to act as if you’ve committed to a belief about x”. Specifically, you say you have no idea about the number of future people, but it seems like you’re saying we should act as if we believe it’s not huge (in expectation). The argument for strong longtermism you’re trying to undermine (assuming we get the chance of success and sign roughly accurate, which to me is more doubtful) goes through for a wide range of numbers. It seems that you’re committed to the belief that expected number is less than 1015, say, since you write in response “This paragraph illustrates one of the central pillars of longtermism. Without positing such large numbers of future people, the argument would not get off the ground”.
Maybe I’m misunderstanding. How would you act differently if you were confident the number was far less than 1015 in expectation, say 1012 (about 100 times the current population), rather than have no idea?
There are certainly things I would commit to believing he was not thinking about, like modern digital computers (probability > 1−10−9), and I’d guess he thought about food/eating at some point during the day (probability > 0.5). Basically, either he ate that day (more likely than not) and thought about food before or while eating, or he didn’t eat and thought about food because he was hungry. Picking precise numbers would indeed be fairly arbitrary and even my precise bounds are pretty arbitrary, but I think these bounds are useful enough to make decisions based on if I had to, possibly after a sensitivity analysis.
If I were forced to bet on whether Genghis Khan thought about food on a randomly selected day during his life (randomly selected to avoid asymmetric information), I would bet yes.
I agree, but also none of these theories tell us how much a chicken can suffer relative to humans, as far as I know, or really anything about this, which is important in deciding how much to prioritize them, if at all. There are different suggestions for how the amount of suffering scales with brain size within the EA community, and there are arguments for these, but they’re a priori and fairly weak. This is one of the most recent discussions.
Thanks for taking the time to write this :)
In your post you say “Of course, it is impossible to know whether $1bn of well-targeted grants could reduce the probability of existential risk, let alone by such a precise amount. The “probability” in this case thus refers to someone’s (entirely subjective) probability estimate — “credence” — a number with no basis in reality and based on some ad-hoc amalgamation of beliefs.”
I just wanted to understand better: Do you think its ever reasonable to make subjective probability estimates (have ‘credences’) over things? If so, in what scenarios is it reasonable to have such subjective probability estimates; and what makes those scenarios different from the scenario of forming a subjective probability estimate of what $1bn in well-target grants could do to reduce existential risk?
Hey James!
Answering this in its entirety would take a few more essays, but my short answer is: When there are no data available, I think subjective probability estimates are basically useless, and do not help in generating knowledge.
I emphasize the condition when there are no data available because data is what allows us to discriminate between different models. And when data is available, well, estimates become less subjective.
Now, I should say that I don’t really care what’s “reasonable” for someone to do—I definitely don’t want to dictate how someone should think about problems. (As an aside, this is a pet peeve of mine when it comes to Bayesianism -it tells you how you must think in order to be a rational person. As if rationality was some law of nature to be obeyed.) In fact, I want people thinking about problems in many different ways. I want Eliezer Yudkowski applying Bayes’ rule and updating in strict accordance with the rules of probability, you being inspired by your fourth grade teacher, and me ingesting four grams of shrooms with a blindfold on in order to generate as many ideas as possible. But how do we discriminate between these ideas? We subject them to ruthless criticism and see which ones stand up to scrutiny. Assigning numbers to them doesn’t tell us anything (again, when there’s no underlying data).
In the piece I’m making a slightly different argument to the above, however. I’m criticizing the tendency for these subjective estimates to be compared with estimates derived from actual data. Whether or not someone agrees with me that Bayesianism is misguided, I would hope that they still recognize the problem in comparing numbers of the form “my best guess about x” with “here’s an average effect estimate with confidence intervals over 5 well-designed RTCs”.
As a major aside—there’s a little joke Vaden and I tell on the podcast sometimes when talking about Bayesianism vs Criticial Rationalism (an alternative philosophy first developed by Karl Popper). The joke is most certainly a strawman of Bayesianism, but I think it gets the point across.
Bob and Alice are at the bar, being served by Carol. Bob is trying to estimate whether Carol has children. He starts with a prior of 1⁄2. He then looks up the base rate of adults with children, and updates on that. Then he updates based on Carol’s age. And what car she drives. And the fact that she’s married. And so on. He pulls out a napkin, does some complex math, and arrives at the following conclusion: It’s 64.745% likely that Carol has children. Bob is proud of his achievement and shows the napkin to Alice. Alice leans over the bar and asks “Hey Carol—do you have kids?”.
Now, obviously this is not how the Bayesian acts in real life. But it demonstrates the care the Bayesian takes in having correct beliefs; about having the optimal brain state. I think this is the wrong target. Instead, we should be seeking to falsify as many conjectures as possible, regardless of where the conjectures came from. I don’t care what Alice thought the probability was before she asked the question, only about the result of the test.
Thanks for the reply and taking the time to explain your view to me :)
I’m curious: My friend has been trying to estimate the liklihood of nuclear war before 2100. It seems like this is a question that is hard to get data on, or to run tests on. I’d be interested to know what you’d recommend them to do?
Is there a way I can tell them to approach the question such that it relies on ‘subjective estimates’ less and ‘estimates derived from actual data’ more?
Or is it that you think they should drop the research question and do something else with their time, since any approach to the question would rely on subjective probability estimates that are basically useless?
Well, far be it from me to tell others how to spend their time, but I guess it depends on what the goal is. If the goal is to literally put a precise number (or range) on the probability of nuclear war before 2100, then yes, I think that’s a fruitless and impossible endeavour. History is not an iid sequence of events. If there is such a war, it will be the result of complex geopolitical factors based on human belief, desires, and knowledge at the time. We cannot pretend to know what these will be. Even if you were to gather all the available evidence we have on nuclear near misses, and generate some sort of probability based on this, the answer would look something like:
“Assuming that in 2100 the world looks the same as it did during the time of past nuclear near misses, and nuclear misses are distributionally similar to actual nuclear strikes, and [a bunch of other assumptions], then the probability of a nuclear war before 2100 is x”.
We can debate the merits of such a model, but I think it’s clear that it would be of limited use.
None of this is to say that we shouldn’t be working on nuclear threat, of course. There are good arguments for why this is a big problem that have nothing to do with probability and subjective credences.
You say that “there are good arguments for working on the threat of nuclear war”. As I understand your argument, you also say we cannot rationally distinguish between the claim “the chance of nuclear war in the next 100 years is 0.00000001%” and the claim “the chance of nuclear war in the next 100 years is 1%”. If you can’t rationally put probabilities on the risk of nuclear war, why would you work on it?
Why are probabilities prior to action—why are they so fundamental? Could Andrew Wiles “rationally put probabilities” on him solving Fermat’s Last Theorem? Does this mean he shouldn’t have worked on it? Arguments do not have to be in number form.
If you refuse to claim that the chance of nuclear war up to 2100 is greater than 0.000000000001%, then I don’t see how you could make a good case to work on it over some other possible intuitively trivial action, such as painting my wall blue. What would the argument be if you are completely agnostic as to whether it is a serious risk?
To me, the fundamental point isn’t probabilities, it’s that you need to make a choice about what you do. If I have the option to give a $1mn grant to preventing nuclear war or give the grant to something else, then no matter what I do, I have made a choice. And so, I need to have a decision theory for making a choice here.
And to me, subjective probabilities and Bayesian epistemology more generally, are by far the best decision theory I’ve come across for making choices under uncertainty. If there’s a 1% chance of nuclear war, the grant is worth making, if there’s a 10^-15 chance of nuclear war, the grant is not worth making. I need to make a decision, and so probabilities are fundamental, because they are my tool for making a decision.
And there are a bunch of important question where we don’t have data, and there’s no reasonable way to get data (eg, nuclear war!). And any approach which rejects the ability to reason under uncertainty in situations like this, is essentially the decision theory of “never make speculative grants like this”. And I think this is a clearly terrible decision theory (though I don’t think you’re actually arguing for this policy?)
Can you give some examples? I expect that someone could respond “That could be too unlikely to matter enough” to each of them, since we won’t have good enough data.
Sure—Nukes exist. They’ve been deployed before, and we know they have incredible destructive power. We know that many countries have them, and have threatened to use them. We know the protocols are in place for their use.
To me this seems like you’re making a rough model with a bunch of assumptions like that past use, threats and protocols increase the risks, but not saying by how much or putting confidences or estimates on anything (even ranges). Why not think the risks are too low to matter despite past use, threats and protocols?
But we also have to make similar (although less strong) assumptions and have generalization error even with RCTs. Doesn’t GiveWell make similar assumptions about the impacts of most of their recommended charities? As far as I know, there are recent studies of GiveDirectly’s effects, but the “recent” studies of the effects of the interventions of the other charities have probably had their samples chosen years ago, so their effects might not generalize to new locations. Where’s the cutoff for your skepticism? Should we boycott the GiveWell-recommended charities whose ongoing intervention impacts of terminal value (lives saved, quality of life improvements) are not being measured rigorously in their new target areas, in favour of GiveDirectly?
To illustrate the issue of generalization, GiveWell did a pretty arbitrary adjustment for El Niño for deworming, although I think this is the most suspect assumption I’ve seen them make.
See Eva Vivalt’s research on generalization (in the Causal Inference section) or her talk here.
Yes, we do! And the strength of those assumptions is key. Our skepticism should rise in proportion to the number/feasibility of the assumptions. So you’re definitely right, we should always be skeptical of social science research—indeed, any empirical research. We should be looking for hasty generalizations, gaps in the analysis, methodological errors etc., and always pushing to do more research. But there’s a massive difference between the assumptions driving GiveWell’s models and the assumptions required in the nuclear threat example.
Hey Ben, thanks a lot for posting this! And props for having the energy to respond to all these comments :)
I’ll try to reframe points that others have made in the comments (and which I tried to make earlier, but less well): I suspect that part of why these conversations sometimes feel like we’re talking past one another is that we’re focusing on different things.
You and Vaden seem focused on creating knowledge. You (I’d say) correctly note that, as frameworks for creating knowledge, EV maximization and Bayesian epistemology aren’t just useless—they’re actively harmful, because they distract us from the empirical studies, data analysis, feedback loops, and argumentative criticism that actually create knowledge.
Some others are focused on making decisions. From this angle, EV maximization and Bayesian epistemology aren’t supposed to be frameworks for creating knowledge—they’re frameworks for turning knowledge into decisions, and your arguments don’t seem to be enough for refuting them as such.
To back up a bit, I think probabilities aren’t fundamental to decision making. But bets are. Every decision we make is effectively taking or refusing to take a bet (e.g. going outside is betting that I won’t be hit in the head by a meteor if I go outside). So it’s pretty useful to have a good answer to the question: “What bets should I take?”
In this context, your post isn’t convincing me because I don’t see a good alternative to the EV approach to making bets, and because maybe there can’t be a good alternative.
1.
One of the questions your post leaves me with is: What kinds of bets do you think I should I take, when I’m uncertain about what will happen? i.e. How do you think I should make decisions under uncertainty?
Maximizing EV under a Bayesian framework offers one answer, as you know, roughly that: we should be willing to bet on X happening in proportion to our best guess about the strength of the evidence for the claim that X will happen.
I think you’re right in pointing that this approach has significant weaknesses: it has counterintuitive results when used with some very low probabilities, it’s very sensitive to arbitrary judgements and bias, and our best guesses about whether far-future events will happen might be totally uncorrelated with whether they actually happen. (I’m not as compelled by some of your other criticisms, largely for reasons others’ comments discuss.)
Despite these downsides, it seems like a bad idea to drop the EV approach to “what kinds of bets should I take?” without a better answer. (Vaden offers a promising approach to making decisions, but it just passes the buck on this—we’ll still need an answer to my question when we get to his step 2.) As your familiarity with catastrophic dictatorships suggests, dumping a flawed status quo is a mistake if we don’t have a better alternative.
2.
Another worry is that probabilities are so useful that we won’t find a better alternative.
I think of probabilities as language for answering the earlier basic question of “What bets should I make?” For example, “There’s a 25% chance (i.e. 1:3 odds) that X will happen” is (as I see it) shorthand for “My potential payoff better be at least 3 times bigger than my potential loss for betting on X to be worth it.” So probabilities express thresholds in your answers to the question “What bets on event X should I take?” That is, from a pragmatic angle, subjective probabilities aren’t supposed to be deep truths about the world; they’re expressions of our best guesses about how willing we should be to bet on various events. (Other considerations also make probabilities particularly well-fitting tools for describing our preferences about bets.)
So rejecting the use of probabilities (as I understand them) under severe uncertainty seems to have an unacceptable, maybe even absurd, conclusion: the rejection of consistent thresholds for deciding whether to bet on uncertain events. This is a mistake—if we accept/reject bets on some event without a consistent threshold for what reward:loss ratios are worth taking, then we’ll necessarily be doing silly things like refusing to take a bet, and then accepting a bet on the same event for a less favorable reward:loss ratio.
You might be thinking something like “ok, so you can always describe an action as endorsing some betting threshold, but that doesn’t mean it’s useful to think about this explicitly.” I’d disagree, because not recognizing our betting threshold makes it harder to notice and avoid mistakes like the one above. It also takes away clarity and precision of thought that’s helpful for criticizing our choices, e.g. it makes an extremely high betting threshold about the value of x-risk reduction look like agnosticism.
Thanks again for your thoughtful post!
Hey Mauricio! Two brief comments -
Yes agreed, but these two things become intertwined when a philosophy makes people decide to stop creating knowledge. In this case, it’s longtermism preventing the creation of moral and scientific knowledge by grinding the process of error correction to a halt, where “error correction” in this context means continuously reevaluating philanthropic organizations based on their near and medium term consequences, in order to compare results obtained against results expected.
Both approaches pass on the buck, that’s why I defined ‘creativity’ here to mean: ‘whatever unknown software the brain is running to get out of the infinite regress problem.’ And one doesn’t necessarily need to answer your question, because there’s no requirement that the criticism take EV form (although it can).
Hey Vaden, thanks!
Yeah, fair. (Although less relevant to less naive applications of this philosophy, which as Ben puts it draw some rather than all of our attention away from knowledge creation.)
I’m not sure I see where you’re coming from here. EV does pass the buck on plenty of things (on how to generate options, utilities, probabilities), but as I put it, I thought it directly answered the question (rather than passing the buck) about what kinds of bets to make/how to act under uncertainty:
Also, regarding this:
I don’t see how that gets you out of facing the question. If criticism uses premises about how we should act under uncertainty (which it must do, to have bearing on our choices), then a discussion will remain badly unfinished until it’s scrutinized those premises. We could scrutinize them on a case-by-case basis, but that’s wasting time if some kinds of premises can be refuted in general.
Check out chapter 13 in Beginning of Infinity when you can—everything I was saying in that post is much better explained there :)
I’d like to make a point about the potential importance of working on current problems which I’m unsure has been made yet (apologies if I’ve missed it).
It seems to me that there are two possibilities here:
Working on current problems allows us to create moral and scientific knowledge that will help us make the long-run future go well
The above isn’t true
If number 1 is the case, a strong longtermist should agree with you and vadmas about the importance of working on current problems.
If number 2 is the case a strong longtermist may not agree with you about the importance of working on current problems either because they don’t think that working on near term problems will generate much knowledge or because they don’t think the knowledge that would be generated will help that much in making the long-run future go well.
Now there are two points I would like to make.
Firstly, you and vadmas seem to assume number 2 is the case. It seems important to me to note that this is certainly not a given.
Secondly you and vadmas seem to think that if number 2 is the case then the conclusion that we shouldn’t work on near-term problems for knowledge creation in some way demonstrates the abusurdity of strong longtermism. I’m genuinely not sure why this would be the case. It’s almost as if you think knowledge creation has some strong intrinsic value and any conclusion that concludes it isn’t important to boost knowledge creation must therefore necessarily be wrong.
Do you in fact think that knowledge creation has strong intrinsic value? I, and I suspect most EAs, only think knowledge creation is instrumentally valuable.
Oops nope the exact opposite! Couldn’t possibly agree more strongly with
Perfect, love it, spot on. I’d be 100% on board with longtermism if this is what it’s about—hopefully conversations like these can move it there. (Ben makes this point near the end of our podcast conversation fwiw)
Well, both. I do think it’s intrinsically valuable to learn about reality, and I support research into fundamental physics, biology, history, mathematics, ethics etc for that reason. I think it would be intellectually impoverishing to only support research that has immediate and foreseeable practical benefits. But fortunately knowledge creation also has enormous instrumental value. So it’s not a one-or-the other thing.
I have to admit that I’m slightly confused as to where the point of contention actually is.
If you believe that working on current problems allows us to create moral and scientific knowledge that will help us make the long-run future go well, then you just need to argue this case and if your argument is convincing enough you will have strong longtermists on your side.
More importantly though I’m not sure people actually do in fact disagree with this. I haven’t come across anyone who has publicly disagreed with this. Have you? It may be the case that both you and strong longtermists are actually on the exact same page without even realising it.
I don’t consider human extermination by AI to be a ‘current problem’ - I think that’s where the disagreement lies. (See my blogpost for further comments on this point)
Either way, the problems to work on would be chosen based on their longterm potential. It’s not clear that say global health and poverty would be among those chosen. Institutional decision-making and improving the scientific process might be better candidates.
I feel a bit confused reading that. I’d thought your case was framed around a values disagreement about the worth of the long-term future. But this feels like a purely empirical disagreement about how dangerous AI is, and how tractable working on it is. And possibly a deeper epistemological disagreement about how to reason under uncertainty.
How do you feel about the case for biosecurity? That might help disentangle whether the core disagreement is about valuing the longterm future/x-risk reduction, vs concerns about epistemology and empirical beliefs, since I think the evidence base is noticeably stronger than for AI.
I think there’s a pretty strong evidence base that pandemics can happen and, eg, dangerous pathogens can get developed in labs and released from labs. And I think there’s good reason to believe that future biotechnology will be able to make dangerous pathogens, that might be able to cause human extinction, or something close to that. And that human extinction is clearly bad for both the present day, and the longterm future.
If a strong longtermist looks at this evidence, and concludes that biosecurity is a really important problem because it risks causing human extinction and thus destroying the value of the longterm future, and is a thus a really high priority, would you object to that reasoning?
Apologies, I do still need to read your blogpost!
It’s true existential risk from AI isn’t generally considered a ‘near-term’ or ‘current problem’. I guess the point I was trying to make is that a strong longtermist’s view that it is important to reduce the existential threat of AI doesn’t preclude the possibility that they may also think it’s important to work on near-term issues e.g. for the knowledge creation it would afford.
Granted any focus on AI work necessarily reduces the amount of attention going towards near-term issues, which I suppose is your point.
Yep :)
This wasn’t clearly worded in hindsight.
What I meant by this was that I think you and Ben both seem to assume that strong longtermists don’t want to work on near-term problems. I don’t think this is a given (although it is of course fair to say that they’re unlikely to only want to work on near-term problems).
Mostly agree here—this was the reason for some of the (perhaps cryptic) paragraphs in the Section “the Antithesis of Moral Progress.” Longtermism erodes our ability to make progress to whatever extent it has us not working on real problems. And, to the extent that it does have us working on real problems, then I’m not sure what longtermism is actually adding.
Also, just a nitpick on terminology—I dislike the term “near-term” problems, because it seems to imply that there is a well-defined class of “future” problems that we can choose to work on. As if there were a set of problems, and they could be classified as either short-term or long-term. But the fact is that the only problems are near-term problems; everything else is just a guess about what the future might hold. So I see it less about choosing what kinds of problems to work on, but a choice between working on real problems, or conjecturing about future ones, and I think the latter is actively harmful.
I don’t necessarily see working on reducing extinction risk as wildly speculating about the far future. In many cases these extinction risks are actually thought to be current risks. The point is that if they happen they necessarily curtail the far future.
I would note that the Greaves and MackAskill paper actually has a section putting forward ‘advancing progress’ as a plausible longtermist intervention! As I have mentioned this is only insofar as it will make the long-run future go well.
Agree with almost all of this. This is why it was tricky to argue against, and also why I say (somewhere? podcast maybe?) that I’m not particularly worried about the current instantiation of longtermism, but what this kind of logic could justify.
I totally agree that most of the existential threats currently tackled by the EA community are real problems (nuclear threats, pandemics, climate change, etc).
Yeah—but I found this puzzling. You don’t need longtermism to think this is a priority—so why adopt it? If you instead adopt a problem/knowledge focused ethics, then you get to keep all the good aspects of longtermism (promoting progress, etc), but don’t open yourself up to what (in my view) are its drawbacks. I try to say this in the “Antithesis of Moral Progress” section, but obviously did a terrible job haha :)
Maybe (just maybe) we’re getting somewhere here. I have no interest in adopting a ‘problem/knowledge focused ethic’. That would seem to presuppose the intrinsic value of knowledge. I only think knowledge is instrumentally valuable insofar as it promotes welfare.
Instead most EAs want to adopt an ethic that prioritises ‘maximising welfare over the long-run’. Longtermism claims that the best way to do so is to actually focus on long-term effects, which may or may not require a focus on near-term knowledge creation—whether it does or not is essentially an empirical question. If it doesn’t require it, then a strong longtermist shouldn’t consider a lack of knowledge creation to be a significant drawback.
I have a few comments on the critique of Bayesian epistemology, a lot of which I think is mistaken.
You say “It frames the search for knowledge in terms of beliefs (which we quantify with numbers, and must update in accordance with Bayes rule, else risk rationality-apostasy!” I don’t think anyone denies that Bayes theorem is true. It is mathematically proven. The most common criticism of Bayesianism is that it is “too subjective”. I don’t really understand what this means, but few sensible people deny Bayes theorem.
“It has imported valid statistical methods used in economics and computer science, and erroneously applied them to epistemology, the study of knowledge creation.” Economics and computer science are epistemic enterprises. If Bayesianism is the right approach in these fields, it will be difficult to show it is not the right approach in other domains, such as political science, forecasting, other questions that long-termists are interested in.
“It is based on confirmation as opposed to falsification”. Falsificationism is implausible as a philosophy of science. Despite his popularity among scientists who get given one philosophy of science class, Karl Popper was a scientific irrationalist who denied that scientific knowledge has increased over the last few hundred years - (on this, I would recommend David Stove’s Scientific Irrationalism). If you deny that observations confirm scientific theories, then you would have no reason to believe scientific theories which are supported by observational evidence, such as that smoking causes lung cancer.
“It leads to paradoxes”. Lots of smart philosophers deny that pascal’s mugging is a genuine paradox.
[redacted—sorry misread the quote]
“It relies on the provably false probabilistic induction”. Popper was a scientific irrationalist because he denied the rationality of induction. If you deny the rationality of induction, then you must be sceptical about all scientific theories that purport to be confirmed by observational evidence. Inductive sceptics must hold that if you jumped out of a tenth floor balcony, you would be just as likely to float upwards as fall downwards. Equally, do you think that smoking causes lung cancer? Do you think that scientific knowledge has increased over the last 200 years? If you do, then you’re not an inductive sceptic. Inductive scepticism can’t be used to ground a criticism that distinguishes uncertain long-termist probability estimates from probability estimates based on “hard data”. e.g. GiveWell’s estimates on the effectiveness of bednets are based on induction—they use data from studies showing that bednets have reduced the incidence of malaria
“(ironically, it’s precisely this aspect of Bayesianism which is so dubious: its inability to reject any hypothesis). ” This isn’t true. Bayesianism rejects some hypotheses. e.g. it assigns zero probability to some hypotheses, such as those that are logically or analytically false, like “smoking does and does not increase the risk of lung cancer”. It also assigns very low probability to some hypotheses that are not logically or analytically false but have little to no observational support, such as “smoking does not increase the risk of lung cancer”. If ‘reject’ means “assigns <0.001% probability to”, then Bayesianism obviously does reject some hypotheses.
Thanks for the engagement!
I think you’re mistaking Bayesian epistemology with Bayesian mathematics. Of course, no one denies Bayes’ theorem. The question is: to what should it be applied? Bayesian epistemology holds that rationality consists in updating your beliefs in accordance with Bayes’ theorem. As this LW post puts it:
Next, it’s not that “Bayesianism is the right approach in these fields,” (I’m not sure what that means) it’s that Bayesian methods are useful for some problems. But Bayesianism falls short when it comes to explaining how we actually create knowledge. (No amount of updating on evidence + Newtonian mechanics gives you relativity.)
Love the ad hominem attack.
Smoking causes lung cancer is a hypothesis, smoking does not cause lung cancer is another. We then discriminate between the hypotheses based on evidence (we falsify incorrect hypotheses). We slowly develop more and more sophisticated explanatory theories of how smoking causes lung cancer, always seeking to falsify them. At any time, we are left with the best explanation of a given phenomenon. This is how falsification works. (I can’t comment on your claim about Popper’s beliefs—but I would be surprised if true. His books are filled with examples of scientific progress.)
Yes. Theories are not confirmed by evidence (there’s no number of white swans you can see which confirms that all swans are white. “Swans are white” is a hypothesis, which can be refuted by seeing a black swan), they are falsified by it. Evidence plays the role of discrimination, not confirmation.
No—because we have explanatory theories telling us why we’ll fall downwards (general relativity). These theories are the only ones which have survived scrutiny, which is why we abide by them. Confirmationism, on the other hand, purports to explain phenomenon by appealing to previous evidence. “Why do we fall downwards? Because we fell downwards before”. The sun rising tomorrow morning does not confirm the hypothesis that the sun rises every day. We should not increase our confidence in the sun rising tomorrow because it rose yesterday. Instead, we have a theory about why and when the sun rises when it does (heliocentric model + axis-tilt theory).
Observing additional evidence in favour of the theory should not increase our “credence” in it. Finding confirming evidence of a theory is easy, as evidenced by astrology and ghost stories. The amount of confirmatory evidence for these theories is irrelevant, what matters is whether and by what they can be falsified. There are more accounts of people seeing UFOs than there are of people witnessing gamma ray bursts. According the confirmationism, we should thus increase our credence in the former, and have almost none in the existence of the latter.
If you haven’t read this piece on the failure of probabilistic induction to favour one generalization over another, I highly encourage you to do so.
Anyway, happy to continue this debate if you’d like, but that was my primer.
He said it has zero probability but is still useful, not nonzero probability.
I think you’re overinterpreting the claim (or Ben’s claim is misleading, based on what’s cited). You don’t have to give equal weight to all hypotheses. You might not even define their weights. The proof cited shows that the ratio of probabilities between two hypotheses doesn’t change in light of new evidence that would be implied by both theories. Some theories are ruled out or made less likely in light of incompatible evidence. Of course, there are always “contrived” theories that survive, but it’s further evidence in the future, Occam’s razor or priors that we use to rule them out.
This depends on your priors, which may be arbitrarily skeptical of causal effects.
Yes thanks my mistake—edited above
I agree that attention is a limited resource, but it feels like you’re imagining split attention leads to something like linear interpolation between focused attention on either end; in fact I think it’s much better than that, and that attention on the two parts are complementary. For example we need to wrestle with problems we face today to give us good enough feedback loops to make substantial progress, but by taking the long-term perspective we can improve our judgement about which of the nearer-term problems should be highest-priority.
I actually think that in the longtermist ideal world (where everyone is on board with longtermism) that over 90% of attention—perhaps over 99% -- would go to things that look like problems already. But that at the present margin in the actual world the longtermist perspective is underappreciated so looks particularly valuable.
I’m tempted to just concede this because we’re very close to agreement here.
If this turns out to be true (i.e., people end up working on actual problems and not, say, defunding the AMF to worry about “AI controlled police and armies”), then I have much less of a problem with longtermism. People can use whatever method they want to decide which problems they want to work on (I’ll leave the prioritization to 80K :) ).
Just apply my critique to the x% of attention that’s spent worrying about non-problems. (Admittedly, of course, this world is better than the one where 100% of attention is on non-existent possible future problems.)
I think this is might be a case of the-devil-is-in-the-details.
I’m in favour of people scanning the horizon for major problems whose negative impacts are not yet being felt, and letting that have some significant impact on which nearer-term problems they wrestle with. I think that a large proportion of things that longtermists are working on are problems that are at least partially or potentially within our foresight horizons. It sounds like maybe you think there is current work happening which is foreseeably of little value: if so I think it could be productive to debate the details of that.
Thanks for writing this! I think it’s important to question longtermism. I’ve actually found myself becoming slowly more convinced by it, but I’m still open to it being wrong. I’m looking forward to chewing on this a bit more (and you’ve reminded me I still have to properly read Vaden’s post) but for now I will leave you with a preliminary thought.
I don’t think this is fair. In their paper the authors say:
It seems possible to me that the claim that the future is in expectation vast could be refuted. The authors actually implicitly acknowledge in the bold text above that the claim would be refuted if one were to accept that we will very likely destroy ourselves, or that it is very likely that we will be destroyed and there is unlikely to be much we can do to reduce that risk.
So could the claim realistically be refuted? I think so. For example, one possible solution to the Fermi paradox is that there is a great filter that causes the vast majority of civilisations to cease to exist before colonising space. It seems possible to me that the great filter could emerge as the best answer to the Fermi paradox, in which case the size of the future may no longer be ‘vast in expectation’.
That is just one way in which the claim could be refuted and I suspect there are others. So I don’t think your unfalsifiable critique is justified, although I would be happy to hear a response to this.
Hi Jack,
I think you’re right, the comparison to astrology isn’t entirely fair. But sometimes one has to stretch a little bit to make a point. And the point, I think, is important. Namely, that these estimates can be manipulated and changed all too easily to fit a narrative. Why not half a quadrillion, or 10 quadrillion people in the future?
On the falsifiability point—I agree that the claims are technically falsifiable. I struggled with the language for this reason while writing it (and Max Heitmann helpfully tried to make this point before, but apparently I ignored him). In principle, all of their claims are falsifiable (if we go extinct, then sure, I guess we’ll know how big the future will be). Perhaps it’s better if I write “easily varied” or “amenable to drastic change” in place of irrefutable/unfalsifiable?
The great filter example is interesting, actually. For if we’re working in a Bayesian framework, then surely we’d assign such a hypothesis a probability. And then the number of future people could again be vast in expectation.
The fact that they can be manipulated and changed doesn’t strike me as much of a criticism. The more relevant question is if people actually do manipulate and change the estimates to fit their narrative. If they do we should call out these particular people, but even in this case I don’t think it would be an argument against longtermism generally, just against the particular arguments these ‘manipulaters’ would put forward.
The authors do at least set out their assumptions for the one quadrillion which they call their conservative estimate. For example, one input into the figure is an estimate that earth will likely be habitable for another 1 billion years, which is cited from another academic text. Now I’m not saying that their one quadrillion estimate is brilliantly thought through (I’m not saying it isn’t either), I’m just countering a claim I think you’re making that Greaves and MacAskill would likely add zeros or inflate this number if required to protect strong longtermism e.g. to maintain that their conservative longtermist EV calculation continues to beat GiveWell’s cost-effectiveness calculation for AMF. I don’t see evidence to suggest they would and I personally don’t think they would manipulate in such a way. That’s not to say that the one quadrillion figure may not change, but I would hope and would expect this to be for a better reason than “to save longtermism”.
To sum up I don’t think your “amenable to drastic change” point is particularly relevant. What I do think is more relevant is that the one quadrillion estimate is slightly arbitrary, and I see this as a subtly different point. I may address this in a different comment.
Yes if you’re happy to let your calculations be driven by very small probabilities of enormous value I suppose you’re right that the great filter would never be conclusive. Whether or not it is reasonable to allow this is an open question in decision theory and I don’t think it’s something that all longtermists accept.
The authors themselves don’t appear to be all that comfortable with accepting it:
This implies if they think a credence is miniscule or a long-lasting influence negligible that they might throw away the calculation.
Coming from an economics background, here’s how to persuade me of longtermism:
Set up a social planner problem with infinite generations and solve for the optimal allocation in each period. Do three cases:
A planner with nonzero time preference and perfect information
A (longtermist) planner with zero time preference and perfect information
A planner with zero time preference and imperfect information
Would the third planner ignore the utility of all generations less than 1000 years in the future? If so, then you’ve proved strong longtermism.
On the point about the arbitrariness of estimates of the size of the future—what is your probability distribution across the size of the future population, provided there is not an existential catastrophe?
I think you should specify a time period (e.g. the next 100 years) or feasibly preventable existential catastrophes. Could the heat death of the universe be an existential catastrophe? If so, I think the future population might be infinite, since anything less might be considered an existential catastrophe.
I’m not the author of this post, but I don’t have only one probability distribution for this, and I don’t think there’s any good way to justify any particular one (although you might rule some out for being less reasonable).
I don’t think the question makes sense. I agree with Vaden’s argument that there’s no well-defined measure over all possible futures.
There are definitely well-defined measures on any set (e.g. pick one atomic outcome to have probability 1 and the rest 0); there’s just not only one, and picking exactly one would be arbitrary. But the same is true for any set of outcomes with at least two outcomes, including finite ones (or it’s at least often arbitrary when there’s not enough symmetry for equiprobability).
For the question of how many people will exist in the future, you could use a Poisson distribution. That’s well-defined, whether or not it’s a reasonable distribution to use.
Of course, trying to make your space more and more specific will run into feasibility issues.
There are non-measurable sets (unless you discard the axiom of choice, but then you’ll run into some significant problems.) Indeed, the existence of non-measurable sets is the reason for so much of the measure-theoretic formalism.
If you’re not taking a measure theoretic approach, and instead using propositions (which I guess, it should be assumed that you are, because this approach grounds Bayesianism), then using infinite sets (which clearly one would have to do if reasoning about all possible futures) leads to paradoxes. As E.T. Jaynes writes in Probability Theory and the Logic of Science:
(Vaden makes this point in the podcast.)
This depends on the space.
It’s at least true for real-valued intervals with continuous measures, of course, but I think you’re never going to ask for the measure of a non-measurable set in real-world applications, precisely because they require the axiom of choice to construct (at least for the real numbers, and I’d assume, by extension, any subset of any Rn), and no natural set you’ll be interested in that comes up in an application will require the axiom of choice (more than dependant choice) to construct. I don’t think the existence of non-measurable sets is viewed as a serious issue for applications.
It is not true in a countable measure space (or, at least, you could always extend the measure to get this to hold), since assuming each singleton (like {x},x∈X) is measurable, every union of countably many singletons is measurable, and hence every subset is measurable (A=∪x∈A{x} is a countable union of singletons, A⊆X, X countable) . In particular, if you’re just interested in the number of future people, assuming there are at most countably infinitely many (so setting aside the many-worlds interpretation of quantum mechanics for now), then your space is just the set of non-negative integers, which is countable.
You could group outcomes to represent them with finite sets. Bayesians get to choose the measure spaces/propositions they’re interested in. But again, I don’t think dealing with infinite sets is so bad in applications.
Do you for example think there is a more than 50% chance that it is greater than 10 billion?
Another way to look at this. What do you think is the probability that everyone will go extinct tomorrow? If you are agnostic about that, then you must also be agnostic about the value of GiveWell-type stuff.
Why? GiveWell charities have developed theories about the effects of various interventions. The theories have been tested and, typically, found to be relatively robust. Of course, there is always more to know, and always ways we could improve the theories.
I don’t see how this relates to not being able to develop a statistical estimate of the probability we go extinct tomorrow. (Of course, I can just give you a number and call it “my belief that we’ll go extinct tomorrow,” but this doesn’t get us anywhere. The question is whether it’s accurate—and what accuracy means in this case.) What would be the parameters of such a model? There are uncountably many things—most of them unknowable—which could affect such an outcome.
The benefits of GiveWell’s charities are worked out as health or economic benefits which are realised in the future. e.g. AMF is meant to be good because it allows people who would have otherwise died to live for a few more years. If you are agnostic about whether everyone will go extinct tomorrow, then you must be agnostic about whether people will actually get these extra years of life.
I don’t have a probability distribution across the size of the future population. That said, I’m happy to interpret the question in the colloquial, non-formal sense, and just take >50% to mean “likely”. In that case, sure, I think it’s likely that the population will exceed 10 billion. Credences shouldn’t be taken any more seriously than that—epistemologically equivalent to survey questions where the respondent is asked to tick a very unlikely, unlikely, unsure, likely, very likely box.
.
Agree! While I do have problems with (weak?) longtermism, this post is a criticism of strong longtermism :)
I found it helpful that you were so clear about these two aspects of what you are saying. My responses to the two are different.
On the first, I think resting on possibilities of large futures is a central part of the strength of the case for longtermism. It doesn’t feel like inoculation from criticism to put the strong argument forwards. Of course this only applies to the argument for longtermism in the abstract and not for particular actions people might want to take; I think that using such reasoning in favour of particular actions tends to be weak (inoculation is sometimes attempted but it is ineffectual).
On the second, I think this might be an important and strong critique, but it is a critique of how the idea is presented and understood rather than of the core tenets of longtermism; indeed one could make the same arguments starting from an assumption that longtermism was certainly correct, but being worried that it would be self-defeating.
So I’m hearing the second critique (perhaps also the first but it’s less clear) as saying that the “blueprints” (in the sense of https://forum.effectivealtruism.org/posts/NdSoipXQhdzozLqW4/blueprints-and-lenses-for-longtermist-decision-making ) people commonly get for longtermism are bad (on both shorttermist and longtermist grounds). Does that sound mostly-correct to you?
Hi Owen!
Re: inoculation of criticism. Agreed that it doesn’t make criticism impossible in every sense (otherwise my post wouldn’t exist). But if one reasons with numbers only (i.e., EV reasoning), then longtermism becomes unavoidable. As soon as one adopts what I’m calling “Bayesian epistemology”, then there’s very little room to argue with it. One can retort: Well, yes, but there’s very little room to argue with General Relativity, and that is a strength of the theory, not a weakness. But the difference is that GR is very precise: It’s hard to argue with because it aligns so well with observation. But there are lots of observations which would refute it (if light didn’t bend around stars, say). Longtermism is difficult to refute for a different reason, namely because it’s so easy to change the underlying assumptions. (I’m not trying to equate moral theories with empirical theories in every sense, but this example gets the point across I think.)
Your second point does seem correct to me. I think I try to capture this sentiment when I say
Here I’m granting that the moral view that future generations matter could be correct. But this, on my problem/knowledge-focused view of progress, is irrelevant for decision making. What matters is maintaining the ability to solve problems and correct our (inevitable) errors.
Cool. I do think that trying to translate your position into the ontology used by Greaves+MacAskill it’s sounding less like “longtermism is wrong” and more like “maybe longtermism is technically correct; who cares?; the practical advice people are hearing sucks”.
I think that’s a pretty interestingly different objection and if it’s what you actually want to say it could be important to make sure that people don’t hear it as “longtermism is wrong” (because that could lead them to looking at the wrong type of thing to try to refute you).
I think that ontology used by Greaves+MacAskill is poor. I skim-read their Case for Strong Longtermsim paper honestly expecting it to be great (Will is generally pretty sensible) but I came away quite confused as to what case was being made.
Ben – maybe there needs to be more of an exercise to disentangle what is meant by longtermism before it can be critiqued fairly.
Owen – I am not sure if you would agree but I as far as I can tell the points you make about bounded rationality in the excellent post you link to above contradicts the the Case for Strong Longtermsim paper. EG:
Greaves+MacAskill: “we assumed that the correct way to evaluate options … is in terms of expected value” (as far as I can tell their entire point is that you can always do an expected value calculation and “ignore all the effects contained in the first 100″ years).
You: “if we want to make decisions on longtermist grounds, we are going to end up using some heuristics”
Yes, exactly. One can always find some expected value calculation that allows one to ignore present-day suffering. And worse, one can keep doing this between now and eternity, to ignore all suffering forever. We can describe this using the language of “falsifiability” or “irrefutability” or whatever—the word choice doesn’t really matter here. What matters is that this is a very dangerous game to be playing.
I think it is worth trying to judge the paper / case for longtermism charitably. I do not honestly think that Will means that we can literally ignore everything in the first 100 years – for a start just because the short-term affects the long-term. If you want to evaluate interventions, even those designed for long-term impact, you need to look at the short-term impacts.
But that is where I get stuck trying to work out what Will + Hillary mean. I think they are saying more than just you should look at the long and short term effects of interventions (trivially true under most ethical views).
They seem to be making empirical, not philosophical, claims about the current state of the world.
They appear to argue that if you use expected value calculations for decision making then you will arrive at the conclusions that suggest that you should care about highly speculative long-term effects over clear short term effects. They combine this with an assumption that expected value calculations are the correct decision making tool to conclude that long-term interventions are most likely to be the best interventions.
I think
the logic of the argument is roughly correct.
the empirical claims made are dubious and ideally need more than a few examples to justify, but it is plausible they are correct. I think there is at least a decent case for marginal extra resources being directed to x-risk prevention in the world today.
the assumption that expected value calculations are the correct decision making tool is incorrect, (as per others at GPI like Owen’s work and Andreas’ work, bounded rationality, the entire field of risk management, economists like Taleb, knightian uncertainty, etc. etc) . A charitable reading would say that they recognises this is an assumption but chooses not to address it.
Hmmm… I now feel I have a slightly better grasp of what the arguments are after having written that. (Ben I think this counts as disentangling some of the claims made and more such work could be useful)
Vadmas – I think there can be grounds for refusing to follow arguments that you cannot disprove based solely on the implausibility or repugnance of their conclusions, which appears to be your response to their paper. I am not sure it is needed as I don’t think think the case for strong longtermism is well made.
I’d say they mean you can effectively ignore the differences in terminal value in the short term, e.g. the welfare of individuals in the short term only really matters for informing long-term consequences and effectively not in itself, since it’s insignificant compared to differences in long-term value.
In other words, short-term welfare is effectively not an end in itself.
Yeah that is a good way of putting it. Thank you.
It is of course a feature of trying to prioritise between causes in order to do the most good, that some groups will be effectively ignored.
Luckily in this case if done in a sensible manner I would expect that there should be a strong correlation between short term welfare and long-run welfare. As managing high uncertainty should involve some amount of ensuring good feedback loops and iterating, so taking action changing things for the better (for the long run but in a way that affects the world now) learning and improving. Building the EA community, developing clean meat, improving policy making, etc.
(Unfortunately I am not sure to what extent this is a key part of the EA longtermist paradigm at present.)
Hmm perhaps you need to read the paper again. They say for example:
Indeed they go on in section 4.5 to consider other decision theories, including Knightian uncertainty, and conclude that strong longtermism is robust to these other theories. I’m not saying they’re definitely right, just that they haven’t assumed expected value theory is correct as you claim.
OK Jack, I have some time today so lets dive in:
So, my initial reading of 4.5 was that they get it very very wrong.
Eg: “we assumed that the correct way to evaluate options in ex ante axiological terms, under conditions of uncertainty, is in terms of expected value”. Any of the points above would disagree with this.
Eg: “[Knightian uncertainty] supports, rather than undermining, axiological strong longtermism”. This is just not true. Some Knightian uncertainty methods would support (eg robust decision making) and some would not support (eg plan-and-adapt).
So why does it look like they get this so wrong?
Maybe they are trying to achieve something different from what we in this thread think they are trying to achieve.
My analysis of their analysis of Knightian uncertainty can shed some light here.
The point of Knightian (or deep) uncertainty tools is that an expected value calculation is the wrong tool for humans to use when making decisions under Knightian uncertainty. That an expected value calculation, as a decision tool it will not lead to the best outcome, the outcome with the highest true expected value. [Note: I use true expected value to mean the expected value if there was no uncertainty, which can be different from the calculated expected value.] The aim is still the same (to maximise true expected value) but the approach is different. Why the different approach – because in practice expected value calculations do not work well – they lead to anchoring, lead to unknown unknows being ignored, are super sensitive to speculation, etc, etc. The tools used are varied but include tactics such as encouraging decision makers to aim for an option that is satisficing (least bad) on a variety of domains rather than maximising (this specific tool is to minimise the risk of unknown unknows being ignored).
But when Will+Hillary explain Knightian uncertainty they explain it as if it is posing a fundamental axiological difference. As if aiming for the least bad option is done because the least bad option is true best option (as opposed to the calculated best option, if that makes sense). This is not at all what anyone I know who uses these tools believes.
Let’s pause and note that as Knightian uncertainty tools are still aiming at guiding actors towards the true highest expected value they could theoretically be explained in terms of expected value. They don’t challenge the expected value axiology
Clearly Will+Hillary are not, in this paper, interested in if it poses an alternative methodology to reaching the true expected value, they are only interested in if it could be used to justify a different axiology. This would explain why this paper ignore all the other tools (like predict-then-act tools) focuses on this one tool and explains it in a strange way.
The case they are making (by my charitable reading) is that if we are aiming for true expected value then, because the future is so so so so so big that we should expect to be able to find at least some options that influence it and the thing that does the most good is likely to be among those options.
They chose expected value calculations as a way to illustrate this.
As Owen says here, they are “talking about how an ideal rational actor should behave – which I think is informative but not something to be directly emulate”.
They do not seem to be aiming to say anything on how to make decisions about what to focus on.
So I stand by my claim that the most charitable reading is that they are deliberately not addressing how to make decisions.
--
As far as I can tell, in layman speak, this paper tries to make the case that: If we had perfect information [edit: on expected value] the options that would be best to do would be those that positively affect the far future. So in practice looking towards those kinds of options is a useful tool to apply when we are deciding what to do.
FWIW I expect this paper is largely correct (if the conclusion is as above). However I think could be improved in some ways:
It is opaque. Maybe it is clearer to fellow philosophers but I reached my view of what the paper was trying to achieve by looking at how they manage to mis-explain a core decision making concept two-thirds of they way through and then extrapolated the ways they could be rationally making their apparent errors. Not easy to understand what they are doing. And I think most people on this thread would have a different view to me about this paper. Would be good if a bit more text for us layfolk.
It could be misconstrued. Work like this leads people to think that Will+Hillary and others believe that expected value calculations are the key tool for decision making. They are not. (I am assuming they only reference expected value calculations for illustrative purposes, if I am incorrect then their paper is either poor or I really don’t get it.)
It leaves unanswered questions, but does not make it clear what those questions are. I do think it is useful to know that we should expect the most high impact actions to be those that have long run positive consequences. But how the hell should anyone actually make a decision and compare short term and long term? This paper does not help on this. It could maybe highlight the need to research this.
Is is a weak argument. It is plausible to me that alternative decision making tools might confuse their conclusions so much that when applied in practice by a philanthropist etc the result largely does not apply.
For example one could believe that economic growth is good for the future, that most people who try to impact the world positively without RCT-level evidence fail, that situations of high uncertainty are best resolved though engineering short feedback loops and quite rationally conclude that AMF (bednets) is the currently charity that has the biggest positive long-run affect on the future. I don’t think this contradicts anything in the paper and I don’t think it would be unreasonable.
There are other flaws with the paper too in the more empirical part with all the examples. Eg even a very very low discount rate to account for things like extinction risk or sudden windfall really quickly reduces the amount the future matters. (Note this is different from pure time preference discounting).
In my view they overstate (or are misleading about) what they have achieved. Eg I do not think, for the reasons given, that they have at all shown that “plausible deviations from [an expected utility treatment of decision-making under uncertainty] do not undermine the core argument”. (This is only true insofar as decision-making approaches are, as far as I can tell, not at all relevant to their core argument). They have maybe shown something like: “plausible deviations from expected utility theory do not undermine the core argument”.
Let me know what you think.
Catch ya about :-)
@ Ben_Chugg
Curious how much you would agree with a statement like:
If we had perfect information [edit: on expected value] the options that would be best to do would be those that positively affect the far future. So in practice looking towards those kinds of options is a useful tool to apply when we are deciding what to do.
(This is my very charitable, weak interpretation of what the Case for Strong Longtermism paper is attempting to argue)
I think I agree, but there’s a lot smuggled into the phrase “perfect information on expected value”. So much in fact that I’m not sure I can quite follow the thought experiment.
When I think of “perfect information on expected value”, my first thought is something like a game of roulette. There’s no uncertainty (about what can affect the system), only chance. We understand all the parameters of the system and can write down a model. To say something like this about the future means we would be basically omniscient—we would know what sort of future knowledge will be developed, etc. Is this also what you had in mind?
(To complicate matters, the roulette analogy is imperfect. For a typical game of roulette we can write down a pretty robust probabilistic model. But it’s only a model. We could also study the precise physics of that particular roulette board, model the hand spinning the wheel (is that how roulette works? I don’t even know), take into account the initial position, the toss of the white ball, and so on and so forth. If we spent a long time doing this, we could come up with a model which was more accurate than our basic probabilistic model. This is all to say that models are tools suited for a particular purpose. So it’s unclear to me what the model would be for the future which allowed us to write down a precise model—which is implicitly required for EV calculations).
Hi Ben. I agree with you. Yes I think roulette is a good analogy. And yes I think the “perfect information on expected value” is a strange claim to make.
But I do think it is useful to think about what could be said and justified. I do think a claim along these lines could be made and it would not be wholly unfalsifiable and it would not require completely preferencing Bayesian expected value calculations.
To give another analogy I think there is a reasonable long-termist equivalent of statements like:
Because of differences in wealth and purchasing power we expect that a donor in the developed west can have a much bigger impact overseas than in their home country. So in practice looking towards those kinds of international development options is a useful tool to apply when we are deciding what to do.
This does not completely exclude the probability that we can have impact locally with donations, but it does direct our searching.
Being charitable to Will+Hillary, maybe that is all they are saying. And maybe it is so confusing because they have dressed it up in philosophical language – but this is because, as per GPI’s goals, this paper is about engaging philosophy academics rather than producing any novel insight.
(If being more critical I am not convinced that Will+Hillary successfully give sufficient evidence to make such a claim in this paper and also see my list of things their paper could improve above.)
Thanks for this! All interesting and I will have to think about this more carefully when my brain is fresher. I admit I’m not very familiar with the literature on Knightian uncertainty and it would probably help if I read some more about that first.
OK if I understand you correctly, what you have said is that Will and Hilary present Knightian uncertainty as axiologically different to EV reasoning, when you don’t think it is. I agree with you that ideally section 4.5 should be considering some axiologically different decision-making theories to EV.
Regarding the actual EV calculations with numbers, I would say, as I did in a different comment, that I think it is pretty clear that they only carry out EV calculations for illustrative purposes. To quote:
This is the point they are trying to get across by doing the actual EV calculations.
I agree that there’s a tension in how we’re talking about it. I think that Greaves+MacAskill are talking about how an ideal rational actor should behave—which I think is informative but not something to be directly emulated for boundedly rational actors.
Ah yes thank you Owen. That helps me construct a sensible positive charitable reading of their paper.
There is of course a risk that people take their paper / views of longtermism and expected value approach to be more decision guiding than perhaps they ought.
(I think it might be an overly charitable reading – the paper does briefly mention and then dismiss concerns about decision making under uncertainty, etc – although it is only a draft so reasonable to be charitable.)
Oh interesting. Did you read my critique as saying that the philosophy is wrong? (Not sarcastic; serious question.) I don’t really even know what “wrong” would mean here, honestly. I think the reasoning is flawed and if taken seriously leads to bad consequences.
I read your second critique as implicitly saying “there must be a mistake in the argument”, whereas I’d have preferred it to say “the things that might be thought to follow from this argument are wrong (which could mean a mistake in the argument that’s been laid out, or in how its consequences are being interpreted)”.
I’m not sold on the cluelessness-type critique of long-termism. The arguments here focus on things we might do now or soon to reduce the direct risk posed by various things such as AI, bio or nuclear war. But even if this is true, this doesn’t undermine the expected value of other long-termist activities.
Gathering more information about the direct risks. If we are clueless about what to do, the value of information from further research must be extremely high, on long-termism.
Building the community of people concerned about the long-term e.g. through community building.
Investing in the stock market and punting the “what to do” question to the future.
I wonder if you have come across the literature on complex cluelessness? GiveWell may use some real, tangible data, but they are missing lots of highly-relevant and important data, most obviously relating to the longer-term consequences of the health interventions. For example they don’t know what the long-term population effects will be nor the corresponding moral value of these population effects. It also really doesn’t seem fair to me to just assume that this would be zero in expectation, which GiveWell implicitly does. It seems highly plausible in fact that these longer-term effects could swamp the near-term effects.
I personally still have to think through cluelessness more to decide what conclusions to draw from it (as does the rest of the EA movement as I don’t think everyone has caught on to just how important this problem is!). As it stands however it has caused me to move away somewhat from cost-benefit analyses that makes use of ‘real, tangible data’ and towards arguments that are supposedly ‘more robust’ to a range of different assumptions and inputs which, funnily enough, I think may lead to certain longtermist interventions.
I appreciate that this is a starkly different view to you and I would be happy to hear your thoughts here!
I have read about (complex) cluelessness. I have a lot of respect for Hilary Greaves, but I don’t think cluelessness is particularly illuminating concept. I view it as a variant of “we can’t predict the future.” So, naturally, if you ground your ethics in expected value calculations over the long term future then, well, there’s going to be problems.
I would propose to resolve cluelessness as follows: Let’s admit we can’t predict the future. Our focus should instead be on error-correction. Our actions will have consequences—both intended and unintended, good and bad. The best we can do is foster a critical, rational environment where we can discuss the negatives consequences, solve them, and repeat. (I know this answer will sound glib, but I’m quite sincere.)
I do think it’s far more illuminating than “we can’t predict the future”.
Really complex cluelessness is saying OK great you’ve carried out a CBA/CEA but you’ve omitted/ignored effects from the analysis that we:
Have good reason to expect will occur
Have good reason to suspect are sufficiently important such that they could change the sign of your final number if properly included in your analysis
If the above factors are in fact true in the case of GiveWell (I think they probably are) then I don’t think GiveWell CBAs are all that useful and the original point you were trying to make—that GiveWell analysis is obviously superior because it makes use of data—sort of breaks down because, quite simply, the data has a massive, gaping hole in it. This is not to criticise GiveWell in the slightest, it’s just to acknowledge the monstrous task they’re up against.
Correct me if I’m wrong but what you seem to be arguing is that we’re actually complexly clueless about everything, so we may as well just ignore the problem. I actually don’t think this is true—we may be clueless about everything but not necessarily in a complex way. Consider the promotion of philosophy in schools, a class of interventions that I have written about. I’m not sure if these are definitely the best interventions (reception to my post was fairly lukewarm), but I also don’t think we are complexly clueless about their effects in the same way that we are about the effects of distributing bednets. This is because it’s just quite hard to think up reasons why it might be bad to promote philosophy in schools. Sure it could be the case that promoting philosophy in schools makes something bad happen, but I don’t really have much of a reason to entertain that possibility if I can’t think of a specific effect that fulfils the two factors I listed above. In the case of distributing bednets we are pretty certain there will be population effects, we are pretty certain these will be very significant in moral terms, but we don’t have much of a clue about the magnitude or even sign of this moral value. Therefore I would say we are complexly clueless about distributing bednets but only simply clueless about promoting philosophy in schools—only complex cluelessness is really a problem according to Greaves.
To clarify I actually think there will be short-termist interventions that don’t run into the problem of complex cluelessness (to give just one example—saving an animal from a factory farm), so I’m not attempting to prove longtermism here, I’m only countering your claim that using data in a CBA/CEA is necessarily better than engaging in an alternative method of analysis.
One response might be that if there are unintended negative consequences, we can address those later or separately. Sometimes it will be the case that optimizing for some positive effect optimizes a negative effect, but usually these won’t correspond. So, the most cost-effective ways to save lives won’t be the ways that maximize the negative effects of population growth—those same negative effects will be cheaper to obtain through something other than population growth -, and we can probably find more cost-effective ways to offset those effects. I wrote a post about hedging like this.
Interesting, thanks for sharing that post. I will have to read it more carefully to fully digest it!
What do you think about using ranges of probabilities instead of single (and seemingly arbitrary) sharp probabilities and doing sensitivity analysis? I suppose when there’s no hard data, there might be no good bounds for the ranges, too, although Scott Alexander has argued against using arbitrarily small probabilities.
Yeah I suppose I would still be skeptical of using ranges in the absence of data (you could just apply all my objections to the upper and low bounds of the range). But I’m definitely all for sensitivity analysis when there are data backing up the estimates!