That there are particular arguments for decisions like bednets or eating sandwiches to have expected impacts that scale with the scope of the universes or galactic civilizations. E.g. the more stars you think civilization will be able to colonize, or the more computation that will be harvested, the greater your estimate of the number of sims in situations like ours (who will act the same as we do, so that on plausible decision theories we should think of ourselves as setting policy at least for the psychologically identical ones). So if you update to think that civilization will be able to generate 10^40 minds per star instead of 10^30, that shouldn’t change the ratio of your EV estimates for x-risk reduction and bednets, since the number appears on both sides of your equations. Here’s a link to another essay making related points.
CarlShulman
This sort of estimate is in general off by many orders of magnitude for thinking about the ratio of impact between different interventions when it only considers paths to very large numbers for the intervention under consideration, and not to reference interventions being compared against. For example, the expected number of lives saved from giving a bednet is infinite. Connecting to size-of-the-accessible-universe estimates, perhaps there are many simulations of situations like ours at an astronomical scale, and so our decisions will be replicated and have effects on astronomical scales.
Any argument purporting to show <20 OOM in cost-effectiveness from astronomical waste considerations is almost always wrong for this kind of reason.
AGI and Lock-In
The implicit utility function in Kelly (log of bankroll) amounts to rejecting additive aggregation/utilitarianism. That would be saying that doubling goodness from 100 to 200 would be of the same decision value as doubling from 100 billion to 200 billion, even though in the latter case the benefit conferred is a billion times greater.
It also absurdly says that loss goes to infinity as you go to zero. So it will reject any finite benefit of any kind to prevent even an infinitesimal chance of going to zero. If you say that the world ending has infinite disutility then of course you won’t press a button with any chance of the end of the world, but you’ll also sacrifice everything else to increment that probability downward, e.g. taking away almost everything good about the world for the last tiny slice of probability.
This is much more of a problem (and an overwhelming one) for risks/opportunities that are microscopic compared to others. Baseline asteroid/comet risk is more like 1 in a billion. Much less opportunity for that with 1% or 10% risks.
They’re wildly quantitatively off. Straight 40% returns are way beyond equities, let alone the risk-free rate. And it’s inconsistent with all sorts of normal planning, e.g. it would be against any savings in available investments, much concern for long-term health, building a house, not borrowing everything you could on credit cards, etc.
Similarly the risk aversion for rejecting a 15% of $1M for $1000 would require a bizarre situation (like if you needed just $500 more to avoid short term death), and would prevent dealing with normal uncertainty integral to life, like going on dates with new people, trying to sell products to multiple customers with occasional big hits, etc.
Hi Brian,
I agree that preferences at different times and different subsystems can conflict. In particular, high discounting of the future can lead to forgoing a ton of positive reward or accepting lots of negative reward in the future in exchange for some short-term change. This is one reason to pay extra attention to cases of near-simultaneous comparisons, or at least to look at different arrangements of temporal ordering. But still the tradeoffs people make for themselves with a lot of experience under good conditions look better than what they tend to impose on others casually. [Also we can better trust people’s self-benevolence than their benevolence towards others, e.g. factory farming as you mention.]
And the brain machinery for processing stimuli into decisions and preferences does seem very relevant to me at least, since that’s a primary source of intuitive assessments of these psychological states as having value, and for comparisons where we can make them. Strong rejection of interpersonal comparisons is also used to argue that relieving one or more pains can’t compensate for losses to another individual.
I agree the hardest cases for making any kind of interpersonal comparison will be for minds with different architectural setups and conflicting univocal viewpoints, e.g. 2 minds with equally passionate complete enthusiasm (with no contrary psychological processes or internal currencies to provide reference points) respectively for and against their own experience, or gratitude and anger for their birth (past or future). They can respectively consider a world with and without their existences completely unbearable and beyond compensation. But if we’re in the business of helping others for their own sakes rather than ours, I don’t see the case for excluding either one’s concern from our moral circle.
Now, one can take take a more nihilistic/personal aesthetics view of morality, and say that one doesn’t personally care about the gratitude of minds happy to exist. I take it this is more your meta-ethical stance around these things? There are good arguments for moral irrealism and nihilism, but it seems to me that going too far down this route can lose a lot of the point of the altruistic project. If it’s not mainly about others and their perspectives, why care so much about shaping (some of) their lives and attending to (some of) their concerns?
David Pearce sometimes uses the Holocaust to argue for negative utilitarianism, to say that no amount of good could offset the pain people suffered there. But this view dismisses (or accidentally valorizes) most of the evil of the Holocaust. The death camps centrally were destroying lives and attempting to destroy future generations of peoples, and the people inside them wanted to live free, and being killed sooner was not a close substitute. Killing them (or willfully letting them die when it would be easy to prevent) if they would otherwise escape with a delay would not be helping them for their own sakes, but choosing to be their enemy by only selectively attending to their concerns. And even though some did choose death. Likewise to genocide by sterilization (in my Jewish household growing up the Holocaust was cited as a reason to have children).
Future generations, whether they enthusiastically endorse or oppose their existence, don’t have an immediate voice (or conventional power) here and now their existence isn’t counterfactually robust. But when I’m in a mindset of trying to do impartial good I don’t see the appeal of ignoring those who would desperately, passionately want to exist, and their gratitude in worlds where they do.
I see demandingness and contractarian/game theory/cooperation reasons that bound sacrifice to realize impartial uncompensated help to others, and inevitable moral dilemmas (almost all beings that could exist in a particular location won’t, wild animals are desperately poor and might on average wish they didn’t exist, people have conflicting desires, advanced civilizations I expect will have far more profoundly self-endorsing good lives than unbearably bad lives but on average across the cosmos will have many of the latter by sheer scope). But being an enemy of all the countless beings that would like to exist, or do exist and would like to exist more (or more of something), even if they’re the vast supermajority, seems at odds to me with my idea of impartial benevolence, which I would identify more with trying to be a friend to all, or at least as much as you can given conflicts.
In the surveys they know it’s all hypothetical.
You do see a bunch of crazy financial behavior in the world, but it decreases as people get more experience individually and especially socially (and with better cognitive understanding).
People do engage in rounding to zero in a lot of cases, but with lots of experience will also take on pain and injury with high cumulative or instantaneous probability (e.g. electric shocks to get rewards, labor pains, war, jobs that involve daily frequencies of choking fumes or injury.
Re lexical views that still make probabilistic tradeoffs, I don’t really see the appeal of contorting lexical views that will still be crazy with respect to real world cases so that one can say they assign infinitesimal value to good things in impossible hypotheticals (but effectively 0 in real life). Real world cases like labor pain and risking severe injury doing stuff aren’t about infinitesimal value too small for us to even perceive, but macroscopic value that we are motivated by. Is there a parameterization you would suggest as plausible and addressing that?
I’d use reasoning like this, so simulation concerns don’t have to be ~certain to drastically reduce EV gaps between local and future oriented actions.
Here’s a good piece.
Today there is room for an intelligence explosion and explosive reproduction of AGI/robots (the Solar System can support trillions of AGIs for every human alive today). If aligned AGI undergoes such intelligence explosion and reproduction there is no longer free energy for rogue AGI to grow explosively. A single rogue AGI introduced to such a society would be vastly outnumbered and would lack special advantages, while superabundant AGI law enforcement would be well positioned to detect or prevent such an introduction in any case.
Already today states have reasonably strong monopolies on the use of force. If all military equipment (and an AI/robotic infrastructure that supports it and most of the economy) is trustworthy (e.g. can be relied on not to engage in military coup, to comply with and enforce international treaties via AIs verified by all states, etc) then there could be trillions of aligned AGIs per human, plenty to block violent crime or WMD terrorism.
For war between states, that’s point #7. States can make binding treaties to renounce WMD war or protect human rights or the like, enforced by AGI systems jointly constructed/inspected by the parties.
Barring simulation shutdown sorts of things or divine intervention I think more like 1 in 1 million per century, on the order of magnitude of encounters with alien civilizations. Simulation shutdown is a hole in the argument that we could attain such a state, and I think a good reason not to say things like ‘the future is in expectation 50 orders of magnitude more important than the present.’
It’s quite likely the extinction/existential catastrophe rate approaches zero within a few centuries if civilization survives, because:
Riches and technology make us comprehensively immune to natural disasters.
Cheap ubiquitous detection, barriers, and sterilization make civilization immune to biothreats
Advanced tech makes neutral parties immune to the effects of nuclear winter.
Local cheap production makes for small supply chains that can regrow from disruption as industry becomes more like information goods.
Space colonization creates robustness against local disruption.
Aligned AI blocks threats from misaligned AI (and many other things).
Advanced technology enables stable policies (e.g. the same AI police systems enforce treaties banning WMD war for billions of years), and the world is likely to wind up in some stable situation (bouncing around until it does).
If we’re more than 50% likely to get to that kind of robust state, which I think is true, and I believe Toby does as well, then the life expectancy of civilization is very long, almost as long on a log scale as with 100%.
Your argument depends on 99%+++ credence that such safe stable states won’t be attained, which is doubtful for 50% credence, and quite implausible at that level. A classic paper by the climate economist Martin Weitzman shows that the average discount rate over long periods is set by the lowest plausible rate (as the possibilities of high rates drop out after a short period and you get a constant factor penalty for the probability of low discount rates, not exponential decay).
- Charting the precipice: The time of perils and prioritizing x-risk by (Oct 24, 2023, 4:25 PM; 86 points)
- Future Matters #5: supervolcanoes, AI takeover, and What We Owe the Future by (Sep 14, 2022, 1:02 PM; 31 points)
- 's comment on Existential risk pessimism and the time of perils by (Sep 5, 2022, 8:55 AM; 19 points)
- 's comment on How bad would human extinction be? by (Oct 24, 2023, 5:16 AM; 15 points)
- 's comment on Vague justifications for longtermist assumptions? by (May 11, 2024, 10:25 PM; 5 points)
- 's comment on The discount rate is not zero by (Sep 23, 2022, 10:44 PM; 3 points)
An example of an argument matching the OP’s thesis: Bryan Caplan rejecting animal rights (disagreeing with his favorite philosopher, Michael Huemer) based on the demands of applying a right to life to wild insects.
Toby, Carl and Brian meet the next day, still looking very pale. They shake hands and agree to not do so much descriptive ethics anymore.
Garbage answers to verbal elicitations on such questions (and real life decisions that require such explicit reasoning without feedback/experience, like retirement savings) are actually quite central to my views. In particular, my reliance on situations where it is easier for individuals to experience things multiple times in easy-to-process fashion and then form a behavioral response. I would be much less sanguine about error theories regarding such utterances if we didn’t also see people in surveys saying they would rather take $1000 than a 15% chance of $1M, or $100 now rather than $140 a year later, i.e. utterances that are clearly mistakes.Looking at the literature on antiaggregationist views, and the complete conflict of those moral intuitions with personal choices and self-concerned practice (e.g. driving cars or walking outside) is also important to my thinking. No-tradeoffs views are much more appealing outside our own domains of rich experience in talk.
Imagine that only a few hundred people in the world thought that climate change is an important problem (rather than at least tens of millions), that philanthropists worldwide spent a few million dollars a year on climate (rather than $10 billion), that society as a whole spent a million dollars on the problem (rather than $1 trillion), and that the international institutions trying to tackle the problem either don’t exist or have a similar budget to a McDonald’s restaurant. How bad would climate change be? This is how bad things are for the other global catastrophic risks, and then some.
This seems to overstate how bad the situation is (although qualitatively it remains an absurd underinvestment, with painfully low-hanging fruit to avert pandemics and AI catastrophe at hand). Surveys of the general public and area experts do show substantial percentages in the abstract endorse nuclear (in particular), bioweapon, and AI risks as important problems (as you mention later). Governments wage wars and spend very large amounts of attention and resources on nuclear proliferation and threats. Biodefense has seen billions of dollars of spending, even if it was not well-crafted to reduce catastrophic bioweapon risk. The low budget for the BWC is in significant part a political coordination problem and not simply a $ supply issue. Annual spending from Open Philanthropy and the Future Fund on catastrophic risks, with priorities close to yours, is now in the hundreds of millions of dollars.
People generally don’t care about their future QALYs in a linear way: a 1/million chance of living 10 million times as long and otherwise dying immediately is very unappealing to most people, and so forth. If you don’t evaluate future QALYs for current people in a way they find acceptable, then you’ll wind up generating recommendations that are contrary to their preferences and which will not be accepted by society at large.
This sort of argument shows that person-affecting utilitarianism is a very wacky doctrine (also see this) that doesn’t actually sweep away issues of the importance of the future as some say, but it doesn’t override normal people concerns by their own lights.
Oh, one more thing: AI timelines put a discount on other interventions. Developing a technology that will take 30 years to have its effect is less than half as important if your median AGI timeline is 20 years.
The funding scale of AI labs/research, AI chip production, and US political spending could absorb billions per year, tens of billions or more for the first two. Philanthropic funding of a preferred AI lab at the cutting edge as model sizes inflate could take all EA funds and more on its own.
There are also many expensive biosecurity interventions that are being compared against an AI intervention benchmark. Things like developing PPE, better sequencing/detection, countermeasures through philanthropic funding rather than hoping to leverage cheaper government funding.
Expected lives saved and taken are both infinite, yes.