CarlShulman

Karma: 5,547

Some personal thoughts on EA and systemic change

CarlShulman26 Sep 2019 21:40 UTC

198 points

17 comments6 min readEA link

CarlShulman 6 Dec 2023 14:43 UTC
156 points
16 ∶ 9
in reply to: Vasco Grilo’s comment on: Open Phil Should Allocate Most Neartermist Funding to Animal Welfare
I can’t speak for Open Philanthropy, but I can explain why I personally was unmoved by the Rethink report (and think its estimates hugely overstate the case for focusing on tiny animals, although I think the corrected version of that case still has a lot to be said for it).

Luke says in the post you linked that the numbers in the graphic are not usable as expected moral weights, since ratios of expectations are not the same as expectations of ratios.

However, I say “naively” because this doesn’t actually work, due to two-envelope effects...whenever you’re tempted to multiply such numbers by something, remember two-envelope effects!)
[Edited for clarity] I was not satisfied with Rethink’s attempt to address that central issue, that you get wildly different results from assuming the moral value of a fruit fly is fixed and reporting possible ratios to elephant welfare as opposed to doing it the other way around.

It is not unthinkably improbable that an elephant brain where reinforcement from a positive or negative stimulus adjust millions of times as many neural computations could be seen as vastly more morally important than a fruit fly, just as one might think that a fruit fly is much more important than a thermostat (which some suggest is conscious and possesses preferences). Since on some major functional aspects of mind there are differences of millions of times, that suggests a mean expected value orders of magnitude higher for the elephant if you put a bit of weight on the possibility that moral weight scales with the extent of, e.g. the computations that are adjusted by positive and negative stimuli. A 1% weight on that plausible hypothesis means the expected value of the elephant is immense vs the fruit fly. So there will be something that might get lumped in with ‘overwhelming hierarchicalism’ in the language of the top-level post. Rethink’s various discussions of this issue in my view missed the mark.

Go the other way and fix the value of the elephant at 1, and the possibility that value scales with those computations is treated as a case where the fly is worth ~0. Then a 1% or even 99% credence in value scaling with computation has little effect, and the elephant-fruit fly ratio is forced to be quite high so tiny mind dominance is almost automatic. The same argument can then be used to make a like case for total dominance of thermostat-like programs, or individual neurons, over insects. And then again for individual electrons.

As I see it, Rethink basically went with the ‘ratios to fixed human value’, so from my perspective their bottom-line conclusions were predetermined and uninformative. But the alternatives they ignore lead me to think that the expected value of welfare for big minds is a lot larger than for small minds (and I think that can continue, e.g. giant AI minds with vastly more reinforcement-affected computations and thoughts could possess much more expected welfare than humans, as many humans might have more welfare than one human).

I agree with Brian Tomasik’s comment from your link:
the moral-uncertainty version of the [two envelopes] problem is fatal unless you make further assumptions about how to resolve it, such as by fixing some arbitrary intertheoretic-comparison weights (which seems to be what you’re suggesting) or using the parliamentary model.
By the same token, arguments about the number of possible connections/counterfactual richness in a mind could suggest superlinear growth in moral importance with computational scale. Similar issues would arise for theories involving moral agency or capacity for cooperation/game theory (on which humans might stand out by orders of magnitude relative to elephants; marginal cases being socially derivative), but those were ruled out of bounds for the report. Likewise it chose not to address intertheoretic comparisons and how those could very sharply affect the conclusions. Those are the kinds of issues with the potential to drive massive weight differences.

I think some readers benefitted a lot from reading the report because they did not know that, e.g. insects are capable of reward learning and similar psychological capacities. And I would guess that will change some people’s prioritization between different animals, and of animal vs human focused work. I think that is valuable. But that information was not new to me, and indeed I had argued for many years that insects met a lot of the functional standards one could use to identify the presence of well-being, and that even after taking two-envelopes issues and nervous system scale into account expected welfare at stake for small wild animals looked much larger than for FAW.

I happen to be a fan of animal welfare work relative to GHW’s other grants at the margin because animal welfare work is so highly neglected (e.g. Open Philanthropy is a huge share of world funding on the most effective FAW work but quite small compared to global aid) relative to the case for views on which it’s great. But for me Rethink’s work didn’t address the most important questions, and largely baked in its conclusions methodologically.
What links here?

CarlShulman 3 Sep 2022 18:48 UTC
103 points
1 ∶ 1
on: The discount rate is not zero
It’s quite likely the extinction/existential catastrophe rate approaches zero within a few centuries if civilization survives, because:
1. Riches and technology make us comprehensively immune to natural disasters.
2. Cheap ubiquitous detection, barriers, and sterilization make civilization immune to biothreats
3. Advanced tech makes neutral parties immune to the effects of nuclear winter.
4. Local cheap production makes for small supply chains that can regrow from disruption as industry becomes more like information goods.
5. Space colonization creates robustness against local disruption.
6. Aligned AI blocks threats from misaligned AI (and many other things).
7. Advanced technology enables stable policies (e.g. the same AI police systems enforce treaties banning WMD war for billions of years), and the world is likely to wind up in some stable situation (bouncing around until it does).
If we’re more than 50% likely to get to that kind of robust state, which I think is true, and I believe Toby does as well, then the life expectancy of civilization is very long, almost as long on a log scale as with 100%.
Your argument depends on 99%+++ credence that such safe stable states won’t be attained, which is doubtful for 50% credence, and quite implausible at that level. A classic paper by the climate economist Martin Weitzman shows that the average discount rate over long periods is set by the lowest plausible rate (as the possibilities of high rates drop out after a short period and you get a constant factor penalty for the probability of low discount rates, not exponential decay).
What links here?

CarlShulman 23 Aug 2020 20:21 UTC
94 points
1 ∶ 0
on: ‘Existential Risk and Growth’ Deep Dive #2 - A Critical Look at Model Conclusions
My main issue with the paper is that it treats existential risk policy as the result of a global collective utility-maximizing decision based on people’s tradeoffs between consumption and danger. But that is assuming away approximately all of the problem.
If we extend that framework to determine how much society would spend on detonating nuclear bombs in war, the amount would be zero and there would be no nuclear arsenals. The world would have undertaken adequate investments in surveillance, PPE, research, and other capacities in response to data about previous coronaviruses such as SARS to stop COVID-19 in its tracks. Renewable energy research funding would be vastly higher than it is today, as would AI technical safety. As advanced AI developments brought AI catstrophic risks closer, there would be no competitive pressures to take risks with global externalities in development either by firms or nation-states.
Externalities massively reduce the returns to risk reduction, with even the largest nation-states being only a small fraction of the world, individual politicians much more concerned with their term of office and individual careers than national-level outcomes, and individual voters and donors constituting only a minute share of the affected parties. And conflict and bargaining problems are entirely responsible for war and military spending, central to the failure to overcome externalities with global climate policy, and core to the threat of AI accident catastrophe.
If those things were solved, and the risk-reward tradeoffs well understood, then we’re quite clearly in a world where we can have very low existential risk and high consumption. But if they’re not solved, the level of consumption is not key: spending on war and dangerous tech that risks global catastrophe can be motivated by the fear of competitive disadvantage/local catastrophe (e.g. being conquered) no matter how high consumption levels are.
What links here?

CarlShulman 4 Sep 2019 22:38 UTC
93 points
0 ∶ 0
on: Are we living at the most influential time in history?
Thanks for this post Will, it’s good to see some discussion of this topic. Beyond our previous discussions, I’ll add a few comments below.

hingeyness
I’d like to flag that I would really like to see a more elegant term than ‘hingeyness’ become standard for referring to the ease of influence in different periods.
Even just a few decades ago, a longtermist altruist would not have thought of risk from AI or synthetic biology, and wouldn’t have known that they could have taken action on them.
I would dispute this. Possibilities of AGI and global disaster were discussed by pioneers like Turing, von Neumann, Good, Minsky and others from the founding of the field of AI.
The possibility of engineered plagues causing an apocalypse was a grave concern of forward thinking people in the early 20th century as biological weapons were developed and demonstrated. Many of the anti-nuclear scientists concerned for the global prospects of humanity were also concerned about germ warfare.
Both of the above also had prominent fictional portrayals to come to mind for longtermist altruists engaging in a wide-ranging search. If there had been a longtermist altruist movement trying to catalog risks of human extinction I think they would have found both of the above, and could have worked to address them (there was reasonable scientific uncertainty about AI timelines, and people could reasonably have developed a lot more of the theory and analysis for AI alignment related topics at the time; on biological weapons arms control could have been much more effective, better governance of DURC developed, etc).
I think this goes to a broader question about the counterfactual to use for your HoH measure: there wasn’t any longtermist altruist community as such in these periods, so the actual returns of all longtermist altruist strategies were zero. To talk about what they would have been one needs to consider a counterfactual in which we anachronistically introduce at least some minimal version of longtermist altruism, and what one includes in that intervention will affect the result one extracts from the exercise.
So, in general, hinginess is increasing, because our ability to think about the long-run effects of our actions, evaluate them, and prioritise accordingly, is increasing.
I agree we are learning more about how to effectively exert resources to affect the future, but if your definition is concerned with the effect of a marginal increment of resources (rather than the total capacity of an era), then you need to wrestle with the issue of diminishing returns. Smallpox eradication was extraordinarily high return compared to the sorts of global health interventions being worked on today with a more crowded field. Founding fields like AI safety or population ethics is much better on a per capita basis than expanding them by 1% after they have developed more. The longtermist of 1600 would indeed have mostly ‘invested’ in building a movement and eventually in things like financial assets when movement-building returns fell below financial returns, but they also should have made concrete interventions like causing the leveraged growth of institutions like science and the Enlightenment that looked to have a fair chance of contributing to HoH scenarios over the coming centuries, and those could have paid off.
This is analogous to the general point in financial markets that asset classes with systematically high returns only have them before those returns are widely agreed on to be valuable and accessible. So startup founders or CEOs can earn large excess returns in expected value for their huge concentrated positions in their firms (in their founder shareholding and stock-based compensation) because of asymmetric information and incentive problems: investors want the founder or CEO to have a concentrated position to ensure good management, but the risk-adjusted value of a concentrated position is less for the same expected value, so the net arrangement delivers a lot of excess expected value.
A world in which everyone has shared correct values and strong knowledge of how to improve things is one in which marginal longtermist resources are gilding the lily. Insofar as one knows that longtermist altruists happen to find themselves with some advantages (e.g. high education in an era of educational inequality, and longtermism relevant values or knowledge in particular) is a potentially important asset to make use of.
The simulation update argument against HoH
I would note that the creation of numerous simulations of HoH-type periods doesn’t reduce the total impact of the actual HoH folk. E.g. say that HoH folk get to influence 10^60 future people, and also get their lives simulated 10^50 times (with no ability to impact things beyond their own lives), while folk in a non-HOH Earthly period get to influence 10^55 future people and get simulated 10^42 times. Because simulations account for a small minority of the total influence, the expected value of an action (or the evidential value of a strategy across all like minds) is still driven primarily by the non-simulated cases. Seeming HoH folk may be simulated more often, but still have most of their influence through unsimulated shaping of history.
If simulations were so numerous that most of the value in history lay in simulations, rather than in basement-level influence, then things might be different. But I think argument #3 doesn’t work for this reason.
Third, even if we’re at some enormously influential time right now, if there’s some future time that is even more influential, then the most obvious EA activity would be to invest resources (whether via financial investment or some sort of values-spreading) in order that our resources can be used at that future, more high-impact, time. Perhaps there’s some reason why that plan doesn’t make sense; but, currently, almost no-one is even taking that possibility seriously.
I think this overstates the case. Diminishing returns to expenditures in a particular time favor a nonzero disbursement rate (e.g. with logarithmic returns to spending at a given time 10x HoH levels would drive a 10x expenditure for a given period).
Most resources associated with EA are in investments. As Michael Dickens writes, small donors are holding most of their assets as human capital and not borrowing against it, while large donors such as Good Ventures are investing the vast majority of their financial assets for future donations. Insofar as people who have not yet entered EA but will are part of the broad EA portfolio, the annual disbursement rate of the total portfolio is even lower, perhaps 1-2% or less. And investment returns mean that equal allocation of NPV of current assets between time periods yields larger total spending in future periods (growing with the investment rate).
Moreover, quite a lot of EA donations actually consist in field- and movement-building (EA, longtermism, x-risk reduction), to the point of drawing criticism about excessive inward focus in some cases. Insofar as those fields are actually built they will create and attract resources with some flexibility to address future problems, and look like investments (this is not universal; e.g. GiveDirectly cash transfers had a larger field-building element when GiveDirectly was newer, but it is hard to recover increased future altruistic capacities later from cash transfers).
Looking through history, some candidates for particularly influential times might include the following (though in almost every case, it seems to me, the people of the time would have been too intellectually impoverished to have known how hingey their time was and been able to do anything about it
I would distinguish between an era being important (on the metric of how much an individual or unit of resource could do) because its population was low, because there was important potential for a lock-in event in a period, and because of high visibility/tractability of longtermist altruists affecting such events (although the effects of that on marginal returns are nonobvious because of crowding, and the highest returns being on neglected assets).
The population factor gets ~monotonically and astronomically worse over time. The chance of lock-in should be distributed across eras (more by technological levels than calendar years), with more as technology advances towards actual high-fidelity stabilization as a possibility (via extinction or lock-in), and less over time thereafter due to pre-emption (if there is a 1/1000 year chance of stabilization in extinction or a locked-in civilization, then the world will almost certainly be in a stable state a million years hence, thus the expected per year chance of stabilization needs to decline enormously on average over the coming era, in addition to falling per capita influence; this is related to LaPlace’s rule of succession, the longer we go under some conditions without an event happening the less likely that it will happen on the next timestep, even aside from the object-level reasons re the speed of light and lock-in tech).
So I would say both the population and pre-emption (by earlier stabillization) factors intensely favor earlier eras in per resource hingeyness, constrained by the era having any significant lock-in opportunities and the presence of longtermists.
When I check that against the opportunities of past periods that does make sense to me. It seems quite plausible that 1960 was a much better time for a marginal altruist to take object-level actions to reduce long-run x-risk (and better in expected terms without the benefit of hindsight regarding things like nuclear doomsday devices and AI and BW timelines) by building relevant fields with less crowding (building a good EAish movement looks even better; ‘which is the hingeyest period’ is distinct from ‘is hingeyness declining faster than ROI for financial instruments or movement-building’).
The growth of a longtermist altruist movement in particular would mean marginal per capita hingeyness (drawn around longtermist interests) should seriously decline going forward.
In contrast, if the hingiest times are in the future, it’s likely that this is for reasons that we haven’t thought of. But there are future scenarios that we can imagine now that would seem very influential
For the later scenarios here you’re dealing with much larger populations. If the plausibility of important lock-in is similar for solar colonization and intergalactic colonization eras, but the population of the latter is billions of times greater, it doesn’t seem to be at all an option that it could be the most HoH period on a per resource unit basis.
What links here?
- Crucial questions about optimal timing of work and donations by MichaelA (14 Aug 2020 8:43 UTC; 45 points)
- Ben Snodin's comment on Ben Snodin’s Quick takes by Ben Snodin (1 May 2021 8:18 UTC; 7 points)

CarlShulman 4 Jun 2021 7:12 UTC
86 points
1 ∶ 0
in reply to: jasoncrawford’s comment on: Help me find the crux between EA/XR and Progress Studies
Your 3 items cover good+top priority, good+not top priority, and bad+top priority, but not #4, bad+not top priority.

I think people concerned with x-risk generally think that progress studies as a program of intervention to expedite growth is going to have less expected impact (good or bad) on the history of the world per unit of effort, and if we condition on people thinking progress studies does more harm than good, then mostly they’ll say it’s not important enough to focus on arguing against at the current margin (as opposed to directly targeting urgent threats to the world). Only a small portion of generalized economic expansion will go to the most harmful activities (and damage there comes from expediting dangerous technologies in AI and bioweapons that we are improving in our ability to handle, so that delay would help) or to efforts to avert disaster, so there is much more leverage focusing narrowly on the most important areas.

With respect to synthetic biology in particular, I think there is a good case for delay: right now the capacity to kill most of the world’s population with bioweapons is not available in known technologies (although huge secret bioweapons programs like the old Soviet one may have developed dangerous things already), and if that capacity is delayed there is a chance it will be averted or much easier to defend against via AI, universal sequencing, and improvements in defenses and law enforcement. This is even moreso for those sub-areas that most expand bioweapon risk. That said, any attempt to discourage dangerous bioweapon-enabling research must compete against other interventions (improved lab safety, treaty support, law enforcement, countermeasure platforms, etc), and so would have to itself be narrowly targeted and leveraged.

With respect to artificial intelligence, views on sign vary depending on whether one thinks the risk of an AI transition is getting better or worse over time (better because of developments in areas like AI alignment and transparency research, field-building, etc; or worse because of societal or geopolitical changes). Generally though people concerned with AI risk think it much more effective to fund efforts to find alignment solutions and improved policy responses (growing them from a very small base, so cost-effectiveness is relatively high) than a diffuse and ineffective effort to slow the technology (especially in a competitive world where the technology would be developed elsewhere, perhaps with higher transition risk).

For most other areas of technology and economic activity (e.g. energy, agriculture, most areas of medicine) x-risk/longtermist implications are comparatively small, suggesting a more neartermist evaluative lens (e.g. comparing more against things like GiveWell).

Long-lasting (centuries) stagnation is a risk worth taking seriously (and the slowdown of population growth that sustained superexponential growth through history until recently points to stagnation absent something like AI to ease the labor bottleneck), but seems a lot less likely than other x-risk. If you think AGI is likely this century then we will return to the superexponential track (but more explosively) and approach technological limits to exponential growth followed by polynomial expansion in space. Absent AGI or catastrophic risk (although stagnation with advanced WMD would increase such risk), permanent stagnation also looks unlikely based on the capacities of current technology given time for population to grow and reach frontier productivity.

I think the best case for progress studies being top priority would be strong focus on the current generation compared to all future generations combined, on rich country citizens vs the global poor, and on technological progress over the next few decades, rather than in 2121. But given my estimates of catastrophic risk and sense of the interventions, at the current margin I’d still think that reducing AI and biorisk do better for current people than the progress studies agenda per unit of effort.

I wouldn’t support arbitrary huge sacrifices of the current generation to reduce tiny increments of x-risk, but at the current level of neglectedness and impact (for both current and future generations) averting AI and bio catastrophe looks more impactful without extreme valuations. As such risk reduction efforts scale up marginal returns would fall and growth boosting interventions would become more competitive (with a big penalty for those couple of areas that disproportionately pose x-risk).

That said, understanding tech progress, returns to R&D, and similar issues also comes up in trying to model and influence the world in assorted ways (e.g. it’s important in understanding AI risk, or building technological countermeasures to risks to long term development). I have done a fair amount of investigation that would fit into progress studies as an intellectual enterprise for such purposes.

I also lend my assistance to some neartermist EA research focused on growth, in areas that don’t very disproportionately increase x-risk, and to development of technologies that make it more likely things will go better.

CarlShulman 3 Jul 2022 15:46 UTC
85 points
0 ∶ 0
on: Why AGI Timeline Research/Discourse Might Be Overrated
There are very expensive interventions that are financially constrained and could use up ~all EA funds, and the cost-benefit calculation takes probability of powerful AGI in a given time period as an input, so that e.g. twice the probability of AGI in the next 10 years justifies spending twice as much for a given result by doubling the chance the result gets to be applied. That can make the difference between doing the intervention or not, or drastic differences in intervention size.
What links here?
- My take on What We Owe the Future by elifland (1 Sep 2022 18:07 UTC; 353 points)
- Forecasting Newsletter: June 2022 by NunoSempere (12 Jul 2022 12:35 UTC; 49 points)

CarlShulman 26 Sep 2019 19:34 UTC
81 points
0 ∶ 0
on: Summary of my academic paper “Effective Altruism and Systemic Change”
My sense of what is happening regarding discussions of EA and systemic change is:
- Actual EA is able to do assessments of systemic change interventions including electoral politics and policy change, and has done so a number of times
  - Empirical data on the impact of votes, the effectiveness of lobbying and campaign spending work out without any problems of fancy decision theory or increasing marginal returns
    E.g. Andrew Gelman’s data on US Presidential elections shows that given polling and forecasting uncertainty a marginal vote in a swing state average something like a 1 in 10 million chance of swinging an election over multiple elections (and one can save to make campaign contributions
    80,000 Hours has a page (there have been a number of other such posts and discussion, note that ‘worth voting’ and ‘worth buying a vote through campaign spending or GOTV’ are two quite different thresholds) discussing this data and approaches to valuing differences in political outcomes between candidates; these suggest that a swing state vote might be worth tens of thousands of dollars of income to rich country citizens
    But if one thinks that charities like AMF do 100x or more good per dollar by saving the lives of the global poor so cheaply, then these are compatible with a vote being worth only a few hundred dollars
    If one thinks that some other interventions, such as gene drives for malaria eradication, animal advocacy, or existential risk interventions are much more cost-effective than AMF, that would lower the value further except insofar as one could identify strong variation in more highly-valued effects
    Experimental data on the effects of campaign contributions suggest a cost of a few hundred dollars per marginal vote (see, e.g. Gerber’s work on GOTV experiments)
    Prediction markets and polling models give a good basis for assessing the chance of billions of dollars of campaign funds swinging an election
    If there are increasing returns to scale from large-scale spending, small donors can convert their funds into a small chance of huge funds, e.g. using a lottery, or more efficiently (more than ¹⁄_1,000 chance of more than 1000x funds) through making longshot bets in financial markets, so IMR are never a bar to action (also see the donor lottery)
    The main thing needed to improve precision for such estimation of electoral politics spending is carefully cataloging and valuing different channels of impact (cost per vote and electoral impact per vote are well-understood)
    More broadly there are also likely higher returns than campaign spending in some areas such as think tanks, lobbying, and grassroots movement-building; ballot initiative campaign spending is one example that seems like it may have better returns than spending on candidates (and EAs have supported several ballot initiatives financially, such as restoration of voting rights to convicts in Florida, cage bans, and increased foreign spending)
  - A recent blog post by the Open Philanthropy Project describes their cost-effectiveness estimates from policy search in human-oriented US domestic policy, including criminal justice reform, housing reform, and others
    It states that thus far even ex ante estimates of effect there seem to have only rarely outperformed GiveWell style charities
    However it says: “One hypothesis we’re interested in exploring is the idea of combining multiple sources of leverage for philanthropic impact (e.g., advocacy, scientific research, helping the global poor) to get more humanitarian impact per dollar (for instance via advocacy around scientific research funding or policies, or scientific research around global health interventions, or policy around global health and development). Additionally, on the advocacy side, we’re interested in exploring opportunities outside the U.S.; we initially focused on U.S. policy for epistemic rather than moral reasons, and expect most of the most promising opportunities to be elsewhere. ”
  - Let’s Fund’s fundraising for climate policy work similarly made an attempt to estimate the impacts of their proposed intervention in this sort of fashion; without endorsing all the details of their analysis, I think it is an example of EA methodologies being quite capable of modeling systemic interventions
  - Animal advocates in EA have obviously pursued corporate campaigns and ballot initiatives which look like systemic change to me, including quantitative estimates of the impact of the changes and the effects of the campaigns
- The great majority of critics of EA invoking systemic change fail to present the simple sort of quantitative analysis given above for the interventions they claim, and frequently when such analysis is done the intervention does not look competitive by EA lights
  - A common reason for this is EAs taking into account the welfare of foreigners, nonhuman animals and future generations; critics may propose to get leverage by working through the political system but give up on leverage from concern for neglected beneficiaries, and in other cases the competition is interventions that get leverage from advocacy or science combined with a focus on neglected beneficiaries
  - Sometimes systemic change critiques come from a Marxist perspective that assumes Marxist revolution will produce a utopia, whereas empirically such revolution has been responsible for impoverishing billions of people, mass killing, the Cold War, (with risk of nuclear war) and increased tensions between China and democracies, creating large object-level disagreements with many EAs who want to accurately forecast the results of political action
- Nonetheless, my view is that historical data do show that the most efficient political/advocacy spending, particularly aiming at candidates and issues selected with an eye to global poverty or the long term, does have higher returns than GiveWell top charities (even ignoring nonhumans and future generations or future technologies); one can connect the systemic change critique as a position in intramural debates among EAs about the degree to which one should focus on highly linear, giving as consumption, type interventions
  - E.g. I would rather see $1000 go to something like the Center for Global Development, Target Malaria’s gene drive effort, or the Swiss effective foreign aid ballot initiative than the Against Malaria Foundation
  - I do think it is true that well-targeted electoral politics spending has higher returns than AMF, because of the impacts of elections on things such as science, foreign aid, great power war, AI policy, etc, provided that one actually directs one’s efforts based on the neglected considerations
- EAs who are willing to consider riskier and less linear interventions are mostly already pursuing fairly dramatic systemic change, in areas with budgets that are small relative to political spending (unlike foreign aid),
  - Global catastrophic risks work is focused on research and advocacy to shift the direction of society as a whole on critical issues, and the collapse of human civilization or its replacement by an undesirable successor would certainly be a systemic change
  - As mentioned previously, short-term animal EA work is overwhelmingly focused on systemic changes, through changing norms and laws, or producing technologies that would replace and eliminate the factory farming system
  - A number of EA global poverty focused donors do give to organizations like CGD, meta interventions to grow the EA movement (which can eventually be cashed in for larger systemic change), and groups like GiveWell or the Poverty Action Lab
    Although there is a relative gap in longtermist and high-risk global poverty work compared to other cause areas, that does make sense in terms of ceiling effects, arguments for the importance of trajectory changes from a longtermist perspective, and the role of GiveWell as a respected charity evaluator providing a service lacking for other areas
  - Issue-specific focus in advocacy makes sense for these areas given the view that they are much more important than the average issue and with very low spending
- As funding expands in focused EA priority issues, eventually diminishing returns there will equalize with returns for broader political spending, and activity in the latter area could increase enormously: since broad political impact per dollar is flatter over a large range political spending should either be a very small or very large portion of EA activity
  - Essentially, the cost per vote achieved through things like campaign spending is currently set by the broader political culture and has the capacity to absorb billions of dollars at similar cost-effectiveness to the current level, so it should either be the case that EA funds very little of it or enormous amounts of it
    There is a complication in that close elections or other opportunities can vary the effectiveness of political spending over time, which would suggest saving most funds for those
  - The considerations are similar to GiveDirectly: since cash transfers could absorb all EA funds many times over at similar cost-effectiveness (with continued rapid scaling), it should take in either very little or almost all EA funding; in a forced choice it should either be the case that most funding goes to cash transfers, whereas for other interventions with diminishing returns on the relevant scale as mixed portfolio will yield more impact
  - For now areas like animal advocacy and AI safety with budgets of only tens of millions of dollars are very small relative to political spending, and the impact of the focused work (including relevant movement building) makes more of a difference to those areas than a typical difference between political candidates; but if billions of dollars were being spent in those areas it would seem that political activity could be a competitive use (e.g. supporting pro-animal candidates for office)

CarlShulman 26 Dec 2020 20:45 UTC
73 points
1 ∶ 0
on: What is the likelihood that civilizational collapse would directly lead to human extinction (within decades)?
Thanks for this detailed post on an underdiscussed topic! I agree with the broad conclusion that extinction via partial population collapse and infrastructure loss, rather than by the mechanism of catastrophe being potent enough to leave no or almost no survivors (or indirectly enabling some later extinction level event) has very low probability. Some comments:
- Regarding case 1, with a pandemic leaving 50% of the population dead but no major infrastructure damage, I think you can make much stronger claims about there not being ‘civilization collapse’ meaning near-total failure of industrial food, water, and power systems. Indeed, collapse so defined from that stimulus seems nonsensical to me for rich quantitative reasons.
  - There is no WMD war here, otherwise there would be major infrastructure damage.
  - If half of people are dead, that cuts the need for food and water by half (doubling per capita stockpiles), while already planted calorie-rich crops can easily be harvested with a half-size workforce.
  - Today agriculture makes up closer to 5% than 10% of the world economy, and most of that effort is expended on luxuries such as animal agriculture, expensive fruits, avoidable food waste, and other things that aren’t efficient ways to produce nutrition. Adding all energy (again, most of which is not needed for basic survival as opposed to luxuries) brings the total to ~15%, and perhaps 5% on necessities (2.5% for half production for half population). That leaves a vast surplus workforce.
  - The catastrophe doubles resources of easily accessible fossil fuels and high quality agricultural land per surviving person, so just continuing to run the best 50% of farmland and the best 50% of oil wells means an increase in food and fossil fuels per person.
  - Likewise, there is a surplus of agricultural equipment, power plants, water treatment plants, and operating the better half of them with the surviving half of the population could improve per capita availability. These plants are parallel and independent enough that running half of them would not collapse productivity, which we can confirm by looking back to when there were half as many, etc.
  - Average hours worked per capita is already at historical lows, leaving plenty of room for trained survivors to work longer shifts while people switch over from other fields and retrain
  - Historical plagues such as the Black Death or smallpox in the Americas did not cause a breakdown of food production per capita for the survivors.
  - Historical wartime production changes show enormous and adequate flexibility in production.
- Re the likelihood of survival without industrial agriculture systems, the benchmark should be something closer to preindustrial European agriculture, not hunter-gatherers. You discuss this but it would be helpful to put more specific credences on those alternatives.
  - The productivity of organic agriculture is still enormously high relative to hunting and gathering.
  - Basic knowledge about crop rotation, access to improved and global crop varieties such as potatoes, ploughs, etc permitted very high population density before industrial agriculture, with very localized supply chains. One can see this in colonial agricultural communities which could be largely self-sustaining (mines for metal tools being one of the worst supply constraints, but fine in a world where so much metal has already been mined and is just sitting around for reuse).
  - By the same token, talking about ‘at least 10%’ of 1-2 billion subsistence farmers continuing agriculture is a very low figure. I assume it is a fairly extreme lower bound, but it would be helpful to put credences on lower bounds and to help distinguish them from more likely possibilities.
- Re food stockpiles:
  - “I’m ignoring animal agriculture and cannibalism, in part because without a functioning agriculture system, it’s not clear to me whether enough people would be able to consume living beings.”
    Existing herds of farmed animals would likely be killed and eaten/preserved.
    If transport networks are crippled, then this could be for local consumption, but that would increase food inequality and likelihood of survival in dire situations
    There are about 1 billion cattle alone, with several hundred kg of edible mass each, plus about a billion sheep, ~700 million pigs, and 450 million goats.
    In combination these could account for hundreds of billions of human-days of nutritional requirements (I think these make up a large share of ‘global food stocks’ in your table of supplies)
  - Already planted crops ready to harvest constitute a huge stockpile for the scenarios without infrastructure damage.
  - Particularly for severe population declines, fishing is limited by fish supplies, and existing fishing boats capture and kill vast quantities of fishes in days when short fishing seasons open. If the oceans are not damaged, this provides immense food resources to any survivors with modern fishing knowledge and some surviving fishing equipment.
- “But if it did, I expect that the ~4 billion survivors would shrink to a group of 10–100 million survivors during a period of violent competition for surviving goods in grocery stores/distribution centers, food stocks, and fresh water sources.”
- “So what, concretely, do I think would happen in the event of a catastrophe like a “moderate” pandemic — one that killed 50% of people, but didn’t cause infrastructure damage or climate change? My best guess is that civilization wouldn’t actually collapse everywhere. But if it did, I expect that the ~4 billion survivors would shrink to a group of 10–100 million survivors during a period of violent competition for surviving goods in grocery stores/distribution centers, food stocks, and fresh water sources.”
  - For the reasons discussed above I strongly disagree with the claim after “I expect.”
- “All this in mind, I think it is very likely that the survivors would be able to learn enough during the grace period to be able to feed and shelter themselves ~indefinitely.”
  - I would say the probability should be higher here.
- Regarding radioactive fallout, an additional factor not discussed is the decline of fallout danger over time: lethal areas are quite different over the first week vs the first year, etc.
- Re Scenario 2: “Given all of this, my subjective judgment is that it’s very unlikely that this scenario would more or less directly lead to human extinction” I would again say this is even less likely.
- In general I think extinction probability from WMD war is going to be concentrated in the plausible future case of greatly increased/deadlier arsenals: millions of nuclear weapons rather than thousands, enormous and varied bioweapons arsenals, and billions of anti-population hunter-killer robotic drones slaughtering survivors including those in bunkers, all released in the same conflict.
- “Given this, I think it’s fairly likely, though far from guaranteed, that a catastrophe that caused 99.99% population loss, infrastructure damage, and climate change (e.g. a megacatastrohe, like a global war where biological weapons and nuclear weapons were used) would more or less directly cause human extinction.”
  - This seems like a sign error, differing from your earlier and later conclusions?
  - “I think it’s fairly unlikely that humanity would go extinct as a direct result of a catastrophe that caused the deaths of 99.99% of people (leaving 800 thousand survivors), extensive infrastructure damage, and temporary climate change (e.g. a more severe nuclear winter/asteroid impact, plus the use of biological weapons).”
What links here?
- Vasco Grilo's comment on Famine deaths due to the climatic effects of nuclear war by Vasco Grilo (17 Oct 2023 9:59 UTC; 2 points)

Risk-neutral donors should plan to make bets at the margin at least as well as giga-donors in expectation

CarlShulman31 Dec 2016 2:19 UTC

64 points

9 comments20 min readEA link

CarlShulman 27 Aug 2022 22:43 UTC
62 points
0 ∶ 0
on: Climate Change & Longtermism: new book-length report
Imagine that only a few hundred people in the world thought that climate change is an important problem (rather than at least tens of millions), that philanthropists worldwide spent a few million dollars a year on climate (rather than $10 billion), that society as a whole spent a million dollars on the problem (rather than $1 trillion), and that the international institutions trying to tackle the problem either don’t exist or have a similar budget to a McDonald’s restaurant. How bad would climate change be? This is how bad things are for the other global catastrophic risks, and then some.
This seems to overstate how bad the situation is (although qualitatively it remains an absurd underinvestment, with painfully low-hanging fruit to avert pandemics and AI catastrophe at hand). Surveys of the general public and area experts do show substantial percentages in the abstract endorse nuclear (in particular), bioweapon, and AI risks as important problems (as you mention later). Governments wage wars and spend very large amounts of attention and resources on nuclear proliferation and threats. Biodefense has seen billions of dollars of spending, even if it was not well-crafted to reduce catastrophic bioweapon risk. The low budget for the BWC is in significant part a political coordination problem and not simply a $ supply issue. Annual spending from Open Philanthropy and the Future Fund on catastrophic risks, with priorities close to yours, is now in the hundreds of millions of dollars.

CarlShulman 30 Jun 2022 16:17 UTC
62 points
14 ∶ 1
on: Fanatical EAs should support very weird projects
Even if you try to follow an unbounded utility function (which has deep mathematical problems, but set those aside for now) these don’t follow.
Generally the claims here fall prey to the fallacy of unevenly applying the possibility of large consequences to some acts where you highlight them and not to others, such that you wind up neglecting more likely paths to large consequences.
For instance, in an infinite world (including infinities creating by infinite branching faster than you can control) with infinite copies of you, any decision, e.g. eating an apple, has infinite consequences on decision theories that account for the fact that all must make the same (distribution of ) decisions . If perpetual motion machines or hypercomputation or baby universes are possible, then making a much more advanced and stable civilization is far more promising for realizing things related to that then giving in to religions where you have very high likelihood ratios that they don’t feed into cosmic consequences.
Any plan for infinite/cosmic impact that has an extremely foolish step in it (like Pascal’s Mugging) is going to be dominated by less foolish plans.
There will still be implications of unbounded utility functions that are weird and terrible by the standards of other values, but they would have to follow from the most sophisticated analysis, and wouldn’t have foolish instrumental irrationalities or uneven calculation of possible consequences.
A lot of these scenarios are analogous to someone caricaturing the case for aid to the global poor as implying that people should give away all of the food they have (sending it by FedEx) to famine-struck regions, until they themselves starve to death. Yes, cosmopolitan concern for the poor can elicit huge sacrifices of other values like personal wellbeing or community loyalty, but that hypothetical is obviously wrong on its own terms as an implication.
What links here?
- Ramiro's comment on Fanatical EAs should support very weird projects by Derek Shiller (30 Jun 2022 17:36 UTC; 3 points)

CarlShulman 7 Nov 2020 16:11 UTC
60 points
1 ∶ 0
on: Nuclear war is unlikely to cause human extinction
I agree it’s very unlikely that a nuclear war discharging current arsenals could directly cause human extinction. But the conditional probability of extinction given all-out nuclear war can go much higher if the problem gets worse. Some aspects of this:

-at the peak of the Cold War arsenals there were over 70,000 nuclear weapons, not 14,000
-this Brookings estimate puts spending building the US nuclear arsenal at several trillion current dollars, with lower marginal costs per weapon, e.g. $20M per weapon and $50-100M all-in for for ICBMs
-economic growth since then means the world could already afford far larger arsenals in a renewed arms race
-current US military expenditure is over $700B annually, about 1/30th of GDP; at the peak of the Cold War in the 50s and 60s it was about 1/10th; Soviet expenditure was proportionally higher
-so with 1950s proportional military expenditures, half going to nukes, the US and China could each produce 20,000+ ICBMs, each of which could be fitted with MIRVs and several warheads, building up to millions of warheads over a decade or so; the numbers could be higher for cheaper delivery systems
-economies of scale and improvements in technology would likely bring down the per warhead cost
-if AI and robotics greatly increase economic growth the above numbers could be increased by orders of magnitude
-radiation effects could be intentionally greatly increased with alternative warhead composition
-all-out discharge of strategic nuclear arsenals is also much more likely to be accompanied by simultaneous deployment of other WMD, including pandemic bioweapons (which the Soviets pursued as a strategic weapon for such circumstances)and drone swarms (which might kill survivors in bunkers); the combined effects of future versions of all of these WMD at once may synergistically cause extinction
What links here?
- CarlShulman's comment on What is the likelihood that civilizational collapse would directly lead to human extinction (within decades)? by Luisa_Rodriguez (26 Dec 2020 20:45 UTC; 73 points)
- Research project idea: Technological developments that could increase risks from nuclear weapons by MichaelA (15 Apr 2023 14:28 UTC; 16 points)

CarlShulman 8 Feb 2023 15:04 UTC
59 points
13 ∶ 0
in reply to: Eva’s comment on: Why People Use Burner Accounts: A Commentary on Blacklists, the EA “Inner Circle”, and Funding
Here’s an example of a past case where a troll (who also trolled other online communities) made up multiple sock-puppet accounts, and assorted lies about sources for various arguments trashing AI safety, e.g. claiming to have been at events they were not and heard bad things, inventing nonexistent experts who supposedly rejected various claims, creating fake testimonials of badness, smearing people who discovered the deception, etc.

CarlShulman 26 Feb 2024 16:03 UTC
58 points
12 ∶ 0
on: Nuclear war tail risk has been exaggerated?
I agree that people should not focus on nuclear risk as a direct extinction risk (and have long argued this), see Toby’s nuke extinction estimates as too high, and would assess measures to reduce damage from nuclear winter to developing neutral countries mainly in GiveWell-style or ordinary CBA terms, while considerations about future generations would favor focus on AI, and to a lesser extent bio.

However, I do think this wrongly downplays the effects on our civilization beyond casualties and local damage of a nuclear war that wrecks the current nuclear powers, e.g. on disrupting international cooperation, rerolling contingent nice aspects of modern liberal democracy, or leading to release of additional WMD arsenals (such as bioweapons, while disrupting defense against those weapons). So the ‘can nuclear war with current arsenals cause extinction’ question misses most of the existential risk from nuclear weapons, which is indirect in contributing to other risks that could cause extinction or lock-in of permanent awful regimes. I think marginal philanthropic dollars can save more current lives and help the overall trajectory of civilization more on other risks, but I think your direct extinction numbers above do greatly underestimate how much worse the future should be expected to be given a nuclear war that laid waste to, e.g. NATO+allies and the Russian Federation.

You dismiss that here:

> Then discussions move to more poorly understood aspects of the risk (e.g. how the distribution of values after a nuclear war affects the longterm values of transformative AI).

But I don’t think it’s a huge stretch to say that a war with Russia largely destroying the NATO economies (and their semiconductor supply chains), leaving the PRC to dominate the world system and the onrushing creation of powerful AGI, makes a big difference to the chance of locked-in permanent totalitarianism and the values of one dictator running roughshod over the low-hanging fruit of many others’ values. That’s very large compared to these extinction effects. It also doesn’t require bets on extreme and plausibly exaggerated nuclear winter magnitude.

Similarly, the chance of a huge hidden state bioweapons program having its full arsenal released simultaneously (including doomsday pandemic weapons) skyrockets in an all-out WMD war in obvious ways.

So if one were to find super-leveraged ways reduce the chance of nuclear war (this applied less to measures to reduce damage to nonbelligerent states) then in addition to beating GiveWell at saving current lives, they could have big impacts on future generations. Such opportunities are extremely scarce, but the bar for looking good in future generation impacts is less than I think this post suggests.

CarlShulman 24 Nov 2020 2:00 UTC
56 points
0 ∶ 0
on: If Causes Differ Astronomically in Cost-Effectiveness, Then Personal Fit In Career Choice Is Unimportant
Longtermists sometimes argue that some causes matter extraordinarily more than others—not just thousands of times more, but 10^30 or 10^40 times more.
I don’t think any major EA or longtermist institution believes this about expected impact for 10^30 differences. There are too many spillovers for that, e.g. if doubling the world economy of $100 trillion/yr would modestly shift x-risk or the fate of wild animals, then interventions that affect economic activity have to have expected absolute value of impact much greater than 10^-30 of the most expected impactful interventions.

This argument requires that causes differ astronomically in relative cost-effectiveness. If causes A is astronomically better than cause B in absolute terms, but cause B is 50% as good in relative terms, then it makes sense for me to take a job in cause B if I can be at least twice as productive.
I suspect that causes don’t differ astronomically in cost-effectiveness. Therefore, people should pay attention to personal fit when choosing an altruistic career, and not just the importance of the cause.

The premises and conclusion don’t seem to match here. A difference of 10^30x is crazy, but rejecting that doesn’t mean you don’t have huge practical differences in impact like 100x or 1000x. Those would be plenty to come close to maxing out the possible effect of differences between causes(since if you’re 1000x as good at rich-country homelessness relief as preventing pandemics, then if nothing else your fame for rich country poverty relief would be a powerful resource to help out in other areas like public endorsements of good anti-pandemic efforts).

The argument seems sort of like “some people say if you go into careers like quant trading you’ll make 10^30 dollars and can spend over a million dollars to help each animal with a nervous system. But actually you can’t make that much money even as a quant trader, so people should pay attention to fit with different careers in the world when trying to make money, since you can make more money in a field with half the compensation per unit productivity if you are twice as productive there.” The range for realistic large differences in compensation between fields (e.g. fast food cashier vs quant trading) is missing from the discussion.

You define astronomical differences at the start as ‘not just thousands of times more’ but the range to thousands of times more is where all the action is.

CarlShulman 13 Feb 2023 22:18 UTC
49 points
12 ∶ 3
in reply to: Arepo’s comment on: There can be highly neglected solutions to less-neglected problems
Helping the global poor is neglected, and that accounts for most bednet outperformance. GiveDirectly, just giving cash, is thought by GiveWell/GHW to be something like 100x better on direct welfare than rich country consumption (although indirect effects reduce that gap), vs 1000x+ for bednets. So most of the log gains come from doing stuff with the global poor at all. Then bednets have a lot of their gains as positive externalities (protecting one person also protects others around them), and you’re left with a little bit of ’being more confident about bednets than some potential users based on more investigation of the evidence (like vaccines), and some effects like patience/discounting.

Really exceptional intervention-within-area picks can get you a multiplier, but it’s hard to get to the level of difference you see on cause selection, and especially so when you compare attempts to pick out the best in different causes.

CarlShulman 11 May 2019 20:10 UTC
48 points
0 ∶ 0
in reply to: Milan_Griffes’s comment on: Cash prizes for the best arguments against psychedelics being an EA cause area
That sounds a bit like the argument ‘either this claim is right, or it’s wrong, so there’s a 50% chance it’s true.’
One needs to attend to base rates. Our bad academic knowledge-generating process throws up many, many illusory interventions with purported massive effects for each amazing intervention we find, and the amazing interventions that we do find disproportionately were easier to show (with the naked eye, visible macro-correlations, consistent effects with well-powered studies, etc).
People are making similar arguments about cold fusion, psychic powers (of many different varieties), many environmental and nutritional contaminants, brain training, carbon dioxide levels, diets, polyphasic sleep, assorted purported nootropics, many psychological/parenting/educational interventions, etc.
Testing how your prior applies across a spectrum of other cases (past and present) is helpful for model checking. If psychedelics are a promising EA cause how many of those others qualify? If many do, then any one isn’t so individually special, although one might want to have a systematic program of systematically doing rigorous testing of all the wacky claims of large impact that can be tested cheaply.
If not, then it would be good to explain what exactly makes psychedelics different from the rest.
I think the case for psychedelics the OP has made doesn’t pass this standard yet, so doesn’t meet the standard for an EA cause area.
What links here?
- Milan_Griffes's comment on Cash prizes for the best arguments against psychedelics being an EA cause area by Milan_Griffes (3 Jun 2019 2:25 UTC; 8 points)

CarlShulman 24 Apr 2023 17:12 UTC
47 points
16 ∶ 1
on: No, the EMH does not imply that markets have long AGI timelines
I disagree with the idea that short AI timelines are not investable (although I agree interest rates are a bad and lagging indicator vs AI stocks). People foreseeing increased expectations of AI sales as a result of scaling laws, shortish AI timelines, and the eventual magnitude of success have already made a lot of money investing in Nvidia, DeepMind and OpenAI. Incremental progress increases those expectations, and they can increase even in worlds where AGI winds up killing or expropriating all investors so long as there is some expectation of enough investors thinking ownership will continue to matter. In practice I know lots of investors expecting near term TAI who are betting on it (in AI stocks, not interest rates, because the returns are better). They also are more attracted to cheap 30 year mortgages and similar sources of mild cheap leverage. They put weight on worlds where society is not completely overturned and property rights matter after AGI, as well as during an AGI transition (e.g. consider that a coalition of governments wanting to build AGI is more likely to succeed earlier and more safely with more compute and talent available to it, so has reason to make credible promises of those who provide such resources actually being compensated for doing so post-AGI, or the philanthropic value of being able to donate such resources).

And at the object level from reading statements from investors and talking to them, investors weighted by trading in AI stocks (and overwhelmingly for the far larger bond market setting interest rates) largely don’t have short AI timelines (confident enough to be willing to invest on) or expect explosive growth in AI capabilities. There are investors like Cathy Woods who do with tens or hundreds of billions of dollars of capital, but they are few enough relative to the investment opportunities available that they are not setting e.g. the prices for the semiconductor industry. I don’t see the point of indirect arguments from interest rates for the possibility that investors or the market as a whole could believe in AGI soon but only versions where owning the AI chips or AI developers won’t pay off, when at the object level that possibility is known to be false.

CarlShulman 15 Nov 2020 17:38 UTC
47 points
1 ∶ 0
in reply to: William_MacAskill’s comment on: Thoughts on whether we’re living at the most influential time in history
This seems important to me because, for someone claiming that we should think that we’re at the HoH, the update on the basis of earliness is doing much more work than updates on the basis of, say, familiar arguments about when AGI is coming and what will happen when it does. To me at least, that’s a striking fact and wouldn’t have been obvious before I started thinking about these things.
It seems to me the object level is where the action is, and the non-simulation Doomsday Arguments mostly raise a phantom consideration that cancels out (in particular, cancelling out re whether there is an influenceable lock-in event this century).

You could say a similar thing about our being humans rather than bacteria, which cumulatively outnumber us by more than 1,000,000,000,000,000,000,000,000 times on Earth thus far according to the paleontologists.

Or you could go further and ask why we aren’t neutrinos? There are more than 1,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000 of them in the observable universe.

However extravagant the class you pick, it’s cancelled out by the knowledge that we find ourselves in our current situation. I think it’s more confusing than helpful to say that our being humans rather than neutrinos is doing more than 10^70 times as much work as object-level analysis of AI in the case for attending to x-risk/lock-in with AI. You didn’t need to think about that in the first place to understand AI or bioweapons, it was an irrelevant distraction.

The same is true for future populations that know they’re living in intergalactic societies and the like. If we compare possible world A, where future Dyson spheres can handle a population of P (who know they’re in that era), and possible world B, where future Dyson spheres can support a population of 2P, they don’t give us much different expectations of the number of people finding themselves in our circumstances, and so cancel out.

The simulation argument (or a brain-in-vats story or the like) is different and doesn’t automatically cancel out because it’s a way to make our observations more likely and common. However, for policy it does still largely cancel out, as long as the total influence of people genuinely in our apparent circumstances is a lot greater than that of all simulations with apparent circumstances like ours: a bigger future world means more influence for genuine inhabitants of important early times and also more simulations. [But our valuation winds up being bounded by our belief about the portion of all-time resources allocated to sims in apparent positions like ours.]

Another way of thinking about this is that prior to getting confused by any anthropic updating, if you were going to set a policy for humans who find ourselves in our apparent situation across nonanthropic possibilities assessed at the object level (humanity doomed, Time of Perils, early lock-in, no lock-in), you would just want to add up the consequences of the policy across genuine early humans and sims in each (non-anthropically assessed) possible world.

A vast future gives more chances for influence on lock-in later, which might win out as even bigger than this century (although this gets rapidly less likely with time and expansion), but it shouldn’t change our assessment of lock-in this century, and a substantial chance of that gives us a good chance of HoH (or simulation-adjusted HoH).

CarlShulman

Some per­sonal thoughts on EA and sys­temic change

Risk-neu­tral donors should plan to make bets at the mar­gin at least as well as giga-donors in expectation

Some personal thoughts on EA and systemic change

Risk-neutral donors should plan to make bets at the margin at least as well as giga-donors in expectation