The main reason I voted for Forethought and MATS was because I believe AI governance/safety is both unusually important, with only Farmed/Wild animal welfare being competitive in terms of EV, and I believe that AI has a reasonable chance to be so powerful as to make other cause area assumptions irrelevant, meaning their impact is much, much less predictable without considering AI governance/safety.
Sharmake
Karma: 1,088
One of the key issues with “making the future go well” interventions is that we start to run up against the reality that what is a desirable outcome for the future is so variable between different humans that the concept of making the future go well requires buying into ethical assumptions that people won’t share, meaning that it’s much less valid as any sort of absolute metric to coordinate around:
(A quote from Steven Byrnes here):
When people make statements that implicitly treat “the value of the future” as being well-defined, e.g. statements like “I define ‘strong utopia’ as: at least 95% of the future’s potential value is realized”, I’m concerned that these statements are less meaningful than they sound.
This level of variability is less for preventing bad outcomes, especially outcomes in which we don’t die (though there is still variability here) because of instrumental convergence, and while there are moral views where dying/suffering isn’t so bad, these moral views aren’t held by many human beings (in part due to selection effects), so there’s less of a chance to have conflict with other agents.
The other reason is humans mostly value the same scarce instrumental goods, but in a world where AI goes well, basically everything but status/identity becomes abundant, and this surfaces up the latent moral disagreements way more than our current world.
I’m commenting late, but I don’t think the better futures perspective gets us back to intuitive/normie ethical views, because what is a better future has far more variation in values than preventing catastrophic outcomes (I’m making an empirical claim that most human values have more convergence in things they want to avoid than in things they want to seek out/are positive), and the other issue is that to a large extent, AGI/ASI in the medium/long-term is very totalizing in its effects, meaning that basically the only thing that matters is getting a friendly ASI to you, and thus promoting peace/democracy don’t matter, while good governance can actually matter (though it’d have to be way more specific than what Will MacAskill defines as good governance.)
An example here is this quote, which straddles dangerously close to “these people have morality that you find to be offensive, therefore they are wrong on the actual facts of the matter” (Otherwise you would make the Nazi source allegations less central to your criticism here):
(I don’t hold the moral views of what the quote is saying, to be clear).
It has never stopped shocking and disgusting me that the EA Forum is a place where someone can write a post arguing that Black Africans need Western-funded programs to edit their genomes to increase their intelligence in order to overcome global poverty and can cite overtly racist and white supremacist sources to support this argument (even a source with significant connections to the 1930s and 1940s Nazi Party in Germany and the American Nazi Party, a neo-Nazi party) and that post can receive a significant amount of approval and defense from people in EA, even after the thin disguise over top of the racism is removed by perceptive readers. That is such a bonkers thing and such a morally repugnant thing, I keep struggling to find words to express my exasperation and disbelief. Effective altruism as a movement probably deserves to fail for that, if it can’t correct it.[2]
Another issue, and why the comment is getting downvoted heavily (including by myself) is because you seem to conflate the is-ought distinction with this post, and without the is-ought distinction being conflated, this post would not exist.
You routinely leap from “a person has moral views that are offensive to you” to “they are wrong about the facts of the matter”, and your evidence for this is paper thin at best.
Being able to separate moral views from beliefs on factual claims is one of the things that is expected if you are in EA/LW spaces.
This is not mutually exclusive with the issues CB has found.
I currently can’t find a source, but to elaborate a little bit, my reason for thinking this is that the GPT-4 to GPT-4.5 scaleup used 15x the compute instead of 100x the compute, and I remember that 10x compute is enough to be competitive with the current algorithmic improvements that don’t involve scaling up models, whereas 100x compute increases result in the wow moments we associated with GPT-3 to GPT-4, and the GPT-5 release was not a scale up of compute, but instead productionizing GPT-4.5.
I’m more in the camp of “I find little reason to believe that pre-training returns have declined” here.
The crux for me is I don’t agree that compute scaling has dramatically changed, because I don’t think pre-training scaling has gotten much worse returns.
I broadly don’t think inference scaling is the only path, primarily because I disagree with the claim that pre-training returns declined much, and attribute the GPT-4.5 evidence as mostly a case of broken compute promises making everything disappointing.
I also have a hypothesis that current RL is mostly serving as an elicitation method for pre-trained AIs.
We shall see in 2026-2027 whether this remains true.
A big part of the issue, IMO is the fact that EA funding is often very skewed by people who have managed to capture the long-tail of wealth/income, and while this is quite necessary for EA to be as impactful as it is in a world where it’s good for EA to remain small, and I’d still say it was positive overall to do the strategy, this also inevitably distorts any conversations, because people reasonably fear that being unable to justify/defer to a funder about what to do means you can’t get off the ground at all, since there are few alternative funders.
So this sort of deference to funders will likely always remain, unfortunately, and we will have to mitigate the downsides that come from seeking the long-tails of wealth/income (which very few people can achieve).
My general take on gradual disempowerment, independent of any other issues raised here, is that I think it’s a coherent scenario, but that it ultimately is very unlikely to arise in practice, because it relies on an equilibrium where the sort of very imperfect alignment needed for divergence between human and AI interests to occur over the long-run being stable, even as the reasons for why the alignment problem in humans being very spotty/imperfect being stable get knocked out.
In particular, I’m relatively bullish on automated AI alignment conditional on non-power seeking/non-sandbagging if we give the AIs reward but misaligned human-level AI, so I generally think it quite rapidly resolves as either the AI is power-seeking and willing to sandbag/scheme on everything, leading to the classic AI takeover, or the AI is aligned to the principal in such a way that the principal-agency cost becomes essentially 0 over time.
Note I’m not claiming that most humans won’t be dead/disempowered, I’m just saying that I don’t think gradual disempowerment is worth spending much time/money on.
The “arbitrariness” of precise EVs is just a matter of our discomfort with picking a precise number (see above).
A non-trivial reason for this is that precise numbers expose ideological assumptions, and a whole of people do not like this.
It’s easy to lie with numbers, but it’s even easier to lie without a number.
Crossposting a comment from LessWrong:
@1a3orn goes deeper into another dynamic that causes groups to have false beliefs while believing they are true, and it’s the fact that some bullshit beliefs help you figure out who to exclude, which is the people who don’t currently hold the belief, and in particular assholery also helps people who don’t want their claims checked, and it’s a reason I think politeness is actually useful in practice for rationality:
(Sharmake’s first tweet): I wrote something on a general version of this selection effect, and why it’s so hard to evaluate surprising/extreme claims relative to your beliefs, and it’s even harder if we expect heavy-tailed performance, as happens in our universe.
(1a3orn’s claims) This is good. I think another important aspect of the multi-stage dynamic here is that it predicts that movements with *worse* stages at some point have fewer contrary arguments at later points...
...and in this respect is like an advance-fee scam, where deliberately non-credible aspects of the story help filter people early on so that only people apt to buy-in reach later parts.
Paper on Why do Nigerian Scammers Say They are from Nigeria?
So it might be adaptive (survivalwise) for a memeplex to have some bullshit beliefs because the filtering effect of these means that there will be fewer refutations of the rest of the beliefs.
It can also be adaptive (survivalwise) for a leader of some belief system to be abrasive, an asshole, etc, because fewer people will bother reading them ⇒ “wow look how no one can refute my arguments”
(Sharmake’s response) I didn’t cover the case where the belief structure is set up as a scam, and instead focused on where even if we are assuming LWers are trying to get at truth and aren’t adversarial, the very fact that this effect exists combined with heavy-tails makes it hard to evaluate claims.
But good points anyway.
(1a3orn’s final point)
Yeah tbc, I think that if you just blindly run natural selection over belief systems, you get belief systems shaped like this regardless of the intentions of the people inside it. It’s just an effective structure.
Quotes from this tweet thread.
But Have They Engaged With The Arguments? [Linkpost]
Another story is that this is a standard diminishing returns case, and once we have removed all the very big blockers like non-functional rule of law, property rights, untreated food and water, as well as disease, it’s very hard to make the people who would still remain poor actually improve their lives, because all the easy wins have been taken, so what we are left with is the harder/near impossible poverty cases.
I think each galactic x-risk on the list can probably be disregarded, but combined, and with the knowledge that we are extremely early in thinking about this, they present a very convincing case to me that at least 1 or 2 galactic x-risks are possible.
I think this is kind of a crux, in that I currently think the only possible galactic scale risks are risks where our standard model of physics breaks down in a deep way once you can get at least one dyson swarm going up, you are virtually invulnerable to extinction methods that doesn’t involve us being very wrong about physics.
This is always a tail risk of interstellar travel, but I would not say that interstellar travel will probably doom the long-term future as stated in the title.
The better title is interstellar travel poses unacknowledged tail risks.
Really interesting point, and probably a key consideration on existential security for a spacefaring civilisation. I’m not sure if we can be confident enough in acausal trade to rely on it for our long-term existential security though. I can’t imagine human civilisation engaging in acausal trade if we expanded before the development of superintelligence. There are definitely some tricky questions to answer about what we should expect other spacefaring civilisations to do. I think there’s also a good argument for expecting them to systematically eliminate other spacefaring civilisations rather than engage in acausal trade.
I agree that if there’s an X-risk that isn’t defendable (for the sake of argument), then acausal trade is reliant on every other civilization choosing to acausally trade in a manner where the parent civilization can prevent x-risk, but the good news is that a lot of the more plausible (in a relative sense) x-risks have a light-speed limit, meaning that given we are probably alone in the observable universe (via the logic of dissolving the fermi paradox), means that humanity only really has to do acausal trade.
And a key worldview crux is conditioning on humanity becoming a spacefaring civilization, I expect superintelligence that takes over the world to come first, because it’s much easier to develop good enough AI tech to develop space sufficiently than it is for humans to go spacefaring alone.
And AI progress is likely to be fast enough such that there’s very little time for rogue spacefarers to get outside of the parent civilization’s control.
The dissolving the fermi paradox paper is here:
On hot take 2, this relies on the risks from each start system being roughly independent, so breaking this assumption seems like a good solution, but then each star system being very correlated maybe seems bad for liberalism and diversity of forms of flourishing and so forth. But maybe some amount of regularity and conformity is the price we need to pay for galactic security.
I think liberalism is unfortunately on a timer that will almost certainly expire pretty soon, no matter what we do.
We either technologically regress due to the human population falling and more anti-democratic civilizations winning outright due to the zero/negative sum games being played, or we create AIs that replace us and due to the incentives plus the sheer difference in power, that AIs by default create something closer to a dictatorship for humans, and in particular value alignment is absolutely critical in the long run for AIs that can take every human job.
Modern civilization is not stable at all.
Acausal trade/cooperation may end up being crucial here too once civilisation is spread across distances where it is hard or impossible to interact normally.
Yeah, assuming no FTL, acausal trade/cooperation is necessary if you want anything like a unified galactic/universal polity.
Interstellar travel will probably doom the long-term future
A lot of the reason for my disagreement stems from thinking that most galactic-scale disasters either don’t actually serve as x-risks (like the von Neumann probe scenario), because they are defendable, or they require some shaky premises about physics to come true.
The change the universe constants is an example.
Also, in most modern theories of time travel, you only get self-consistent outcomes, and a lot of the classic portrayals of using time travel to destroy the universe through paradoxical inputs wouldn’t work, because only self-consistent outcomes are allowed, and would almost certainly be prevented beforehand.
The biggest uncertainty here is how much acausal trade lets us substitute for the vast distances that make traditional causal governance impossible.
For those unaware of acausal trade, it’s basically replacing direct communication for predicting what each other wants, and if you have the ability to do vast amounts of simulations, you can get very, very good predictive models of what the other wants such that both of you can trade without requiring any communication, which is necessary for realistic galactic empires/singletons to exist:
https://www.lesswrong.com/w/acausal-trade
I don’t have much of an opinion on the question, but if it’s true that acausal trade can basically substitute wholly for communication that is traditionally necessary to suppress rebellions in empires, then most galactic/universe scale risks are pretty easily avoidable because we don’t have to roll the dice on every civilization trying to do it’s own research that may lead to x-risk.
The main unremovable advantages of AIs over humans will probably be in the following 2 areas:
-
A serial speed advantage, from 50-1000x, with my median in the 100-500x speed advantage range, and more generally the ability to run slower or faster to do more work proportionally, albeit there are tradeoffs at either extreme of either running slow or fast.
-
The ability for compute/software improvements to directly convert into more researchers with essentially 0 serial time necessary, unlike basically all of reproduction (about the only cases where it even gets close are the days/hours doubling time of flies and some bacteria/viruses, but these are doing much simpler jobs and it’s uncertain whether you could add more compute/learning capability without slowing down their doubling time.)
This is the mechanism by which you can get way more AI researchers very fast, while human researchers don’t increase proportionally.
Humans probably do benefit assuming AI is useful enough to automate say AI research away, but these 2 unremovable limitations fundamentally prevent anything like an explosion in research, unlike AI research.
-
I’m trying to identify why the trend has lasted, so that we can predict when the trend will break down.
That was the purpose of my comment.
Nice write-up on the issue.
One thing I will say is that I’m maybe unusually optimistic on power concentration compared to a lot of EAs/LWers, and the main divergence I have is that I basically treat this counter-argument as decisive enough to make me think the risk of power-concentration doesn’t go through, even in scenarios where humanity is basically as careless as possible.
This is due to evidence on human utility functions showing that most people have diminishing returns on utility on exclusive goods to use personally that are fast enough that altruism matters much more than their selfish desires on stellar/galaxy wide scales, combined with me being a relatively big believer in quite a few risks like suffering risks being very cheap to solve via moral trade where most humans are apathetic on.
More generally, I’ve become mostly convinced of the idea that a crucial positive consideration on any post-AGI/ASI future is that it’s really, really easy to prevent most of the worst things that can happen in those futures under a broad array of values, even if moral objectivism/moral realism is false and there isn’t much convergence on values amongst the broad population.