yefreitor

Karma: 163

yefreitor Apr 22, 2023, 9:05 PM
2 points
0 ∶ 0
in reply to: Greg_Colbourn ⏸️ ’s comment on: If your AGI x-risk estimates are low, what scenarios make up the bulk of your expectations for an OK outcome?
I expect that
- Riding transformer scaling laws all the way to the end of the internet still only gets you something at most moderately superhuman. This would be civilization-of-immortal-geniuses dangerous, but not angry-alien-god dangerous: MAD is still in effect, for instance. No nanotech, no psychohistory.
- In particular, they won’t be smart enough to determine whether an alternative architecture can go Foom a priori.
- Foom candidates will not be many orders of magnitude cheaper to train than mature language models
and that as a result the marginal return on trying to go Foom will be zero. If it happens, it’ll be the result of deliberate effort by an agent with lots and lots of slack to burn, not something that accidentally falls out of market dynamics.
We’ve had ~50 years of software development so far and gone from 0 to GPT-4.
and a 10,000,000-fold increase in transistor density. We might return to 20th century compute cost improvements for a bit, if things get really really cheap, but it’s not going to move anywhere near as fast as software.

yefreitor Apr 22, 2023, 1:29 PM
1 point
0 ∶ 0
in reply to: Greg_Colbourn ⏸️ ’s comment on: If your AGI x-risk estimates are low, what scenarios make up the bulk of your expectations for an OK outcome?
By doom I mean there are no humans left. How does your scenario not lead to that eventually?
Eventually, it probably does. But the same is true of our current trajectory. I think the sort of AI we seem likely to get first raises the short run p(doom) somewhat, but primarily by intensifying existing risks.
Silicon Moloch is likely to foom, surely? (Cf. the “rapid dose of history” that will happen.)
In the limit of maximally intense competition (which we won’t see, but might approach), probably not. We’d get a thousand years worth of incremental software improvements in a decade and a (slower) takeoff in chip production, but not runaway technological progress: in principle the risk free rate will eventually drop to a rate where basic research becomes economical, but I expect the ascended economy will go the way of all feedback loops and undermine its own preconditions long before that.
What sort of civilization comes out the other end is anyone’s guess, but I doubt it’ll be less equipped to protect itself than we are.

yefreitor Apr 22, 2023, 12:40 AM
8 points
5 ∶ 0
on: If your AGI x-risk estimates are low, what scenarios make up the bulk of your expectations for an OK outcome?
Depends on what you mean by doom: I don’t think we’re getting paperclipped, but I don’t think things will be ok by default either.
I expect early AI agents to be purpose-built ensembles of bleeding-edge predictive models glued together by a bunch of crude hacks. Whatever gestalt entity emerges out of their internal dynamics is going to be only weakly agential: more like a corporation than a human being.
An individual agent like this could not defeat all of us together. A competitive ecosystem of them, however, can and will defeat all of us separately—Moloch made silicon. Most of us are going to make it, at least in the short term, but the window in which we might have steered human civilization towards human ends will shut forever.

yefreitor Apr 13, 2023, 9:55 AM
1 point
0 ∶ 0
on: Evolution provides no evidence for the sharp left turn
It seems to me that humans have had (at least) two sharp left turns:
- behavioral modernity: humans rapidly (compared to the pace of early human evolution) become vastly more capable than other hominins
- the neolithic revolution and everything after: complex agricultural societies rapidly (compared to the pace of early human cultural change) become vastly more capable than other human groups
You’ve offered a plausible account of the development of behavioral modernity—but why do we get another qualitative jump in capabilities once we get enough behaviorally modern humans together? I suspect the answer hinges more on high-level facts about human intelligence than the specific mechanisms that brought it about. If so, then worrying about analogous developments in AI is appropriate.

yefreitor Apr 11, 2023, 4:08 AM
21 points
3 ∶ 3
in reply to: Geoffrey Miller’s comment on: Nobody’s on the ball on AGI alignment
I agree that the very strong sort of alignment you describe—with the Coherent Extrapolated Volition of humanity, or the collective interest of all sentient beings, or The Form of The Good—is probably impossible and perhaps ill-posed. Insofar as we need this sort of aligned AI for things to go as well as they possibly could, they won’t.
But I don’t see why that’s the only acceptable target. Aligning a superintelligence with the will of basically any psychologically normal human being (narrower than any realistic target except perhaps a profit-maximizer—in which case yeah, we’re doomed) would still be an ok outcome for humans: it certainly doesn’t end in paperclips. And alignment with someone even slightly inclined towards impartial benevolence probably goes much better than the status quo, especially for the extremely poor.
(Animals are at much more risk here, but their current situation is also much worse: I’m extremely uncertain how a far richer world would treat factory farming)

yefreitor Apr 11, 2023, 3:06 AM
3 points
0 ∶ 0
in reply to: Jim Buhler’s comment on: What the Moral Truth might be makes no difference to what will happen
I remain skeptical because this would imply the orthogonality thesis is almost completely false.
The orthogonality thesis could be (and I think almost certainly is) false with respect to some agent-generating processes (e.g., natural selection) and true with respect to others (e.g. Q-learning).

yefreitor Apr 11, 2023, 2:56 AM
1 point
0 ∶ 0
in reply to: Jim Buhler’s comment on: What the Moral Truth might be makes no difference to what will happen
All I’m saying is that those who are the most bullish on space colonization will colonize more space.
Sure, but that doesn’t tell you much about what happens afterwards. If the initial colonists’ values are locked in ~forever, we should probably expect value drift to be weak in general, which means frontier selection effects have a lot less variation to work with.
At the extreme lower limit with no drift at all, most agents within a mature civilization are about as expansionist as the most expansionist of the initial colonists—but no more so. And this might not be all that much in the grand scheme of things.
At the other end, where most of the space of possible values gets explored, maybe you do get a shockwave of superintelligent sociopaths racing outwards at relativistic speeds—but you also get a vast interior that favors (relatively speaking) long-term survival and material efficiency.

yefreitor Apr 9, 2023, 11:23 PM
8 points
2 ∶ 0
on: What the Moral Truth might be makes no difference to what will happen
Several objections in no particular order
- I think your first section is correct, but you conclude far too much from it: failing to act in perfect accord with the moral truth does not mean you’re not influenced by it at all. Humans fail your conditions 4-7 and yet are occasionally influenced by moral facts in ways that matter.
- On the one hand, you assume that civilizations are agents which can simply decide to adopt this or that strategy; on the other hand, you expect intense selection within civilizations, such that their members behave so as to maximize their own reproductive success. But these can’t both be true: you can throw all of your surplus into expanding as fast as possible, or you can spend it on internal competition, or you can do something in between, but you can’t spend it all on both.
- I don’t think
  
  > Within a civilization, we should expect the agents who have the values that are the most adapted/competitive to survival, replication, and expansion to eventually be selected for
  
  is the right conclusion to draw from Hanson’s paper. The selection effects he’s talking about act on different regions of the frontier of a given civilization. Those living in the interior (who will be the vast majority of the population in the extreme long-run) may be disproportionately descended from fast expanders, but will not face the same pressure themselves.
- There are many different units of selection, and they can’t all be subject to arbitrarily intense selection pressure simultaneously: cancer cells don’t build spaceships.

yefreitor Apr 6, 2023, 2:54 AM
1 point
0 ∶ 0
in reply to: Marcel D’s comment on: The Orthogonality Thesis is Not Obviously True
If yefreitor is saying what I planned to say, the simpler version is just “there’s nothing ‘irrational’ about having a utility function that says ‘no experience matters every Tuesday.’”
Parfit’s position (and mine) is that Future Tuesday Indifference is manifestly irrational. But this has little to do with what sort of preferences sufficiently intelligent agents can have.
No, they literally have no negative (or positive) experience on Tuesdays
No, that’s explicitly ruled out in the setup. They have experiences on Tuesday, those experiences have the usual valence—they just fail to act accordingly. Here’s the full context from Reasons and Persons:
Consider next this imaginary case. A certain hedonist cares greatly about the quality of his future experiences. With one exception, he cares equally about all the parts of his future. The exception is that he has Future-Tuesday-Indifference. Throughout every Tuesday he cares in the normal way about what is happening to him. But he never cares about possible pains or pleasures on a future Tuesday. Thus he would choose a painful operation on the following Tuesday rather than a much less painful operation on the following Wednesday. This choice would not be the result of any false beliefs. This man knows that the operation will be much more painful if it is on Tuesday. Nor does
he have false beliefs about personal identity. He agrees that it will be just as much him who will be suffering on Tuesday. Nor does he have false beliefs about time. He knows that Tuesday is merely part of a conventional calendar, with an arbitrary name taken from a false religion. Nor has he any other beliefs that might help to justify his indifference to pain on future Tuesdays. This indifference is a bare fact. When he is planning his future, it is simply true that he always prefers the prospect of great suffering on a Tuesday to the mildest pain on any other day.
This man’s pattern of concern is irrational. Why does he prefer agony on Tuesday to mild pain on any other day? Simply because the agony will be on a Tuesday. This is no reason. If someone must choose between suffering agony on Tuesday or mild pain on Wednesday, the fact that the agony will be on a Tuesday is no reason for preferring it. Preferring the worse of two pains, for no reason, is irrational.

yefreitor Apr 5, 2023, 10:46 PM
6 points
2 ∶ 0
on: The Orthogonality Thesis is Not Obviously True
It seems that, if this ‘certain hedonist’ were really fully rational, they would start caring about their pleasures and pains equally across days. They would recognize that the day of the week does not matter to the badness of their pains. Thus, in a similar sense, if something is 10,000,000 times smarter than Von Neumann, and can think hundreds of thousands of pages worth of thoughts in the span of minutes, it would conclude that pleasure is worth pursuing and paperclips are not.
This argument is really straightforward. If moral realism were true, then if something became super smart, so too would it realize that some things were worth pursuing.
This is not what Parfit is arguing. The Future Tuesday Indifference thought experiment is part of Parfit’s defense of irreducible normativity:
- if the subjectivist position that acting rationally is reducible to acting in accordance with some consistent set of beliefs and preferences is true, then Future Tuesday Indifference is rational
- Future Tuesday Indifference is irrational
- So the subjectivist position is false
Our certain hedonist’s problem is not an epistemic one: it’s not that they don’t know what pain is like on Tuesday, or that they’re not smart enough to realize that Tuesday is an arbitrary label and in truth days are simply days, or that they’ve failed to reach reflective equilibrium. They’re acting in perfect accord with their coherent extrapolated volition—the problem is just that it’s a ridiculous thing to want.
Assuming that a sufficiently intelligent agent would necessarily be rational in this sense to argue against the orthogonality thesis is circular.

yefreitor Apr 3, 2023, 5:13 AM
1 point
0 ∶ 0
in reply to: titotal’s comment on: Orthogonality is Expensive
Your definition is trivially true: all it requires is that an AGI having a specified goal is not physically impossible. But that doesn’t prove that all goals are equally likely to occur, or even that AGI will have “goals” at all.
Yes, of course (hence the footnote).
The way I see it deployed in practice is to say that a “dumb” AI will have some silly goal like “build squiggles”, will go through an intelligence scale-up, and will keep that goal in hyper-intelligent form. (and then pursuing that goal will result in disaster).
My reading of the doomer view (which I don’t necessarily endorse) is quite different: a dumb AI starts with some useful goal, goes through an intelligence scale-up that slightly perturbs its goal in some direction—and because goals compatible with human life are a tiny thread winding their way through a stupidly high-dimensional manifold of all possible goals, ends up misaligned by default.
This doesn’t especially hinge on whether these perturbations can be in any direction or only a few (as is the case if goals are strongly constrained by architecture), except in the case where they run only along the human-survival curve. Any transverse component whatsoever means you get pushed off-manifold almost always. And this is plausible (I think) only in the case where human values are not a tiny golden thread, but actually rather large and fuzzily full-dimensional.

yefreitor Apr 3, 2023, 3:02 AM
1 point
0 ∶ 0
on: Orthogonality is Expensive
From the linked post
A common assumption about AGI is the orthogonality thesis, which argues that goals/utility functions and the core intelligence of an AGI system are orthogonal or can be cleanly factored apart.
This may or may not be an “orthogonality thesis” (I haven’t seen this usage before, but I also haven’t looked for it), but the orthogonality thesis I’m familiar with has the quantifiers the other way around: for any goal, there exists a possible AGI that will attempt carry it out.^[1] Even if mixing-and-matching a dumb industrial control system with a smart friendly AI is safe, that doesn’t mean that a smart industrial control system arrived at through some other route won’t paperclip you.
1. ^
  Though in practice what people making likely-doom arguments really mean is that a generic human-created AI will not have recognizably human values, which is a somewhat stronger claim.

yefreitor Mar 23, 2023, 1:56 AM
3 points
0 ∶ 0
on: Common-sense sentience is dubious
I’ve been told that this essay supports the position of ‘illusionism’.
I don’t think this is correct. The illusionist position is that the only actual explananda are our beliefs about phenomenal properties (or rather, the brain-states responsible for utterances like “pain is bad”, since beliefs are another folk-psychological concept supposedly ripe for elimination): once those are accounted for, there will be nothing left to explain and no reason to continue believing in qualia.
For an attack on the reliability of our common-sense beliefs about qualia to constitute a defense of illusionism, we would have to think that our belief in qualia would be unjustified were it not for those intuitions. But this is begging the question—if phenomenal properties do exist and we do have first-hand access to them, then that, not common-sense, is our principal justification for believing in them.