Error

Unrecognized LW server error:

Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?

Richard Y Chappell🔸Apr 5, 2023, 11:06 PM
6 points
1 ∶ 0

Distinguish substantive vs procedural rationality. Procedural rationality = following neutrally describable processes like “considering competing evidence”, “avoiding incoherence”, etc. -- roughly corresponding to intelligence. Substantive rationality = responding correctly to normative reasons—roughly corresponding to wisdom.
The Parfitian view is that the two come apart (i.e., supporting the orthogonality thesis). Future Tuesday Indifference may be procedurally rational, or compatible with perfect intelligence, and yet still objectively crazy (unwise). More generally, there’s nothing in non-naturalist moral realism that implies that intelligent agents per se are likely to converge on the normative truth. (I discuss this more in Knowing What Matters. I think we can reasonably take ourselves to be on the right track, but that’s because of our substantive starting points, not the mere fact of our general intelligence.)
- Bentham's Bulldog Apr 6, 2023, 12:41 AM
  1 point
  0 ∶ 0
  Parent
  
  Re substantive vs procedural rationality, procedural rationality just seems roughly like instrumental rationality. For the reasons I explain, I’d expect AI to be rational in general, not just instrumentally so. Do you think ignorance of the modal facts would be possible for an arbitrarily smart agent? I’d think the moral facts would be like the modal facts in that they’d figure them out. I think that when we are smart we can figure things out and they are more likely to be true. The reason I believe modal rationalism, for example, is that there is some sense in which I feel I’ve grasped it, which wouldn’t be possible if I were much less smart.
  - Richard Y Chappell🔸Apr 6, 2023, 3:52 AM
    3 points
    0 ∶ 0
    Parent
    
    Depends whether procedural rationality suffices for modal knowledge (e.g. if false modal views are ultimately incoherent; false moral views certainly don’t seem incoherent).
    Smartness might be necessary for substantive insights, but doesn’t seem sufficient. There are plenty of smart philosophers with substantively misguided views, after all.
    A metaphor: think of belief space as a giant spider web, with no single center, but instead a large number of such “central” clusters, each representing a maximally internally coherent and defensible set of beliefs. We start off somewhere in this web. Reasoning leads us along a strand, typically in the direction of greater coherence—i.e., towards a cluster. But if the clusters are not differentiated in any neutrally-recognizable way—the truths do not glow in a way that sets them apart from ideally coherent falsehoods—then there’s no guarantee that philosophical reasoning (or “intelligence”) will lead you to the truth. All it can do is lead you towards greater coherence.
    That’s still worth pursuing, because the truth sure isn’t going to be somewhere incoherent. But it seems likely that from most possible starting points (e.g. if chosen arbitrarily), the truth would be forever inaccessible.
    - Bentham's Bulldog Apr 6, 2023, 2:04 PM
      1 point
      0 ∶ 0
      Parent
      
      I think I just disagree about what reasoning is. I think that reasoning does not just make our existing beliefs more coherent, but allows us to grasp new deep truths. For example, I think that an anti-realist who didn’t originally have the FTI irrational intuition could grasp it by reflection, and that one can, over time, discover that some things are just not worth pursuing and others are.
yefreitor Apr 5, 2023, 10:46 PM
6 points
2 ∶ 0

It seems that, if this ‘certain hedonist’ were really fully rational, they would start caring about their pleasures and pains equally across days. They would recognize that the day of the week does not matter to the badness of their pains. Thus, in a similar sense, if something is 10,000,000 times smarter than Von Neumann, and can think hundreds of thousands of pages worth of thoughts in the span of minutes, it would conclude that pleasure is worth pursuing and paperclips are not.
This argument is really straightforward. If moral realism were true, then if something became super smart, so too would it realize that some things were worth pursuing.
This is not what Parfit is arguing. The Future Tuesday Indifference thought experiment is part of Parfit’s defense of irreducible normativity:
- if the subjectivist position that acting rationally is reducible to acting in accordance with some consistent set of beliefs and preferences is true, then Future Tuesday Indifference is rational
- Future Tuesday Indifference is irrational
- So the subjectivist position is false
Our certain hedonist’s problem is not an epistemic one: it’s not that they don’t know what pain is like on Tuesday, or that they’re not smart enough to realize that Tuesday is an arbitrary label and in truth days are simply days, or that they’ve failed to reach reflective equilibrium. They’re acting in perfect accord with their coherent extrapolated volition—the problem is just that it’s a ridiculous thing to want.
Assuming that a sufficiently intelligent agent would necessarily be rational in this sense to argue against the orthogonality thesis is circular.
- Bentham's Bulldog Apr 6, 2023, 12:42 AM
  1 point
  0 ∶ 0
  Parent
  
  I agree with everything you’ve said after the sentence “This is not what Parfit is arguing.” But how does that conflict with the things I said?
  - Marcel D Apr 6, 2023, 2:38 AM
    2 points
    0 ∶ 0
    Parent
    
    If yefreitor is saying what I planned to say, the simpler version is just “there’s nothing ‘irrational’ about having a utility function that says ‘no experience matters every Tuesday.’” It certainly wouldn’t seem to be a good instrumental value, but if that’s your terminal value function that’s what it is.
    
    They would recognize that the day of the week does not matter to the badness of their pains.
    
    No, they literally have no negative (or positive) experience on Tuesdays, unless the experience on Tuesdays affects their experience on different days.
    
    it would begin pursuing what is objectively worth pursuing.
    
    ??? “Objectively worth pursuing?” Where did that come from? Certainly not a Tuesday-impartial utility function, which is the only “objective” thing I’m seeing here? I didn’t see where you clearly explain this through a short ctrl+f for “objective.”
    - Bentham's Bulldog Apr 6, 2023, 2:52 AM
      2 points
      1 ∶ 0
      Parent
      
      I agree one could have that value in theory. My claim is that if one were very rational, they would not. Note that, contrary to your indication, they do have experience on Tuesday, and their suffering feels just as bad on a Tuesday as on another day. They just have a higher order indifference to future suffering. I claim that what is objectively worth pursuing is indifferent to the day of the week.
    - yefreitor Apr 6, 2023, 2:54 AM
      1 point
      0 ∶ 0
      Parent
      
      If yefreitor is saying what I planned to say, the simpler version is just “there’s nothing ‘irrational’ about having a utility function that says ‘no experience matters every Tuesday.’”
      Parfit’s position (and mine) is that Future Tuesday Indifference is manifestly irrational. But this has little to do with what sort of preferences sufficiently intelligent agents can have.
      No, they literally have no negative (or positive) experience on Tuesdays
      No, that’s explicitly ruled out in the setup. They have experiences on Tuesday, those experiences have the usual valence—they just fail to act accordingly. Here’s the full context from Reasons and Persons:
      Consider next this imaginary case. A certain hedonist cares greatly about the quality of his future experiences. With one exception, he cares equally about all the parts of his future. The exception is that he has Future-Tuesday-Indifference. Throughout every Tuesday he cares in the normal way about what is happening to him. But he never cares about possible pains or pleasures on a future Tuesday. Thus he would choose a painful operation on the following Tuesday rather than a much less painful operation on the following Wednesday. This choice would not be the result of any false beliefs. This man knows that the operation will be much more painful if it is on Tuesday. Nor does
      he have false beliefs about personal identity. He agrees that it will be just as much him who will be suffering on Tuesday. Nor does he have false beliefs about time. He knows that Tuesday is merely part of a conventional calendar, with an arbitrary name taken from a false religion. Nor has he any other beliefs that might help to justify his indifference to pain on future Tuesdays. This indifference is a bare fact. When he is planning his future, it is simply true that he always prefers the prospect of great suffering on a Tuesday to the mildest pain on any other day.
      This man’s pattern of concern is irrational. Why does he prefer agony on Tuesday to mild pain on any other day? Simply because the agony will be on a Tuesday. This is no reason. If someone must choose between suffering agony on Tuesday or mild pain on Wednesday, the fact that the agony will be on a Tuesday is no reason for preferring it. Preferring the worse of two pains, for no reason, is irrational.
      - Bentham's Bulldog Apr 6, 2023, 3:50 AM
        1 point
        0 ∶ 0
        Parent
        
        I think our disagreement is that I think that superintelligences would be rational and avoid FTI for the same reason they’d be epistemically rational and good at reasoning in general.
Brendan Mooney Apr 6, 2023, 6:38 AM
1 point
0 ∶ 0

Three things:
more or less any level of intelligence could in principle be combined with more or less any final goal.
This seems to me kind of a weird statement of the thesis, “could in principle” being too weak.
If I understand, you’re not actually denying that just about any combination of intelligence and values could in principle occur. As you said, we can take a fact like the truth of evolution and imagine an extremely smart being that’s wrong about that specific thing. There’s no obvious impossibility there. It seems like the same would go for basically any fact or set of facts, normative or not.
I take it the real issue is one of probability, not possibility. Is an extremely smart being likely to accept what seem like glaringly obvious moral truths (like “you shouldn’t turn everyone into paperclips”) in virtue of being so smart?
(2) I was surprised to see you say your case depended completely on moral realism. Of course, if you’re a realist, it makes some sense to approach things that way. Use your background knowledge, right?
But I think even an anti-realist may still be able to answer yes to the question above, depending on how the being in question is constructed. For example, I think something in this anti-orthogonality vein is true of humans. They tend to be constructed so that understanding of certain non-normative facts puts pressure on certain values or normative views: If you improve a human’s ability to imaginatively simulate the experience of living in slavery (a non-moral intellectual achievement), they will be less likely to support slavery, and so on.
This is one direction I kind of expected you to go at some point after I saw the Aaronson quote mention “the practical version” of the thesis. That phrase has a flavor of, “Even if the thesis is mostly true because there are no moral facts to discover, it might still be false enough to save humanity.”
(3) But perhaps the more obvious the truths, the less intelligence matters. The claim about slavery is clearer to me than the claim that learning more about turning everyone into paperclips would make a person less likely to do so. It seems hard to imagine a person so ignorant as to not already appreciate all the morally relevant facts about turning people into paperclips. It’s as if, when the moral questions get so basic, intelligence isn’t going to make a difference. You’ve either got the values or you don’t. (But I’m a committed anti-realist, and I’m not sure how much that’s coloring these last comments.)
- Bentham's Bulldog Apr 6, 2023, 2:00 PM
  1 point
  0 ∶ 0
  Parent
  
  I think 1 is right.
  2 I agree that it would depend on how the being is constructed. My claim is that it’s plausible that they’d be moral by default just by virtue of being smart.
  3 I think there is a sense in which I have—and most modern people have—unlike most people historically, grasped the badness of slavery.