Researcher at the Center on Long-Term Risk. All opinions my own.
Anthony DiGiovanni đ¸
Karma: 2,577
AnÂnouncÂing the Safe Pareto ImÂproveÂments (SPI) FunÂdaÂmenÂtals Program
Hey Ben â your understanding is correct. Option 2 is meant to allow for the possibility that the inference from P1-P3 to Conclusion, as stated in my summary post, is logically invalid. (Of course I think thatâs unlikely, but philosophy can be subtle.) Does that clear things up?
AMA: AnÂthony DiGioÂvanni, auÂthor of the âChallenge of UnawareÂnessâ sequence
Hi Toby, glad it was helpful!
Fair question â the idea is:
P1 claims that c-preferences can only be normatively justified by arguing that one option has better âexpectedâ consequences.
The objections youâre referring to are all implicitly of the form, âWe should always c-prefer either A or B (or be indifferent) because of [some reason other than a comparison of the âexpectedâ consequences].â
E.g. in the first row, [some reason other than a comparison of the âexpectedâ consequences] = âwe always have to choose somethingâ.
Iâll add a note on this to the post itself.
(Due to time constraints I expect I can only give brief replies/âclarifications, going forward. I hope a full read of the sequence will suffice, though I realize itâs quite long, sorry!)
But you donât need to restrict yourself to concerns about the whole future lightcone to run into this problemâat the foundational level this is true of every statement. ⌠I donât see why we should do so with credences (which are of course usually non-foundational statements)
(See my last para for the âfuture lightconeâ thing.)
I donât understand your Munchausen trilemma argument yet. You say credences are âof course usually non-foundationalâ. Agreed! Thatâs exactly why I think our choices of credences require deeper justification. (Whereas foundational things, like Huemerâs âseemingsâ, donât.[1])
forecasting 0.1234567% chance of rain if the extra precision was actually decision-relevant
The extra precision might be âdecision-relevantâ in the sense that: if you were justified in a credence of 0.1234567% + 0.0000001%, you should choose A, and if you were justified in a credence of 0.1234567% â 0.0000001%, you should choose B. But the whole question is why weâd be justified in the former vs. the latter, epistemically. (âI need to make a choiceâ isnât a justification for any particular option you choose.)
Your counterpoint seems to be that in some cases that feel sort-of- equal (and about which, in the cases you describe we actually have a lot of information), we might be inclined to give equal credence.
Thatâs not what Iâm saying, sorry â Iâm denying we should give equal credence. Please see my reply to a similar comment here, and section 3.2.1 and 4.1.1 of the sequence (you might need to CTRL+F some terms defined earlier in the sequence). If itâs still unclear, Iâm happy to try to explain further if you could point to particular passages that need clarification.
The precise EV approach is well evidenced in short-term decision-making
I donât know what exactly this means. If you mean âwe seem to be justified in using precise EVs in short term decision makingâ:
I think our beliefs shouldnât be precise in basically any real-world case, not just beliefs about the far future. (Sec 2.2)
So I think whatâs going on is simply that short term decisions arenât sensitive to the imprecision in the beliefs weâre actually justified in having. The principled difference from the far future case is that in the latter, our decisions are sensitive to the imprecision.
- ^
That is, they donât require deeper justification prima facie. Theyâre still defeasible.
Hi Arepo, thanks for sharing your cruxes here.
The argument I give against assigning precise credences is that itâs arbitrary â literally, you pick one precise credence over many others for no reason. To me, âyou have no reason to do this thingâ is a pretty darn strong argument. :) (ETA: I like the intuition pump in this very short post, if it helps.)
doesnât say why this means we shouldnât/âcanât pick credences according to our best effort.
Why does âour best effortâ need to be precise? Can you say more what exactly you mean? (If the intuition is that more precision = more information, I address that in the post.)
It also doesnât say why, if we can measure short term value, we shouldnât use that as a justification for our decisionmaking process and assume EV from events that we donât think we can assess is 0.
I address this in the unawareness sequence. I recommend reading the table in my summary post â the row with âEven if our impact is dominated by consequences weâre unaware of...â â for the high-level idea, and the links therein for details.
especially when weâre not given an alternative
Isnât this privileging the hypothesis? My claim is that we donât have a positive argument in favor of doing what the precise EV approach recommends (or fuzzier âbest guessesâ, either). If our best defense of that approach is âwhat else is there?â, that seems rather damning.
Ah, sorry, I thought you were making the first-order wager argument (Q3 here), but IIUC youâre making a metanormative wager argument as Toby suggested. I discuss why Iâm unconvinced of that here. (And as another commenter pointed out, this is supplemented by âWhy cluelessness mattersâ in the OP.)
taking your probability-weighted expectation of that range
I think youâre misunderstanding the framework. The whole problem is that we canât assign a (non-arbitrary) âprobability-weighted expectationâ. Thatâs the motivation for representing with a range rather than a single expectation.
(ETA: By default I plan not to reply further.)
I address this objection here (Q3), if I understand what youâre saying correctly. (Iâd recommend first reading sec. 2.1 of the post for crucial background on the epistemology, though, as I noted in another comment.)
(In general, I think you should not expect this post to be a self-contained explanation of the argument by any means. Itâs a high-level summary.)
Iâm not saying we have information that updates us in a particular direction about the bias. Iâm saying we have information suggesting various different directions, and itâs ambiguous what the update should be overall â which is fundamentally different from âno updateâ. I strongly recommend reading sections 2.1 and 2.4 of this post, as well as 3.2.1 of this post, to understand the epistemology thatâs at play here.
(ETA: The final paragraphs of sec 4.1.1, linked right after the part you quoted, also discuss this point.)
(Edited for tone)
Sorry, I donât understand. The snippet I quoted â about acausal stuff and simulations â is whatâs at issue in this discussion.
Regardless, Iâm still interested in where you object to my response to Extrapolation. Could you please say more on that?
they seem to argue more for âweâre clueless about how much we should do ECLâ?
I think they suggest that thereâs just a lot of subtlety in working out the implications of acausal decision theories in practice. Which is reason to expect more crucial considerations in this domain generally /â reason to doubt your âby definitionâ argument.
but why should I expect their attempts to do so to backfire
Why should you expect them to be positive in expectation either? (The broader point of the unawareness sequence is that thereâs an ambiguous pile of positive and negative effects to weigh up.)
On the simulators, it just seems like its hard to think of possible simulator-motivations where us reaching good outcomes in the simulation would be bad for the base reality, and easy to think of ones where it would be neutral or good.
But then weâre back to the Extrapolation argument, which you claimed you werenât committed to. Iâm saying, even if the balance of effects we can think of looks good, weâre looking at a super tiny sliver of the set of effects our fully aware selves would be weighing up â and itâs a biased sample of such effects, so extrapolating from that sample is dubious.
Hmm, these arguments seem too anchored on what we happen to currently be aware of.
I think weâre very confused about the theory of acausal control /â what âsimilarityâ is, and should expect lots of crucial considerations to come up there. (Some relevant writings on this here and here.) As one example: If you try to do something cooperative because this gives evidence (or logically causes) that other agents try to do good things for your values, their attempts to help your values might backfire.
I donât see what the basis is for inferring âitâs probably good for their reality too if we reach a good outcome in the simulationâ. Why would we think we can understand their motivations? Theyâd be alien intelligences.
Most of our impact comes from acausal effects, and effects on the base reality if we are in a simulation: Iâm confused here like everyone else, but I currently donât buy this as a major factor because we only know our reality, and therefore the same things that are good here should naively also have good acausal effects in expectation.
If I understand correctly, this is the âExtrapolationâ response to unawareness I discuss here. What do you think of my response?
ClueÂlessÂness: SumÂmary of the arÂguÂment, why it matÂters, and counterarguments
Linkpost: Bracketing violates the sure-thing principle
Iâd been meaning to turn this into a proper post, but didnât get around to it. So Iâve just linked to this Google doc, which shows that the âbracketingâ decision rule violates a version of the sure-thing principle. I personally donât think this is actually much of a problem (see the conclusion of this post, for one), but it might be of interest.
Oh, I donât think the worry hinges on particular infohazards that arenât public in EA. Iâm thinking of a pretty general problem like: âThe value of the future from the perspective of your altruistic values, epistemology, and decision theory upon reflection is probably a non-monotonic function of how much you increase wisdom etc. broadly. More âwisdomâ or knowledge for actors who are misaligned with you can be quite bad.â And this is at least somewhat borne out by examples like AI movement building, biorisk, and technological progress making factory farming worse.
Cool! For other readers, I think the most relevant sections of the sequence to your question here are 4.1.3 âMeta-extrapolationâ and 4.1.6 âCapacity-buildingâ. They donât go into much concrete detail on backfire risks of âpromoting wisdom, cooperativeness, knowledge, etcâ. But yeah, mostly stuff like infohazards and dual use, plus the unknown unknown downsides we should expect from pessimistic induction. The idea is that:
The historical mechanisms by which promoting wisdom/âcooperativeness/âknowledge made things better occurred in the context of fairly non-weird human socioeconomics. AI takeoff and space colonization are much weirder contexts.
Even if the downsides seem unlikely in absolute terms, the intended upsides from promoting wisdom etc. are so indirect that I think we should also consider them similarly unlikely.
(More controversial, probably: On #6, IDK, the abstraction of general âintelligenceâ seems too coarse to me. LLMsâ (and humansâ!) capability profile seems to depend on a lot more domain-specific fiddly things than the intelligence explosion argument suggests. But Iâd be interested in evidence otherwise. ETA: Put differently, I basically co-sign this post.)
Hmm yeah maybe I shouldnât have fully endorsed Benâs summary. I think many forms of bracketing are impartial in the sense that they donât arbitrarily favor some moral patients over others. But the forms of bracketing Iâm aware of are either (1) not âimpartialâ in the sense that some moral patients/âconsequences are bracketed out for not very well-motivated reasons, or (2) not action-guiding.
That latter definition of impartial might be confusing though, so in general Iâd just list my specific dissatisfactions with different forms of bracketing, âimpartialityâ aside.