Researcher at the Center on Long-Term Risk. All opinions my own.
Anthony DiGiovanni đ¸
Karma: 2,413
Cool! For other readers, I think the most relevant sections of the sequence to your question here are 4.1.3 âMeta-extrapolationâ and 4.1.6 âCapacity-buildingâ. They donât go into much concrete detail on backfire risks of âpromoting wisdom, cooperativeness, knowledge, etcâ. But yeah, mostly stuff like infohazards and dual use, plus the unknown unknown downsides we should expect from pessimistic induction. The idea is that:
The historical mechanisms by which promoting wisdom/âcooperativeness/âknowledge made things better occurred in the context of fairly non-weird human socioeconomics. AI takeoff and space colonization are much weirder contexts.
Even if the downsides seem unlikely in absolute terms, the intended upsides from promoting wisdom etc. are so indirect that I think we should also consider them similarly unlikely.
(More controversial, probably: On #6, IDK, the abstraction of general âintelligenceâ seems too coarse to me. LLMsâ (and humansâ!) capability profile seems to depend on a lot more domain-specific fiddly things than the intelligence explosion argument suggests. But Iâd be interested in evidence otherwise. ETA: Put differently, I basically co-sign this post.)
The ordering of #7-#9 at least seems debatable to me, because many aspects of (e.g.) natural sciences and medical breakthroughs seem bottlenecked on things other than remote-only labor. Like physical understanding and dexterity (for the scientific labor), and feedback from the physical world (for medical breakthroughs).
Thanks! I do something like this in post #2, sec. 2.3. This first post is just meant to set up the problem at a high level. Interested to hear your thoughts on those examples.
(See also hereâthese arenât âreally weirdâ, but they do seem like pretty plausible backfire mechanisms to me.)
I should have played the game more. I thought BOTECs for alignment work were dumb â how would estimating differential x-risk reduction basis points mean anything? I figured smart funders would just get the obvious argument around reducing elicitation overhang. They didnât, and ego stopped me from making up the numbers anyway. Probably should have.
I think it would be a shame if you took away this lesson. A perfectly good reason to not make up numbers, that has nothing to do with ego, is: âMade-up numbers are arbitrary! Thereâs no reason to think these numbers are calibrated with enough precision to be action-guiding, or at least, as action-guiding as people often take them to be.â
(My gripes discussed in the linked post above, FWIW: Re: (1), the typical EA operationalization of âwell-calibratedâ, and judgments about how to defer to people based on their calibration on some reference class of past questions, are based on very questionable epistemological assumptions. See also this great post.)
When talking about forecasting, people often ask questions like âHow can we leverage forecasting into better decisions?â This is the wrong way to go about solving problems. You solve problems by starting with the problem, and then you see which tools are useful for solving it.
I definitely have my own gripes about EA/ârationalist attitudes towards forecasting (see here), but maybe your objection is a level confusion:
I think when people talk about âleveraging forecasting into better decisionsâ, theyâre saying: ââBetterâ decisions just are decisions guided by the normatively correct beliefs. Namely, theyâre decisions that make reasonable-seeming tradeoffs between possible outcomes given the normatively correct beliefs about the plausibility of those outcomes. So our decisions will be more aligned with this standard of âbetterâ if our beliefs are formed by deferring to well-calibrated forecasts.â
E.g. theyâre saying, âWhen navigating AI risk, weâll make decisions that we endorse more if those decisions are guided by the credences of folks whoâve been unusually successful at forecasting AI developments.â
(At least, thatâs the steelman. Maybe Iâm being too charitable!)
Whereas you seem to be asking something like: âWe already know which beliefs are reasonable. Do these beliefs tell us that âplug forecasts into some decision-making procedureâ seems likely to lead to good outcomes (i.e., that this is a âuseful toolâ)?â
CLRâs Safe Pareto ImÂproveÂments ReÂsearch Agenda
I meant my future decisions would be the same in reality if I could not gather additional evidence
Perhaps, but thatâs consistent with incomparability. Given the independent motivations weâve discussed (/âgiven in my post) for calling the two options incomparable, Iâd say you should call them incomparable.
I think I address your questions in the second paragraph in âWhy weâre especially unaware of large-scale consequencesâ (this post) and âMeta-extrapolationâ (post #4). See also my discussion with Richard here.
I donât know what you mean by âpractically the sameâ, can you say more?
Regardless, the problem is that âgathering evidenceâ vs âdoing something elseâ is itself a decision, whose consequences youâll be clueless about. I discuss this more here.
But we also have strong reasons to trust that some process designed our cooperative instincts to allow groups of humans to cooperate effectively.
â[A]llowing [small] groups of humans to cooperate effectivelyâ is very far from âmaking the far future better, impartially speakingâ. Iâd be interested in your responses to the arguments here.
I also think that many individuals need to decide how to make their lives go well in pretty confusing circumstances. Imagine deciding whether to immigrate to America in the 1700s, or how to live in the shadow of the Cold War, or whether to genetically engineer your children.
First, itâs not clear to me these people werenât clueless â i.e. really had more reason to choose whatever they chose than the alternatives â depending on how long a time horizon they were aiming to make go well.
Second, insofar as we think these peopleâs choices were justified, I donât see why you think their instincts gave them such justification. Why would these instincts track unprecedented consequences so well?
and the net value of the things that itâs done so far may well be dominated by what updates it makes based on that experience
I donât think âmay wellâ gets us very far. Can you say more why this hypothesis is so much more likely than, say, âthe dominant impacts are the damage thatâs already been doneâ, or âthe dominant impacts will come from near-future decisions, made by actors who are still too ignorant about the extremely complex system theyâre intervening inâ?
do you think that, if we had a theory of sociopolitics that was about as good as 20th-century economics, then we wouldnât be clueless about how to do sociopolitical interventions (like founding AI safety movements) effectively?
No, because I think âfounding AI safety movements that succeed at making the far future go betterâ is a pretty out-of-distribution kind of sociopolitical intervention.
Thanks for explaining! To summarize, I think there are crucial disanalogies between the âmake their own life go wellâ case and the âmake the future of humanity go wellâ case:
Should they follow those instincts even when they donât know why evolution instilled them? ⌠(Note that this doesnât rely on them understanding evolutionâeven 1000 years ago they could have trusted that some process designed their body and mind to function well, despite radical uncertainty about what that process was.)
In this case, the reasons to trust âthat some process designed their body and mind to function wellâ are relatively strong. Because of how weâre defining âwellâ: an individualâs survival in a not-too-extreme environment. Even if they donât understand evolution, they can think about how their instincts plausibly wouldâve been honed on feedback relevant to this objective. And/âor look at how other individuals (or their past self) have tended to survive when they trusted these kinds of instincts.[1]
Now consider someone who wants to make the future of humanity go well. Similar, they have certain cooperative instincts ingrained in themâe.g. the instinct towards honesty. All else equal, it seems pretty reasonable to think that following them will help humanity to cooperate better, and that this will allow humanity to avoid internal conflict.
Here, the reasons to trust that the instincts track the objective seem way weaker to me,[2] for all the reasons I discuss here: No feedback loops, radically unfamiliar circumstances due to the advent of ASI and the like, a track record of sign-flipping considerations. All else is really not equal.
You yourself have written about how the AIS movement arguably backfired hard, which I really appreciate! I expect that ex ante, people founding this movement told themselves: âAll else equal, it seems pretty reasonable to think that trying to warn people of a source of x-risk, and encouraging research on how to prevent it, will help humanity avoid that x-risk.â
(I think analogous problems apply to your subagent and forward-chaining framings. Theyâre justified when the larger system provides feedback, or the forward steps have been validated in similar contexts â which weâre missing here.)
How does this relate to cluelessness? Mostly I donât really know what the term means
The way I use the term, youâre clueless about how to compare A vs. B, relative to your values, if: It seems arbitrary (upon reflection) to say A is better than, worse than, or exactly as good as B. And instead it seems we should consider Aâs and Bâs goodness incomparable.
- ^
What if someone has always been totally solitary, doesnât understand evolution or feedback loops, and hasnât made many decisions based on similar instincts? Seems like such a person wouldnât have reasons to trust their instincts! Theyâd just be getting lucky.
- ^
See here for my reply to: âSure, âway weakerâ, but theyâre still slightly better than chance right?â Tl;dr: This doesnât work because the problem isnât just noise that weakens the signal, itâs âitâs ambiguous what the direction of the signal even isâ.
- ^
Thanks â Iâve read both but neither seems to answer my objection.
The stuff on cluelessness feels like itâs conceding a little too much to the EA/âbayesian frame. Itâs implying that you should have a model of the entire future in order to make decisions. But what I think you actually want to claim is that itâs sensible and even ârationalâ to make non-model-based decisions (e.g. via heuristics, intuitions, etc).
Iâd be interested in hearing more on what exactly you mean by this. Insofar as someone wants to make decisions based on impartially altruistic values, I think cluelessness is their problem, even if they donât make decisions by explicitly optimizing w.r.t. a model of the entire future. If such a person appeals to some heuristics or intuitions as justification for their decisions, then (as argued here) they need to say why those heuristics or intuitions reliably track impact on the impartial good. And the case for that looks pretty dubious to me.
(If youâre rejecting the âmake decisions based on impartially altruistic valuesâ step, fair enough, though I think weâd do well to be explicit about that.)
If you had more evidence, you could make the comparison. But you currently have no clue which direction the comparison would go, in expectation over the evidence you might receive. So how are you supposed to compare them right now?
My best guess about which of 2 identical objects has a larger mass in expectation will be arbitrary is their mass only differs by 10^-6 kg, and I have no way of assessing this small difference. However, this does not mean the expected mass of the 2 objects is fundamentally incomparable
I worry youâre reifying âexpectationsâ as something objective here. The relative actual masses of the objects are clearly comparable. But if you subjectively canât compare them, then theyâre indeed incomparable âin expectationâ in the relevant sense.
Besides the links Michael shared, I highly recommend this really short post.
However, the same goes for comparisons among the expected mass of seemingly identical objects with a similar mass if I can only assess their mass using my hands, but this does not mean their mass is incomparable.
I donât exactly understand what argument youâre making here.
My core argument in the post is: Take any intervention X. We want to weigh up its impact for all sentient beings across the cosmos, where this âweighing upâ is aggregation over all hypotheses. Now suppose we want to force ourselves to compare X with inaction, i.e., say either UEV(do X) > UEV(donât do X) or vice versa. We have such an extremely coarse-grained understanding (if any) of these hypotheses[1] that, when we do the weighing-up, whether we say UEV(do X) > UEV(donât do X) or vice versa seems to depend on an arbitrary choice.
Can you say how your argument relates to mine?
- ^
Relative to the amount of fine-grained detail necessary to evaluate the hypothesis, when what we value is âwell-being of all sentient beings across the cosmosâ.
- ^
In normal situations, an agent can rationally come to a single probability distribution, but, Greaves argues that, in a situation with complex cluelessness, an individual should instead have a set of probability functions that they are ârationally required to remain neutral between.â Iâm not entirely sure what this means.
You might be interested in this post I wrote explaining imprecision â hopefully answers âwhat this meansâ.
Oh, I donât think the worry hinges on particular infohazards that arenât public in EA. Iâm thinking of a pretty general problem like: âThe value of the future from the perspective of your altruistic values, epistemology, and decision theory upon reflection is probably a non-monotonic function of how much you increase wisdom etc. broadly. More âwisdomâ or knowledge for actors who are misaligned with you can be quite bad.â And this is at least somewhat borne out by examples like AI movement building, biorisk, and technological progress making factory farming worse.