Whatâs the clearest example of a complex cluelessness sign flip youâre aware of?
(By âclearâ I mean âhad a very narrow confidence interval before encountering some consideration and a narrow interval after encountering that consideration but the CIs now center points with opposite signsâ.[1])
The clearest examples I know of (e.g. rescuing Hitler as a child) seem to me like examples of simple cluelessness. You list some examples here, but they donât seem that clear to me, e.g. I disagree that âEarly awareness-raising about AGI x-risk presumably seemed robustly goodâ and would guess most people involved in that had CIs which comfortably straddled zero.
Hi Ben, I like the spirit of this question, though Iâm not sure itâs the most relevant formulation. Thoughts on that, before I answer your literal question:
To get clear on terms:
Iâm assuming by âCIâ here, you mean something like your precise 90% (or whatever) confidence interval for your idealized selfâs EV of the intervention (as per Premise 1).
By ârobustâ, I donât mean a narrow /â strictly positive CI. I mean that the verdict âthis intervention is positive âin expectationââ isnât sensitive to arbitrary choices about how to factor in the considerations weâre unaware of.
I think itâs fair to say most people involved in early AI risk advocacy considered their work robustly positive in that sense.
This definition of ârobustâ is what matters for Premise 3. Because P3 says, our verdicts about interventions having positive âEVâ are sensitive to such arbitrary choices â even if we admit weâre very uncertain (i.e. we have wide CIs).
So, if weâre asking whether a given âsign flipâ counts as evidence for P3, I donât see why the bar should be ânarrow CIs with opposite-sign center points before and after the considerationâ.
If weâve discovered a consideration that flipped us from âwide CI centered at a positive EVâ to âwide CI centered at a negative EVâ, isnât that some evidence that our initial âpositive in expectationâ verdict wasnât robust, in the sense above? (And hence inductive evidence that our current âpositive in expectationâ verdicts arenât robust (i.e., evidence for P3), as argued here.)
Anyway, Iâd agree that the clearest evidence for P3 would come from sign flips that meet your bar. Maybe the small animal replacement problem? Iâd guess lots of people who care about animal welfare thought that getting people to eat less beef was clearly good before being aware of SARP, and think itâs clearly net-bad after being aware of SARP. (Itâs harder to come by examples of sign flips by your def for longtermist causes, because non-clueless longtermists typically agree that we should be very uncertain about the far future. But per the above, this is to be expected if weâre clueless.)
Thanks! I canât tell if this is cruxy, but for what itâs worth your âpessimal inductionâ vignettes donât resonate with me in a way which makes me less motivated by the unawareness concerns.
For example, Bostrom coined the phrase âattention hazardâ in 2011. I remember someone telling me that MIRI was net-negative for this reason at EAG 2015, and I would be surprised if e.g. Habryka hadnât considered this risk before starting Lightcone. So I disagree with citing him/âthis as a good example of unawareness; itâs more that they mis-estimated a known risk factor.
Similarly, I remember talking about SARP at one of my first EAGs. I think I came across it in Brian Tomasikâs 2007 post, maybe even before I had encountered EA. Perhaps Iâve mis-estimated those concerns, but it doesnât seem like unawareness.
My overall experience is kind of the opposite of yours: when I got involved in EA people talked a lot about âCause Xâ and âCrucial Considerationsâ and now theyâve mostly just⌠stopped? Like people tried to find other considerations, and thereâs some new stuff around s-risks and weird decision theories etc., but if you look at what people talk about at EAGs today it feels mostly like more precise versions of what was discussed in 2016, rather than a large and unpredictable jump from the older understanding. Or, more technically: it feels like weâve had updates in evidence-space, but not as many updates in hypothesis-space, and I understand the latter to be motivating imprecision.
Obviously, this could be because EAs suck at cause prio research, or we just havenât been hit yet with the big update, etc., but the âpessimal inductionâ seems less pessimal to me.
Not a comprehensive reply, but: I think many of the examples youâre talking about are arguably cases of coarse awareness. People were coarsely aware of the potential backfire risks earlier on, but (arguably) the reason they didnât give these risks enough weight was that they didnât have a more fine-grained awareness of the specific causal pathways. I think such cases count as evidence for the pessimistic induction.
Sorry Iâm probably missing something, but Iâm not understanding why real world examples from EA would be particularly relevant given how young a movement it is. I think someone could grant that we have the ability to be justified in assigning probabilities to things that are likely to happen soon, and agree that the risk of things weâre totally unaware of happening in the next ~ 10-50 years might be (at least in some circumstances) sufficiently small to not have unawareness problems.
But once you start trying to be an impartial altruist about far future beings, that seems to me where you really canât get away from unawareness problems. And so I guess if you wanted to convince me I was wrong about that, we should be looking at things that people thought 1000 years ago, and how things they caused today were bad even though they were trying to do good for reasons they werenât only poorly calibrated on but in fact totally unaware ofâand it just seems likely to me there would be tons of examples of that?
Maybe the development of gunpowder stands out here as something being pursued in the hopes of achieving eternal life (ostensibly an altruistic motivation) and presumably the possibility of guns was not on peopleâs radar. I guess it would eventually have been figured out anyway, but how much harm did having gunpowder X years earlier cause?
Maybe an objection here is that an âidealâ agent would have of course considered the possibility of any chemical work being misused, but IDKâthey werenât even trying to make something explosive. I donât see how even a perfectly rational being could have predicted all the harms gunpowder would cause given that they were aiming to do alchemy. What probability could they have possibly been justified, given their epistemic position, in assigning to âsuper bad outcomes from pursuing eternal life chemistryâ given that they probably could not have imagined the scale of modern warfare?
I do get a little mixed up on this between âpeople are not ideal and so regularly make large mistakes that look like cluelessnessâ vs âeven an ideal agent could not be justified in their probability assignments given what is theoretically knowableâ so maybe Iâm misunderstanding something.
Anthony cites Greaves and MacAskill giving an example similar to your gunpowder one:
Consider, for example, would-be longtermists in the Middle Ages. It is plausible that the considerations most relevant to their decision â such as the benefits of science, and therefore the enormous value of efforts to help make the scientific and industrial revolutions happen sooner â would not have been on their radar. Rather, they might instead have backed attempts to spread Christianity, perhaps by violence: a putative route to value that, by our more enlightened lights today, looks wildly off the mark. The suggestion, then, is that our current predicament is relevantly similar to that of our medieval would-be longtermists.
I personally think these examples are less compelling than they first appear (e.g. the persistence literature generally finds weaker effects than what you might imagine), but I agree that a failure of EAs to find examples of sign flips doesnât mean that future ones wonât exist.
Whatâs the clearest example of a complex cluelessness sign flip youâre aware of?
(By âclearâ I mean âhad a very narrow confidence interval before encountering some consideration and a narrow interval after encountering that consideration but the CIs now center points with opposite signsâ.[1])
The clearest examples I know of (e.g. rescuing Hitler as a child) seem to me like examples of simple cluelessness. You list some examples here, but they donât seem that clear to me, e.g. I disagree that âEarly awareness-raising about AGI x-risk presumably seemed robustly goodâ and would guess most people involved in that had CIs which comfortably straddled zero.
Or alternatively: there are two representors with narrow but non-overlapping CIs.
Hi Ben, I like the spirit of this question, though Iâm not sure itâs the most relevant formulation. Thoughts on that, before I answer your literal question:
To get clear on terms:
Iâm assuming by âCIâ here, you mean something like your precise 90% (or whatever) confidence interval for your idealized selfâs EV of the intervention (as per Premise 1).
By ârobustâ, I donât mean a narrow /â strictly positive CI. I mean that the verdict âthis intervention is positive âin expectationââ isnât sensitive to arbitrary choices about how to factor in the considerations weâre unaware of.
I think itâs fair to say most people involved in early AI risk advocacy considered their work robustly positive in that sense.
This definition of ârobustâ is what matters for Premise 3. Because P3 says, our verdicts about interventions having positive âEVâ are sensitive to such arbitrary choices â even if we admit weâre very uncertain (i.e. we have wide CIs).
So, if weâre asking whether a given âsign flipâ counts as evidence for P3, I donât see why the bar should be ânarrow CIs with opposite-sign center points before and after the considerationâ.
If weâve discovered a consideration that flipped us from âwide CI centered at a positive EVâ to âwide CI centered at a negative EVâ, isnât that some evidence that our initial âpositive in expectationâ verdict wasnât robust, in the sense above? (And hence inductive evidence that our current âpositive in expectationâ verdicts arenât robust (i.e., evidence for P3), as argued here.)
Anyway, Iâd agree that the clearest evidence for P3 would come from sign flips that meet your bar. Maybe the small animal replacement problem? Iâd guess lots of people who care about animal welfare thought that getting people to eat less beef was clearly good before being aware of SARP, and think itâs clearly net-bad after being aware of SARP. (Itâs harder to come by examples of sign flips by your def for longtermist causes, because non-clueless longtermists typically agree that we should be very uncertain about the far future. But per the above, this is to be expected if weâre clueless.)
Thanks! I canât tell if this is cruxy, but for what itâs worth your âpessimal inductionâ vignettes donât resonate with me in a way which makes me less motivated by the unawareness concerns.
For example, Bostrom coined the phrase âattention hazardâ in 2011. I remember someone telling me that MIRI was net-negative for this reason at EAG 2015, and I would be surprised if e.g. Habryka hadnât considered this risk before starting Lightcone. So I disagree with citing him/âthis as a good example of unawareness; itâs more that they mis-estimated a known risk factor.
Similarly, I remember talking about SARP at one of my first EAGs. I think I came across it in Brian Tomasikâs 2007 post, maybe even before I had encountered EA. Perhaps Iâve mis-estimated those concerns, but it doesnât seem like unawareness.
My overall experience is kind of the opposite of yours: when I got involved in EA people talked a lot about âCause Xâ and âCrucial Considerationsâ and now theyâve mostly just⌠stopped? Like people tried to find other considerations, and thereâs some new stuff around s-risks and weird decision theories etc., but if you look at what people talk about at EAGs today it feels mostly like more precise versions of what was discussed in 2016, rather than a large and unpredictable jump from the older understanding. Or, more technically: it feels like weâve had updates in evidence-space, but not as many updates in hypothesis-space, and I understand the latter to be motivating imprecision.
Obviously, this could be because EAs suck at cause prio research, or we just havenât been hit yet with the big update, etc., but the âpessimal inductionâ seems less pessimal to me.
Interesting, thatâs helpful to know.
Not a comprehensive reply, but: I think many of the examples youâre talking about are arguably cases of coarse awareness. People were coarsely aware of the potential backfire risks earlier on, but (arguably) the reason they didnât give these risks enough weight was that they didnât have a more fine-grained awareness of the specific causal pathways. I think such cases count as evidence for the pessimistic induction.
Sorry Iâm probably missing something, but Iâm not understanding why real world examples from EA would be particularly relevant given how young a movement it is. I think someone could grant that we have the ability to be justified in assigning probabilities to things that are likely to happen soon, and agree that the risk of things weâre totally unaware of happening in the next ~ 10-50 years might be (at least in some circumstances) sufficiently small to not have unawareness problems.
But once you start trying to be an impartial altruist about far future beings, that seems to me where you really canât get away from unawareness problems. And so I guess if you wanted to convince me I was wrong about that, we should be looking at things that people thought 1000 years ago, and how things they caused today were bad even though they were trying to do good for reasons they werenât only poorly calibrated on but in fact totally unaware ofâand it just seems likely to me there would be tons of examples of that?
Maybe the development of gunpowder stands out here as something being pursued in the hopes of achieving eternal life (ostensibly an altruistic motivation) and presumably the possibility of guns was not on peopleâs radar. I guess it would eventually have been figured out anyway, but how much harm did having gunpowder X years earlier cause?
Maybe an objection here is that an âidealâ agent would have of course considered the possibility of any chemical work being misused, but IDKâthey werenât even trying to make something explosive. I donât see how even a perfectly rational being could have predicted all the harms gunpowder would cause given that they were aiming to do alchemy. What probability could they have possibly been justified, given their epistemic position, in assigning to âsuper bad outcomes from pursuing eternal life chemistryâ given that they probably could not have imagined the scale of modern warfare?
I do get a little mixed up on this between âpeople are not ideal and so regularly make large mistakes that look like cluelessnessâ vs âeven an ideal agent could not be justified in their probability assignments given what is theoretically knowableâ so maybe Iâm misunderstanding something.
Anthony cites Greaves and MacAskill giving an example similar to your gunpowder one:
I personally think these examples are less compelling than they first appear (e.g. the persistence literature generally finds weaker effects than what you might imagine), but I agree that a failure of EAs to find examples of sign flips doesnât mean that future ones wonât exist.