Thanks! I do something like this in post #2, sec. 2.3. This first post is just meant to set up the problem at a high level. Interested to hear your thoughts on those examples.
(See also hereâthese arenât âreally weirdâ, but they do seem like pretty plausible backfire mechanisms to me.)
Thank you! Yeah, Iâm aware of theseâwhich is part of why Iâm not working on AI safety and generally donât have strong opinions on interventions in that area (too much uncertainty). I shouldâve been more specific, sorryâI mean more like âpromoting wisdom, cooperativeness, knowledge, etcâ. I only skimmed the series, so I may have missed itâdo you list some plausible ways this could backfire anywhere? It just seems like the world would have to be âweirdâ in some wayâI can roughly grasp at some possibilities (knowledge could be used for bad /â infohazards, cooperation could make bad lock-in more likely), but Iâm sure you have some concrete visions /â this has been written up? (Also feel free to say if you feel constrained here due to infohazard considerations).
Claude is pointing me to some related older CLR writing, so I may read that.
Cool! For other readers, I think the most relevant sections of the sequence to your question here are 4.1.3 âMeta-extrapolationâ and 4.1.6 âCapacity-buildingâ. They donât go into much concrete detail on backfire risks of âpromoting wisdom, cooperativeness, knowledge, etcâ. But yeah, mostly stuff like infohazards and dual use, plus the unknown unknown downsides we should expect from pessimistic induction. The idea is that:
The historical mechanisms by which promoting wisdom/âcooperativeness/âknowledge made things better occurred in the context of fairly non-weird human socioeconomics. AI takeoff and space colonization are much weirder contexts.
Even if the downsides seem unlikely in absolute terms, the intended upsides from promoting wisdom etc. are so indirect that I think we should also consider them similarly unlikely.
ThanksâIâm curious, how much more convincing would you guess is the case if you were to be completely open about the infohazardous stuff? 1.5x probability of convincing someone? 3x? 10x?
Oh, I donât think the worry hinges on particular infohazards that arenât public in EA. Iâm thinking of a pretty general problem like: âThe value of the future from the perspective of your altruistic values, epistemology, and decision theory upon reflection is probably a non-monotonic function of how much you increase wisdom etc. broadly. More âwisdomâ or knowledge for actors who are misaligned with you can be quite bad.â And this is at least somewhat borne out by examples like AI movement building, biorisk, and technological progress making factory farming worse.
Huh, Iâm a little surprised to hear that, to be honest. To be clear, I mean something more like âvisceralâ/âârhetoricalâ/ââde factoâ convincingness, not whether it purely logically hinges on it.
Also, just thinking of this because Iâm reading it right nowâif you want to convince more people of your view, a critical review /â ârebuttalâ of https://ââwww.forethought.org/ââresearch/ââhow-to-make-the-future-better might be cool. Would be memetically strong. (I could also imagine reasons why you might not want that, of course).
Thanks! I do something like this in post #2, sec. 2.3. This first post is just meant to set up the problem at a high level. Interested to hear your thoughts on those examples.
(See also hereâthese arenât âreally weirdâ, but they do seem like pretty plausible backfire mechanisms to me.)
Thank you! Yeah, Iâm aware of theseâwhich is part of why Iâm not working on AI safety and generally donât have strong opinions on interventions in that area (too much uncertainty). I shouldâve been more specific, sorryâI mean more like âpromoting wisdom, cooperativeness, knowledge, etcâ. I only skimmed the series, so I may have missed itâdo you list some plausible ways this could backfire anywhere? It just seems like the world would have to be âweirdâ in some wayâI can roughly grasp at some possibilities (knowledge could be used for bad /â infohazards, cooperation could make bad lock-in more likely), but Iâm sure you have some concrete visions /â this has been written up? (Also feel free to say if you feel constrained here due to infohazard considerations).
Claude is pointing me to some related older CLR writing, so I may read that.
Okay, I read more and understand it better now.
Cool! For other readers, I think the most relevant sections of the sequence to your question here are 4.1.3 âMeta-extrapolationâ and 4.1.6 âCapacity-buildingâ. They donât go into much concrete detail on backfire risks of âpromoting wisdom, cooperativeness, knowledge, etcâ. But yeah, mostly stuff like infohazards and dual use, plus the unknown unknown downsides we should expect from pessimistic induction. The idea is that:
The historical mechanisms by which promoting wisdom/âcooperativeness/âknowledge made things better occurred in the context of fairly non-weird human socioeconomics. AI takeoff and space colonization are much weirder contexts.
Even if the downsides seem unlikely in absolute terms, the intended upsides from promoting wisdom etc. are so indirect that I think we should also consider them similarly unlikely.
ThanksâIâm curious, how much more convincing would you guess is the case if you were to be completely open about the infohazardous stuff? 1.5x probability of convincing someone? 3x? 10x?
Oh, I donât think the worry hinges on particular infohazards that arenât public in EA. Iâm thinking of a pretty general problem like: âThe value of the future from the perspective of your altruistic values, epistemology, and decision theory upon reflection is probably a non-monotonic function of how much you increase wisdom etc. broadly. More âwisdomâ or knowledge for actors who are misaligned with you can be quite bad.â And this is at least somewhat borne out by examples like AI movement building, biorisk, and technological progress making factory farming worse.
Huh, Iâm a little surprised to hear that, to be honest. To be clear, I mean something more like âvisceralâ/âârhetoricalâ/ââde factoâ convincingness, not whether it purely logically hinges on it.
Also, just thinking of this because Iâm reading it right nowâif you want to convince more people of your view, a critical review /â ârebuttalâ of https://ââwww.forethought.org/ââresearch/ââhow-to-make-the-future-better might be cool. Would be memetically strong. (I could also imagine reasons why you might not want that, of course).
I ended up writing my own version of the list (for AI interventions, not for wisdom/âcooperation) in a post, would be curious for your thoughts!