Thank you! Yeah, I’m aware of these—which is part of why I’m not working on AI safety and generally don’t have strong opinions on interventions in that area (too much uncertainty). I should’ve been more specific, sorry—I mean more like “promoting wisdom, cooperativeness, knowledge, etc”. I only skimmed the series, so I may have missed it—do you list some plausible ways this could backfire anywhere? It just seems like the world would have to be “weird” in some way—I can roughly grasp at some possibilities (knowledge could be used for bad / infohazards, cooperation could make bad lock-in more likely), but I’m sure you have some concrete visions / this has been written up? (Also feel free to say if you feel constrained here due to infohazard considerations).
Claude is pointing me to some related older CLR writing, so I may read that.
Cool! For other readers, I think the most relevant sections of the sequence to your question here are 4.1.3 “Meta-extrapolation” and 4.1.6 “Capacity-building”. They don’t go into much concrete detail on backfire risks of “promoting wisdom, cooperativeness, knowledge, etc”. But yeah, mostly stuff like infohazards and dual use, plus the unknown unknown downsides we should expect from pessimistic induction. The idea is that:
The historical mechanisms by which promoting wisdom/cooperativeness/knowledge made things better occurred in the context of fairly non-weird human socioeconomics. AI takeoff and space colonization are much weirder contexts.
Even if the downsides seem unlikely in absolute terms, the intended upsides from promoting wisdom etc. are so indirect that I think we should also consider them similarly unlikely.
Thanks—I’m curious, how much more convincing would you guess is the case if you were to be completely open about the infohazardous stuff? 1.5x probability of convincing someone? 3x? 10x?
Oh, I don’t think the worry hinges on particular infohazards that aren’t public in EA. I’m thinking of a pretty general problem like: “The value of the future from the perspective of your altruistic values, epistemology, and decision theory upon reflection is probably a non-monotonic function of how much you increase wisdom etc. broadly. More ‘wisdom’ or knowledge for actors who are misaligned with you can be quite bad.” And this is at least somewhat borne out by examples like AI movement building, biorisk, and technological progress making factory farming worse.
Huh, I’m a little surprised to hear that, to be honest. To be clear, I mean something more like “visceral”/”rhetorical”/”de facto” convincingness, not whether it purely logically hinges on it.
Also, just thinking of this because I’m reading it right now—if you want to convince more people of your view, a critical review / “rebuttal” of https://www.forethought.org/research/how-to-make-the-future-better might be cool. Would be memetically strong. (I could also imagine reasons why you might not want that, of course).
Thank you! Yeah, I’m aware of these—which is part of why I’m not working on AI safety and generally don’t have strong opinions on interventions in that area (too much uncertainty). I should’ve been more specific, sorry—I mean more like “promoting wisdom, cooperativeness, knowledge, etc”. I only skimmed the series, so I may have missed it—do you list some plausible ways this could backfire anywhere? It just seems like the world would have to be “weird” in some way—I can roughly grasp at some possibilities (knowledge could be used for bad / infohazards, cooperation could make bad lock-in more likely), but I’m sure you have some concrete visions / this has been written up? (Also feel free to say if you feel constrained here due to infohazard considerations).
Claude is pointing me to some related older CLR writing, so I may read that.
Okay, I read more and understand it better now.
Cool! For other readers, I think the most relevant sections of the sequence to your question here are 4.1.3 “Meta-extrapolation” and 4.1.6 “Capacity-building”. They don’t go into much concrete detail on backfire risks of “promoting wisdom, cooperativeness, knowledge, etc”. But yeah, mostly stuff like infohazards and dual use, plus the unknown unknown downsides we should expect from pessimistic induction. The idea is that:
The historical mechanisms by which promoting wisdom/cooperativeness/knowledge made things better occurred in the context of fairly non-weird human socioeconomics. AI takeoff and space colonization are much weirder contexts.
Even if the downsides seem unlikely in absolute terms, the intended upsides from promoting wisdom etc. are so indirect that I think we should also consider them similarly unlikely.
Thanks—I’m curious, how much more convincing would you guess is the case if you were to be completely open about the infohazardous stuff? 1.5x probability of convincing someone? 3x? 10x?
Oh, I don’t think the worry hinges on particular infohazards that aren’t public in EA. I’m thinking of a pretty general problem like: “The value of the future from the perspective of your altruistic values, epistemology, and decision theory upon reflection is probably a non-monotonic function of how much you increase wisdom etc. broadly. More ‘wisdom’ or knowledge for actors who are misaligned with you can be quite bad.” And this is at least somewhat borne out by examples like AI movement building, biorisk, and technological progress making factory farming worse.
Huh, I’m a little surprised to hear that, to be honest. To be clear, I mean something more like “visceral”/”rhetorical”/”de facto” convincingness, not whether it purely logically hinges on it.
Also, just thinking of this because I’m reading it right now—if you want to convince more people of your view, a critical review / “rebuttal” of https://www.forethought.org/research/how-to-make-the-future-better might be cool. Would be memetically strong. (I could also imagine reasons why you might not want that, of course).
I ended up writing my own version of the list (for AI interventions, not for wisdom/cooperation) in a post, would be curious for your thoughts!