Hmm, I think you’re overstating the disanalogy here. In the case of the individual life, you say “the reasons to trust “that some process designed their body and mind to function well” are relatively strong”. But we also have strong reasons to trust that some process designed our cooperative instincts to allow groups of humans to cooperate effectively.
I also think that many individuals need to decide how to make their lives go well in pretty confusing circumstances. Imagine deciding whether to immigrate to America in the 1700s, or how to live in the shadow of the Cold War, or whether to genetically engineer your children. There’s a lot of stuff that’s unprecedented, and which you only get one shot at.
Re the experience of AI safety so far: I certainly do think that a bunch of actions taken by the AI safety movement have backfired. I also think that a bunch have succeeded. If you think of the AI safety movement as a young organism going through its adolescence, it’s gaining a huge amount of experience from all of these interactions with the world—and the net value of the things that it’s done so far may well be dominated by what updates it makes based on that experience.
I guess that’s where we disagree. It seems like you think that we should update towards being clueless. Whereas I think we can extract a bunch of generalizable lessons that make us less clueless than we used to be—and one of those lessons is that many of these mistakes could have been prevented by using the right kinds of non-model-based decision-making.
EDIT: another way of trying to get at the crux: do you think that, if we had a theory of sociopolitics that was about as good as 20th-century economics, then we wouldn’t be clueless about how to do sociopolitical interventions (like founding AI safety movements) effectively?
But we also have strong reasons to trust that some process designed our cooperative instincts to allow groups of humans to cooperate effectively.
“[A]llowing [small] groups of humans to cooperate effectively” is very far from “making the far future better, impartially speaking”. I’d be interested in your responses to the arguments here.
I also think that many individuals need to decide how to make their lives go well in pretty confusing circumstances. Imagine deciding whether to immigrate to America in the 1700s, or how to live in the shadow of the Cold War, or whether to genetically engineer your children.
First, it’s not clear to me these people weren’t clueless — i.e. really had more reason to choose whatever they chose than the alternatives — depending on how long a time horizon they were aiming to make go well.
Second, insofar as we think these people’s choices were justified, I don’t see why you think their instincts gave them such justification. Why would these instincts track unprecedented consequences so well?
and the net value of the things that it’s done so far may well be dominated by what updates it makes based on that experience
I don’t think “may well” gets us very far. Can you say more why this hypothesis is so much more likely than, say, “the dominant impacts are the damage that’s already been done”, or “the dominant impacts will come from near-future decisions, made by actors who are still too ignorant about the extremely complex system they’re intervening in”?
do you think that, if we had a theory of sociopolitics that was about as good as 20th-century economics, then we wouldn’t be clueless about how to do sociopolitical interventions (like founding AI safety movements) effectively?
No, because I think “founding AI safety movements that succeed at making the far future go better” is a pretty out-of-distribution kind of sociopolitical intervention.
Hmm, I think you’re overstating the disanalogy here. In the case of the individual life, you say “the reasons to trust “that some process designed their body and mind to function well” are relatively strong”. But we also have strong reasons to trust that some process designed our cooperative instincts to allow groups of humans to cooperate effectively.
I also think that many individuals need to decide how to make their lives go well in pretty confusing circumstances. Imagine deciding whether to immigrate to America in the 1700s, or how to live in the shadow of the Cold War, or whether to genetically engineer your children. There’s a lot of stuff that’s unprecedented, and which you only get one shot at.
Re the experience of AI safety so far: I certainly do think that a bunch of actions taken by the AI safety movement have backfired. I also think that a bunch have succeeded. If you think of the AI safety movement as a young organism going through its adolescence, it’s gaining a huge amount of experience from all of these interactions with the world—and the net value of the things that it’s done so far may well be dominated by what updates it makes based on that experience.
I guess that’s where we disagree. It seems like you think that we should update towards being clueless. Whereas I think we can extract a bunch of generalizable lessons that make us less clueless than we used to be—and one of those lessons is that many of these mistakes could have been prevented by using the right kinds of non-model-based decision-making.
EDIT: another way of trying to get at the crux: do you think that, if we had a theory of sociopolitics that was about as good as 20th-century economics, then we wouldn’t be clueless about how to do sociopolitical interventions (like founding AI safety movements) effectively?
“[A]llowing [small] groups of humans to cooperate effectively” is very far from “making the far future better, impartially speaking”. I’d be interested in your responses to the arguments here.
First, it’s not clear to me these people weren’t clueless — i.e. really had more reason to choose whatever they chose than the alternatives — depending on how long a time horizon they were aiming to make go well.
Second, insofar as we think these people’s choices were justified, I don’t see why you think their instincts gave them such justification. Why would these instincts track unprecedented consequences so well?
I don’t think “may well” gets us very far. Can you say more why this hypothesis is so much more likely than, say, “the dominant impacts are the damage that’s already been done”, or “the dominant impacts will come from near-future decisions, made by actors who are still too ignorant about the extremely complex system they’re intervening in”?
No, because I think “founding AI safety movements that succeed at making the far future go better” is a pretty out-of-distribution kind of sociopolitical intervention.