Thanks, this makes sense! Yeah, this is why many arguments I see start at a more abstract level, e.g.
We are building machines that will become vastly more intelligent than us (c.f. superior strategic planning), and it seems reasonable that then we won’t be able to predict/control them
Any rational agent will strategically develop instrumental goals that could make it hard for us to ensure alignment (e.g., self-preservation → can’t turn them off)
I might have entered at a different vector (all online) so I experienced a different introduction to the idea! If my experience is atypical, and most people get the “gentle” introduction you described, that is great news.
Thanks, this makes sense! Yeah, this is why many arguments I see start at a more abstract level, e.g.
We are building machines that will become vastly more intelligent than us (c.f. superior strategic planning), and it seems reasonable that then we won’t be able to predict/control them
Any rational agent will strategically develop instrumental goals that could make it hard for us to ensure alignment (e.g., self-preservation → can’t turn them off)
I might have entered at a different vector (all online) so I experienced a different introduction to the idea! If my experience is atypical, and most people get the “gentle” introduction you described, that is great news.