I’m also still a bit confused about what exactly this concept refers to. Is a ‘consequentialist’ basically just an ‘optimiser’ in the sense that Yudkowsky uses in the sequences (e.g. here), that has later been refined by posts like this one (where it’s called ‘selection’) and this one?
In other words, roughly speaking, is a system a consequentialist to the extent that it’s trying to take actions that push its environment towards a certain goal state?
Found the source. There, he says that an “explicit cognitive model and explicit forecasts” about the future are necessary to true consequentialist cognition (CC). He agrees that CC is already common among optimisers (like chess engines); the dangerous kind is consequentialism over broad domains (i.e. where everything in the world is in play, is a possible means, while the chess engine only considers the set of legal moves as its domain).
“Goal-seeking” seems like the previous, less-confusing word for it, not sure why people shifted.
I’m also still a bit confused about what exactly this concept refers to. Is a ‘consequentialist’ basically just an ‘optimiser’ in the sense that Yudkowsky uses in the sequences (e.g. here), that has later been refined by posts like this one (where it’s called ‘selection’) and this one?
In other words, roughly speaking, is a system a consequentialist to the extent that it’s trying to take actions that push its environment towards a certain goal state?
Found the source. There, he says that an “explicit cognitive model and explicit forecasts” about the future are necessary to true consequentialist cognition (CC). He agrees that CC is already common among optimisers (like chess engines); the dangerous kind is consequentialism over broad domains (i.e. where everything in the world is in play, is a possible means, while the chess engine only considers the set of legal moves as its domain).
“Goal-seeking” seems like the previous, less-confusing word for it, not sure why people shifted.