Found the source. There, he says that an “explicit cognitive model and explicit forecasts” about the future are necessary to true consequentialist cognition (CC). He agrees that CC is already common among optimisers (like chess engines); the dangerous kind is consequentialism over broad domains (i.e. where everything in the world is in play, is a possible means, while the chess engine only considers the set of legal moves as its domain).
“Goal-seeking” seems like the previous, less-confusing word for it, not sure why people shifted.
Found the source. There, he says that an “explicit cognitive model and explicit forecasts” about the future are necessary to true consequentialist cognition (CC). He agrees that CC is already common among optimisers (like chess engines); the dangerous kind is consequentialism over broad domains (i.e. where everything in the world is in play, is a possible means, while the chess engine only considers the set of legal moves as its domain).
“Goal-seeking” seems like the previous, less-confusing word for it, not sure why people shifted.