During the century we’re in right now, we will develop technologies that cause us to transition to a state in which humans as we know them are no longer the main force in world events. This is our last chance to shape how that transition happens.
Whatever the main force in world events is (perhaps digital people, misaligned AI, or something else) will create highly stable civilizations that populate our entire galaxy for billions of years to come. The transition taking place this century could shape all of that.
I think it’s very unclear whether this would be a good or bad thing.
There are some people (e.g. Evitable) who are trying to stop this transition, but I think more AI safety people would self-describe their work as “shap[ing] how that transition happens.”
What I said was unclear; I probably should have just quoted Holden Karnofsky:
There are some people (e.g. Evitable) who are trying to stop this transition, but I think more AI safety people would self-describe their work as “shap[ing] how that transition happens.”