I expect that a significant dynamic in this world would be that there’d be major investment in attempting to recover knowledge from the previous civilization. That’s because:
Intellectually, it seems fascinating.
If a previous civilization more advanced than ours had existed and then collapsed, I imagine today’s historians would be hugely interested in that.
More importantly, there would be a huge economic incentive to understand the previous civilization:
Many of the richest and most successful people today are people who anticipated (the consequences of) important technological developments. E.g. people who specialized or invested in AI a decade ago, people who anticipated that internet commerce would be a huge deal, people who anticipated that software development was going to be very valuable, etc.
Similarly, there’s a huge edge to be had in science from knowing which domains are promising at which time.
This is important because it’s perhaps the main argument for the Optimistic view regarding whether a post-setback society would retain alignment knowledge.
You name various arguments for the Pessimistic view. Those seem reasonable to me, but I think they do have to be weighed against the fact that people would be trying pretty hard to recover a lot of knowledge about today’s world, s.t. substantial difficulties could very plausibly be overcome (e.g. deciphering old hard drives).
This is esp. true once that new civilization has technology advanced enough to be at the cusp of AGI.
I don’t have a strong view on where that leaves me overall, but intuitively, I probably feel more optimistic than you seem to be.
Separately, I think it’s worth noting that regardless of whether historians manage to recover technical knowledge about alignment, it would likely be obvious very early on that the previous civilization reached something like AGI. This would radically change the governance landscape relative to today’s world, and would plausibly make the problem easier the second time around.
Finally, I also wanted to note that it seems intuitively likely to me that the effect on trajectory change (described in the Appendix) is more important than the effect on existential risk.
Aside: another consequence if historians are able to recover a lot of information is that economic growth in the rerun might be substantially faster than today. Scientists, entrepreneurs, and investors could learn a ton about which pursuits are most promising at what points. In particular, AI and deep learning investment might happen earlier. This might be good (e.g. because faster growth means there’s generally more surplus around the crucial period and less zero-sum mentality) or bad (e.g. because AI progress is already scary fast today, and it might be even faster in this world, since the payoff would be much clearer to everyone).
Interesting post!
I expect that a significant dynamic in this world would be that there’d be major investment in attempting to recover knowledge from the previous civilization. That’s because:
Intellectually, it seems fascinating.
If a previous civilization more advanced than ours had existed and then collapsed, I imagine today’s historians would be hugely interested in that.
More importantly, there would be a huge economic incentive to understand the previous civilization:
Many of the richest and most successful people today are people who anticipated (the consequences of) important technological developments. E.g. people who specialized or invested in AI a decade ago, people who anticipated that internet commerce would be a huge deal, people who anticipated that software development was going to be very valuable, etc.
Similarly, there’s a huge edge to be had in science from knowing which domains are promising at which time.
This is important because it’s perhaps the main argument for the Optimistic view regarding whether a post-setback society would retain alignment knowledge.
You name various arguments for the Pessimistic view. Those seem reasonable to me, but I think they do have to be weighed against the fact that people would be trying pretty hard to recover a lot of knowledge about today’s world, s.t. substantial difficulties could very plausibly be overcome (e.g. deciphering old hard drives).
This is esp. true once that new civilization has technology advanced enough to be at the cusp of AGI.
I don’t have a strong view on where that leaves me overall, but intuitively, I probably feel more optimistic than you seem to be.
Separately, I think it’s worth noting that regardless of whether historians manage to recover technical knowledge about alignment, it would likely be obvious very early on that the previous civilization reached something like AGI. This would radically change the governance landscape relative to today’s world, and would plausibly make the problem easier the second time around.
Finally, I also wanted to note that it seems intuitively likely to me that the effect on trajectory change (described in the Appendix) is more important than the effect on existential risk.
Aside: another consequence if historians are able to recover a lot of information is that economic growth in the rerun might be substantially faster than today. Scientists, entrepreneurs, and investors could learn a ton about which pursuits are most promising at what points. In particular, AI and deep learning investment might happen earlier. This might be good (e.g. because faster growth means there’s generally more surplus around the crucial period and less zero-sum mentality) or bad (e.g. because AI progress is already scary fast today, and it might be even faster in this world, since the payoff would be much clearer to everyone).