I work at Open Philanthropy, and I recently let Gavin know that Open Phil is planning to recommend a grant of $5k to Arb for the second project on your list: Overview of AI Safety in 2024 (they had already raised ~$10k by the time we came across it). Thanks for writing this post Austin — it brought the funding opportunity to our attention.
Like other commenters on Manifund, I believe this kind of overview is a valuable reference for the field, especially for newcomers.
I wanted to flag that this project would have been eligible for our RFP for work that builds capacity to address risks from transformative AI. I worry that not all potential applicants are aware of the RFP or its scope, so I’ll take this opportunity to mention that this RFP’s scope is quite broad, including funding for:
Training and mentorship programs
Events
Groups
Resources, media, and communications
Almost any other type of project that builds capacity for advanced AI risks (in the sense of increasing the number of careers devoted to these problems, supporting people doing this work, and sharing knowledge related to this work).
More details at the link above. People might also find this page helpful, which lists all currently open application programs at Open Phil.
Interesting post!
I expect that a significant dynamic in this world would be that there’d be major investment in attempting to recover knowledge from the previous civilization. That’s because:
Intellectually, it seems fascinating.
If a previous civilization more advanced than ours had existed and then collapsed, I imagine today’s historians would be hugely interested in that.
More importantly, there would be a huge economic incentive to understand the previous civilization:
Many of the richest and most successful people today are people who anticipated (the consequences of) important technological developments. E.g. people who specialized or invested in AI a decade ago, people who anticipated that internet commerce would be a huge deal, people who anticipated that software development was going to be very valuable, etc.
Similarly, there’s a huge edge to be had in science from knowing which domains are promising at which time.
This is important because it’s perhaps the main argument for the Optimistic view regarding whether a post-setback society would retain alignment knowledge.
You name various arguments for the Pessimistic view. Those seem reasonable to me, but I think they do have to be weighed against the fact that people would be trying pretty hard to recover a lot of knowledge about today’s world, s.t. substantial difficulties could very plausibly be overcome (e.g. deciphering old hard drives).
This is esp. true once that new civilization has technology advanced enough to be at the cusp of AGI.
I don’t have a strong view on where that leaves me overall, but intuitively, I probably feel more optimistic than you seem to be.
Separately, I think it’s worth noting that regardless of whether historians manage to recover technical knowledge about alignment, it would likely be obvious very early on that the previous civilization reached something like AGI. This would radically change the governance landscape relative to today’s world, and would plausibly make the problem easier the second time around.
Finally, I also wanted to note that it seems intuitively likely to me that the effect on trajectory change (described in the Appendix) is more important than the effect on existential risk.
Aside: another consequence if historians are able to recover a lot of information is that economic growth in the rerun might be substantially faster than today. Scientists, entrepreneurs, and investors could learn a ton about which pursuits are most promising at what points. In particular, AI and deep learning investment might happen earlier. This might be good (e.g. because faster growth means there’s generally more surplus around the crucial period and less zero-sum mentality) or bad (e.g. because AI progress is already scary fast today, and it might be even faster in this world, since the payoff would be much clearer to everyone).