Thanks for your detailed answers Peter. Caroline Jenmaire’s “minimal menace” is a good definition of AGI for our purposes (but also so is Holden Karnofsky’s PASTA, OpenPhil’s Transformative AI and Matthew Barnett’s TAI.)
I’m curious about your 5% by 2035 figure. Has this changed much as a result of GPT-4? And what is happening in the remaining 95%? How much of that is extra “secret sauce” remaining undiscovered? A big reason for me updating so heavily toward AGI being near (and correspondingly, doom being high given the woeful state-of-the-art in Alignment) is the realisation that there very well may be no additional secret sauce necessary and all that is needed is more compute and data (read: money) being thrown at it (and 2 OOMs increase in training FLOP over GPT-4 is possible within 6-12 months).
the possibility of policy delaying AI development
How likely do you consider this to be, conditional on business as usual? I think things are moving in the right direction, but we can’t afford to be complacent. Indeed we should be pushing maximally for it to happen (to the point where, to me, almost anything else looks like “rearranging deckchairs on the Titanic”).
Whilst I may not be a professional forecaster, I am a successful investor and I think I have a reasonable track record of being early to a number of significant global trends: veganism (2005), cryptocurrency (several big wins from investing early—BTC, ETH, DOT, SOL, KAS; maybe a similar amount of misses but overall up ~1000x), Covid (late Jan 2020), AI x-risk (2009), AGI moratorium (2023, a few days before the FLI letter went public).
Thanks for your detailed answers Peter. Caroline Jenmaire’s “minimal menace” is a good definition of AGI for our purposes (but also so is Holden Karnofsky’s PASTA, OpenPhil’s Transformative AI and Matthew Barnett’s TAI.)
I’m curious about your 5% by 2035 figure. Has this changed much as a result of GPT-4? And what is happening in the remaining 95%? How much of that is extra “secret sauce” remaining undiscovered? A big reason for me updating so heavily toward AGI being near (and correspondingly, doom being high given the woeful state-of-the-art in Alignment) is the realisation that there very well may be no additional secret sauce necessary and all that is needed is more compute and data (read: money) being thrown at it (and 2 OOMs increase in training FLOP over GPT-4 is possible within 6-12 months).
How likely do you consider this to be, conditional on business as usual? I think things are moving in the right direction, but we can’t afford to be complacent. Indeed we should be pushing maximally for it to happen (to the point where, to me, almost anything else looks like “rearranging deckchairs on the Titanic”).
Whilst I may not be a professional forecaster, I am a successful investor and I think I have a reasonable track record of being early to a number of significant global trends: veganism (2005), cryptocurrency (several big wins from investing early—BTC, ETH, DOT, SOL, KAS; maybe a similar amount of misses but overall up ~1000x), Covid (late Jan 2020), AI x-risk (2009), AGI moratorium (2023, a few days before the FLI letter went public).