Global moratorium on AGI, now (Twitter). Founder of CEEALAR (née the EA Hotel; ceealar.org)
Greg_Colbourn
Karma: 4,927
See my comment on that post for why I don’t agree. I agree nuclear extinction risk is low (but probably not that low)[1]. ASI is really the only thing that is likely to kill every last human (and I think it is quite likely to do that given it will be way more powerful than anything else[2]).
Interesting. Obviously I don’t want to discourage you from the bet, but I’m surprised you are so confident based on this! I don’t think the prior of mammal species duration is really relevant at all, when for 99.99% of the last 1M years there hasn’t been any significant technology. Perhaps more relevant is homo sapiens wiping out all the less intelligent hominids (and many other species).
I think the chance of humans going extinct until the end of 2027 is basically negligible. I would guess around 10^-7 per year.
Would be interested to see your reasoning for this, if you have it laid out somewhere. Is it mainly because you think it’s ~impossible for AGI/ASI to happen in that time? Or because it’s ~impossible for AGI/ASI to cause human extinction?
I don’t have a stable income so I can’t get bank loans (I have tried to get a mortgage for the property before and failed—they don’t care if you have millions in assets, all they care about is your income[1], and I just have a relatively small, irregular rental income (Airbnb). But I can get crypto-backed smart contract loans, and do have one out already on Aave, which I could extend.).
Also, the signalling value of the wager is pretty important too imo. I want people to put their money where their mouth is if they are so sure that AI x-risk isn’t a near term problem. And I want to put my money where my mouth is too, to show how serious I am about this.- ^
I think this is probably because they don’t want to go through the hassle of actually having to repossess your house, so if this seems at all likely they won’t bother with the loan in the first place.
- ^
It’s in Manchester, UK. I live elsewhere—renting currently, but shortly moving into another owned house that is currently being renovated (I’ve got a company managing the would-be-collateral house as an Airbnb, so no long term tenants either). Will send you more details via DM.
Cash is a tricky one, because I rarely hold much of it. I’m nearly always fully invested. But that includes plenty of liquid assets like crypto. Net worth wise, in 2027, assuming no AI-related craziness, I would be expect it to be in the 7-8 figure range, 5-95% maybe $500k-$100M).
Re risk, as per my offer on X, I’m happy to put my house up as collateral if you can be bothered to get the paperwork done. Otherwise happy to just trade on reputation (you can trash mine publicly if I don’t pay up).
As I say above, I’ve been offering a similar bet for a while already. The symbolism is a big part of it.
I can currently only take out crypto-backed loans, which have been quite high interest lately (don’t have a stable income so can’t get bank loans or mortgages), and have considered this but not done it yet.
Hi Vasco, sorry for the delay getting back to you. I have actually had a similar bet offer up on X for nearly a year (offering to go up to $250k) with only one taker for ~$30 so far! My one is you give x now and I give 2x in 5 years, which is pretty similar. Anyway, happy to go ahead with what you’ve suggested.
I would donate the $10k to PauseAI (I would say $10k to PauseAI in 2024 is much greater EV than $19k to PauseAI at end of 2027).
[BTW, I have tried to get Bryan Caplan interested too, to no avail—if anyone is in contact with him, please ask him about it.]
I’d say it’s more than a vague intuition. It follows from alignment/control/misuse/coordination not being (close to) solved and ASI being much more powerful than humanity. I think it should be possible to formalise it, even. “AGIs will be helping us on a lot of tasks”, “collusion is hard” and “people will get more scared over time” aren’t anywhere close to overcoming it imo.
More like, some people did share their concerns, but those they shared them with didn’t do anything about it (because of worrying about bad PR, but also maybe just as a kind of “ends justify the means” thing re his money going to EA. The latter might actually have been the larger effect.).
Maybe half the community sees it that way. But not the half with all the money and power it seems. There aren’t (yet) large resources being put into playing the “outside game”. And there hasn’t been anything in the way of EA leadership (OpenPhil, 80k) admitting the error afaik.
What makes you think the consciousness is expressed in human language by LLMs? Could it not be that the human language output is more akin to our unconscious physiological processes, and the real consciousness is in inscrutable (to us) floating point numbers (if it is there at all)?
What does Claude 3 produce from a null prompt (inc no pre-prompt)? Is it just gibberish? Does it show signs of consciousness? Has anyone done this experiment?
See all my comments and replies on the anti-pause posts. I don’t think any of the anti-pause arguments stand up if you put significant weight on timelines being short and p(doom) high (and viscerally grasp that yes, that means your own life is in danger, and those of your friends and family too, in the short term! It’s no longer just an abstract concern!).
As part of an AMA I put on X, I was asked for my “top five EA hot takes”. If you’ll excuse the more X-suited tone and spiciness, here they are:
1. OpenAI, Anthropic (and to a lesser extent DeepMind) were the worst cases of Unilateralists Curse of all time. EAs love to discourage enthusiastic newcomers by warning to not do “net negative” unilateralist actions (i.e. don’t start new projects in case they crowd out better, more “well thought through” projects in future, with “more competent” people doing them), but nothing will ever top the monumental unilateralist curse fuck up that was supporting Big AGI in it’s beginnings.
2. AI Safety is nothing without a Pause. Too many EAs are stuck in the pre-GPT-4 paradigm of maxing research, when it’ll all be for nothing unless we get a Pause first. More EAs should switch to Notkilleveryoneism/PauseAI/StopAGI.
3. EA is too elitist. We should be triaging the world’s problems like crazy, and the top 1-2% of people are more than capable of that (most jobs that need doing in EA don’t require top 0.1%).
4. EA is too PR focused—to the point where it actually backfires spectacularly and now there is lots of bad press [big example: SBF’s bad character being known about but not addressed].
5. Despite all it’s flaws, EA is good (and much better than the alternatives in most cases).
[Separating out this paragraph into a new comment as I’m guessing it’s what lead to the downvotes, and I’d quite like the point of the parent paragraph to stand alone. Not sure if anyone will see this now though.]
I think it’s imperative to get the leaders of AGI companies to realise that they are in a suicide race (and that AGI will likely kill them too). The default outcome of AGI is doom. For extinction risk at the 1% level, it seems reasonable (even though it’s still 80M lives in expectation) to pull the trigger on AGI for a 99% chance of utopia. This is totally wrong-headed and is arguably contributing massively to current x-risk.
Also, in general I’m personally much more sceptical of such a moonshot paying off, given shorter timelines and the possibility that x-safety from ASI may well be impossible. I think OP was 2022′s best idea for AI Safety. 2024′s is PauseAI.
People from those orgs were aware, but none were keen enough about the idea to go as far as attempting a pilot run (e.g. the 2 week retreat idea). I think general downside risk aversion was probably a factor. This was in the pre-chatGPT days of a much narrower Overton Window though, so maybe it’s time for the idea to be revived? On the other hand, maybe it’s much less needed now there is government involvement, and national AI Safety Institutes attracting top talent.
I see in your comment on that post, you say “human extinction would not necessarily be an existential catastrophe” and “So, if advanced AI, as the most powerful entity on Earth, were to cause human extinction, I guess existential risk would be negligible on priors?”. To be clear: what I’m interested in here is human extinction (not any broader conception of “existential catastrophe”), and the bet is about that.