P(misalignment x-risk|AGI) is between 7% 0% to 35%
AGI will be developed by January 1, 2100 is between 30% 0% and 60%
I’m not adding the “by 2043” section:
because it is too complicated for me to currently think about clearly so I don’t think I’d be a good discussion partner, but I’d appreciate help there too
My current opinion
Is that we’re almost certainly doomed (80%? more?), I can’t really see a way out before 2100 except for something like civilizational collapse.
My epistemic status
I’m not sure, I’m not FTX.
Pitch: You will be doing a good thing if you change my mind
You will help me decide whether to work on AI Safety, and if I do, I’ll have better models to do it with. If I don’t, I’ll go back to focusing on the other projects I’m up to. I’m a bit isolated (I live in Israel), and talking to people from the international community who can help me not get stuck in my current opinions could really help me.
Technicalities
How to talk to me? I think the best would be to comment here so our discussion will be online and people can push back, but there are more contact methods in my profile. I don’t officially promise to talk to anyone, but I do expect to. If you want me to read an article, I do prefer audio. I’m also happy with voice messages.
Paying up to $100 in total. By my decision (just to have some formality), but feel free to ask for something else or something more specific. (Sorry I’m not as rich as FTX)
I also expect/guess my disagreement with many people would be around our priors, not the specifics. I think many people have a prior of “I’m not sure, and so let’s assume we won’t all die”, which seems wrong, but I’m open to talk.
I think most of the work with changing each other’s mind will be locating the crux (as I suggested FTX would help us do with them).
$100 to change my mind to FTX’s views
If you change my mind to any of:
P(misalignment x-risk|AGI) is between
7%0% to 35%AGI will be developed by January 1, 2100 is between
30%0% and 60%I’m not adding the “by 2043” section:
because it is too complicated for me to currently think about clearly so I don’t think I’d be a good discussion partner, but I’d appreciate help there too
My current opinion
Is that we’re almost certainly doomed (80%? more?), I can’t really see a way out before 2100 except for something like civilizational collapse.
My epistemic status
I’m not sure, I’m not FTX.
Pitch: You will be doing a good thing if you change my mind
You will help me decide whether to work on AI Safety, and if I do, I’ll have better models to do it with. If I don’t, I’ll go back to focusing on the other projects I’m up to. I’m a bit isolated (I live in Israel), and talking to people from the international community who can help me not get stuck in my current opinions could really help me.
Technicalities
How to talk to me? I think the best would be to comment here so our discussion will be online and people can push back, but there are more contact methods in my profile. I don’t officially promise to talk to anyone, but I do expect to. If you want me to read an article, I do prefer audio. I’m also happy with voice messages.
Paying up to $100 in total. By my decision (just to have some formality), but feel free to ask for something else or something more specific. (Sorry I’m not as rich as FTX)
AMA
Replying to a DM:
My current priors are roughly represented by AGI ruin scenarios are likely (and disjunctive).
I also expect/guess my disagreement with many people would be around our priors, not the specifics. I think many people have a prior of “I’m not sure, and so let’s assume we won’t all die”, which seems wrong, but I’m open to talk.
I think most of the work with changing each other’s mind will be locating the crux (as I suggested FTX would help us do with them).
I’m willing to discuss this over Zoom, or face to face once I return to Israel in November.
What I think my main points are:
We don’t seem to be anywhere near AGI. The amount of compute might very soon be enough but we also need major theoretical breakthroughs.
Most extinction scenarios that I’ve read about or thought about require some amount of bad luck, at least if AGI is born out of the ML paradigm
AGI is poorly defined, so it’s hard to reason on what it would do once it comes into existence, of you could even describe that as a binary event
It seems unlikely that a malignant AI succeeds in deceiving us until it is capable of preventing us from shutting it off
I’m not entirely convinced in any of them—I haven’t thought about this carefully.
Edit: there’s a doom scenario that I’m more worried about, and it doesn’t require AGI—and that’s global domination by a tyrannical government.
For transparency:
I’m discussing this with Andrew Timm here.
(But please don’t let this stop you from opening another conversation with me)