Some additional rough notes that didn’t make it into the post
Maybe another goal in the category of “gaining indirect benefits for other EA/longtermist goals” could be having good feedback loops (e.g. on our methods for influencing policy and how effective we are at that) that let us learn things relevant to other areas too?
Similar to what Open Phil have said about some of their non-longtermist work
One reviewer said “Maybe place more emphasis on 1b? For example, after even a limited nuclear exchange between say China and the US, getting cooperation on AI development seems really hard.” I replied:
“That seems plausible and worth looking into, but unsure why to be confident on it?
The default path looks like low cooperation on AI, I think. And both the League of Nations and the UN were formed after big scary wars. Those two points together make it seem like >20% likely that, after 2 months of research, I’d conclude that a US-China nuclear exchange at least 5 years before TAI development will increase cooperation between those countries, in expectation, rather than decrease
Does that sound incorrect to you? (Genuine, non-rhetorical question. I feel confused about why other people feel more confident on this than me, so maybe I’m missing something.)
It feels to me like confidence on this is decently likely to be in part motivated reasoning / spurious convergence? (Though also one should in general be cautious about alleging bias.)”
Regarding alternative fields we could work in for the same benefits mentioned in section 2, one reviewer said “I think one of the questions here is do these topics have the features of both plausibly an x risk or x risk factor (even if the exact chance is v low or uncertain) AND provides these learning opportunities. I think this is actually plausible with cybersecurity, less sure on the others.” I replied:
“Why do you want the first feature?
Because that will add direct value, which (when added to instrumental value) can tip working on this over the line to being worthwhile?
Or because you think we get more instrumental value if there’s plausible x-risk from the thing? If so, I’d be keen to hear more on why
I have both intuitions myself, but haven’t got them clearly worked out in my head and would be interested in other people’s views”
Regarding 2c, one reviewer said:
“my intuition on this:
it seems like many things in the world are about as relevant to AI/bio issues as nuclear risk is. So it seems sort of improbable that adding a few reports worth of analysis of nuke risk on top of the huge pile of writings that are already relevant for AI/bio would be a significant contribution.
On the other hand, it seems much more plausible that it could be significant for some EAs to learn all about the nuclear situation. Because there isn’t already a ton of people in the world who know a lot about the nuclear situation, who also know enough about AI/bio to draw relevant connections, who are also willing to talk with EAs about that”
Some additional rough notes that didn’t make it into the post
Maybe another goal in the category of “gaining indirect benefits for other EA/longtermist goals” could be having good feedback loops (e.g. on our methods for influencing policy and how effective we are at that) that let us learn things relevant to other areas too?
Similar to what Open Phil have said about some of their non-longtermist work
One reviewer said “Maybe place more emphasis on 1b? For example, after even a limited nuclear exchange between say China and the US, getting cooperation on AI development seems really hard.” I replied:
“That seems plausible and worth looking into, but unsure why to be confident on it?
The default path looks like low cooperation on AI, I think. And both the League of Nations and the UN were formed after big scary wars. Those two points together make it seem like >20% likely that, after 2 months of research, I’d conclude that a US-China nuclear exchange at least 5 years before TAI development will increase cooperation between those countries, in expectation, rather than decrease
Does that sound incorrect to you?
(Genuine, non-rhetorical question. I feel confused about why other people feel more confident on this than me, so maybe I’m missing something.)
It feels to me like confidence on this is decently likely to be in part motivated reasoning / spurious convergence? (Though also one should in general be cautious about alleging bias.)”
Regarding alternative fields we could work in for the same benefits mentioned in section 2, one reviewer said “I think one of the questions here is do these topics have the features of both plausibly an x risk or x risk factor (even if the exact chance is v low or uncertain) AND provides these learning opportunities. I think this is actually plausible with cybersecurity, less sure on the others.” I replied:
“Why do you want the first feature?
Because that will add direct value, which (when added to instrumental value) can tip working on this over the line to being worthwhile?
Or because you think we get more instrumental value if there’s plausible x-risk from the thing? If so, I’d be keen to hear more on why
I have both intuitions myself, but haven’t got them clearly worked out in my head and would be interested in other people’s views”
Regarding 2c, one reviewer said:
“my intuition on this:
it seems like many things in the world are about as relevant to AI/bio issues as nuclear risk is. So it seems sort of improbable that adding a few reports worth of analysis of nuke risk on top of the huge pile of writings that are already relevant for AI/bio would be a significant contribution.
On the other hand, it seems much more plausible that it could be significant for some EAs to learn all about the nuclear situation. Because there isn’t already a ton of people in the world who know a lot about the nuclear situation, who also know enough about AI/bio to draw relevant connections, who are also willing to talk with EAs about that”