Some additional rough notes that didnāt make it into the post
Maybe another goal in the category of āgaining indirect benefits for other EA/ālongtermist goalsā could be having good feedback loops (e.g. on our methods for influencing policy and how effective we are at that) that let us learn things relevant to other areas too?
Similar to what Open Phil have said about some of their non-longtermist work
One reviewer said āMaybe place more emphasis on 1b? For example, after even a limited nuclear exchange between say China and the US, getting cooperation on AI development seems really hard.ā I replied:
āThat seems plausible and worth looking into, but unsure why to be confident on it?
The default path looks like low cooperation on AI, I think. And both the League of Nations and the UN were formed after big scary wars. Those two points together make it seem like >20% likely that, after 2 months of research, Iād conclude that a US-China nuclear exchange at least 5 years before TAI development will increase cooperation between those countries, in expectation, rather than decrease
Does that sound incorrect to you? (Genuine, non-rhetorical question. I feel confused about why other people feel more confident on this than me, so maybe Iām missing something.)
It feels to me like confidence on this is decently likely to be in part motivated reasoning /ā spurious convergence? (Though also one should in general be cautious about alleging bias.)ā
Regarding alternative fields we could work in for the same benefits mentioned in section 2, one reviewer said āI think one of the questions here is do these topics have the features of both plausibly an x risk or x risk factor (even if the exact chance is v low or uncertain) AND provides these learning opportunities. I think this is actually plausible with cybersecurity, less sure on the others.ā I replied:
āWhy do you want the first feature?
Because that will add direct value, which (when added to instrumental value) can tip working on this over the line to being worthwhile?
Or because you think we get more instrumental value if thereās plausible x-risk from the thing? If so, Iād be keen to hear more on why
I have both intuitions myself, but havenāt got them clearly worked out in my head and would be interested in other peopleās viewsā
Regarding 2c, one reviewer said:
āmy intuition on this:
it seems like many things in the world are about as relevant to AI/ābio issues as nuclear risk is. So it seems sort of improbable that adding a few reports worth of analysis of nuke risk on top of the huge pile of writings that are already relevant for AI/ābio would be a significant contribution.
On the other hand, it seems much more plausible that it could be significant for some EAs to learn all about the nuclear situation. Because there isnāt already a ton of people in the world who know a lot about the nuclear situation, who also know enough about AI/ābio to draw relevant connections, who are also willing to talk with EAs about thatā
Some additional rough notes that didnāt make it into the post
Maybe another goal in the category of āgaining indirect benefits for other EA/ālongtermist goalsā could be having good feedback loops (e.g. on our methods for influencing policy and how effective we are at that) that let us learn things relevant to other areas too?
Similar to what Open Phil have said about some of their non-longtermist work
One reviewer said āMaybe place more emphasis on 1b? For example, after even a limited nuclear exchange between say China and the US, getting cooperation on AI development seems really hard.ā I replied:
āThat seems plausible and worth looking into, but unsure why to be confident on it?
The default path looks like low cooperation on AI, I think. And both the League of Nations and the UN were formed after big scary wars. Those two points together make it seem like >20% likely that, after 2 months of research, Iād conclude that a US-China nuclear exchange at least 5 years before TAI development will increase cooperation between those countries, in expectation, rather than decrease
Does that sound incorrect to you?
(Genuine, non-rhetorical question. I feel confused about why other people feel more confident on this than me, so maybe Iām missing something.)
It feels to me like confidence on this is decently likely to be in part motivated reasoning /ā spurious convergence? (Though also one should in general be cautious about alleging bias.)ā
Regarding alternative fields we could work in for the same benefits mentioned in section 2, one reviewer said āI think one of the questions here is do these topics have the features of both plausibly an x risk or x risk factor (even if the exact chance is v low or uncertain) AND provides these learning opportunities. I think this is actually plausible with cybersecurity, less sure on the others.ā I replied:
āWhy do you want the first feature?
Because that will add direct value, which (when added to instrumental value) can tip working on this over the line to being worthwhile?
Or because you think we get more instrumental value if thereās plausible x-risk from the thing? If so, Iād be keen to hear more on why
I have both intuitions myself, but havenāt got them clearly worked out in my head and would be interested in other peopleās viewsā
Regarding 2c, one reviewer said:
āmy intuition on this:
it seems like many things in the world are about as relevant to AI/ābio issues as nuclear risk is. So it seems sort of improbable that adding a few reports worth of analysis of nuke risk on top of the huge pile of writings that are already relevant for AI/ābio would be a significant contribution.
On the other hand, it seems much more plausible that it could be significant for some EAs to learn all about the nuclear situation. Because there isnāt already a ton of people in the world who know a lot about the nuclear situation, who also know enough about AI/ābio to draw relevant connections, who are also willing to talk with EAs about thatā