I think at this point, we are not far off this being
“Should you work at a leading oil company? (including in non-renewables roles)”.
Or even
“Hans Bethe has just calculated that the chance of the first A-bomb test igniting the atmosphere is 10%; should you work at the Manhattan Project? (including in non-shutting-it-down roles)”.
EA has already contributed massively to the safety-washing of the big AI companies (not to mention kicking off and accelerating the race toward AGI in the first place!) I think EAs should be focusing more on applying external pressure now. There are ways to have higher leverage on existential safety by joining (not yet captured) AI governance, lobbying and publiccampaigningefforts.
Thanks, this is an interesting heuristic, but I think I don’t find it as valuable as you do.
First, while I do think it’d probably be harmful in expectation to work at leading oil companies / at the Manhattan project, I’m not confident in that view — I just haven’t thought about this very much.
Second, I think that AI labs are in a pretty different reference class from oil companies and the development of nuclear weapons.
Why? Roughly:
Whether, in a broad sense, capabilities advances are good or bad is pretty unclear. (Note some capabilities advances in particular areas are very clearly harmful.) In comparison, I do think that, in a broad sense, the development of nuclear weapons, and the release of greenhouse gases are harmful.
Unlike with oil companies and the Manhattan Project, I think that there’s a good chance that a leading, careful AI project could be a huge force for good, substantially reducing existential risk — and so it seems weird not to consider working at what could be one of the world’s most (positively) impactful organisations. Of course, you should also consider the chance that the organisation could be one of the world’s most negatively impactful organisations.
I think that there’s a good chance that a leading, careful AI project could be a huge force for good, substantially reducing existential risk
I think the burden of proof should be on the big AI companies to show that this is actually a possibility. Because right now, the technology, as based on the current paradigm, looks like it’s fundamentallyuncontrollable.
I don’t think any of the big AI labs have overcome that prior, but I also have the prior that their safety plans don’t even make sense theoretically—hence the “burden of proof” is on them to show that it is possible to align the kind of AI they are building. Another thing pointing in the opposite direction.
I think at this point, we are not far off this being
“Should you work at a leading oil company? (including in non-renewables roles)”.
Or even
“Hans Bethe has just calculated that the chance of the first A-bomb test igniting the atmosphere is 10%; should you work at the Manhattan Project? (including in non-shutting-it-down roles)”.
EA has already contributed massively to the safety-washing of the big AI companies (not to mention kicking off and accelerating the race toward AGI in the first place!) I think EAs should be focusing more on applying external pressure now. There are ways to have higher leverage on existential safety by joining (not yet captured) AI governance, lobbying and public campaigning efforts.
Thanks, this is an interesting heuristic, but I think I don’t find it as valuable as you do.
First, while I do think it’d probably be harmful in expectation to work at leading oil companies / at the Manhattan project, I’m not confident in that view — I just haven’t thought about this very much.
Second, I think that AI labs are in a pretty different reference class from oil companies and the development of nuclear weapons.
Why? Roughly:
Whether, in a broad sense, capabilities advances are good or bad is pretty unclear. (Note some capabilities advances in particular areas are very clearly harmful.) In comparison, I do think that, in a broad sense, the development of nuclear weapons, and the release of greenhouse gases are harmful.
Unlike with oil companies and the Manhattan Project, I think that there’s a good chance that a leading, careful AI project could be a huge force for good, substantially reducing existential risk — and so it seems weird not to consider working at what could be one of the world’s most (positively) impactful organisations. Of course, you should also consider the chance that the organisation could be one of the world’s most negatively impactful organisations.
Because these issues are difficult and we don’t think we have all the answers, I also published a range of opinions about a related question in our anonymous advice series. Some of the respondents took a very sceptical view of any work that advances capabilities, but others disagreed.
I think the burden of proof should be on the big AI companies to show that this is actually a possibility. Because right now, the technology, as based on the current paradigm, looks like it’s fundamentally uncontrollable.
TL;DR: I don’t like talking about “burden of proof”
I prefer talking about “priors”.
Seems like you ( @Greg_Colbourn ) have priors that AI labs will cause damage, and I’d assume @Benjamin Hilton would agree with that?
I also guess you both have priors that ~random (average) capabilities research will be net negative?
If so, I suggest we should ask if the AI lab (or the specific capabilities research) has overcome that prior somehow.
wdyt?
I don’t think any of the big AI labs have overcome that prior, but I also have the prior that their safety plans don’t even make sense theoretically—hence the “burden of proof” is on them to show that it is possible to align the kind of AI they are building. Another thing pointing in the opposite direction.
Whoever downvoted this, I’d really prefer if you tell me why
You can do it anonymously:
https://docs.google.com/forms/d/e/1FAIpQLSca6NOTbFMU9BBQBYHecUfjPsxhGbzzlFO5BNNR1AIXZjpvcw/viewform