To my knowledge it doesn’t meet the “Was motivated or instigated by EA” criterion, since Nick had been developing those ideas since well before the EA movement started. I guess he might have gotten EA money while writing the book, but even if that’s the case it doesn’t feel like a central example of what I’m interested in.
That moves the question to, why not count Nick as part of EA itself? :) It seems reasonable to count him given that he wrote Astronomical Waste which seems to be part of EA’s intellectual foundations, and he personally seems to be very interested in doing altruism effectively.
Or maybe you can explain more your motivations for writing this post, which would help me understand how best to interpret it.
Let me try answer the latter question (and thanks for pushing me to flesh out my vague ideas more!) One very brief way you could describe the development of AI safety is something like “A few transhumanists came up with some key ideas and wrote many blog posts. The rationalist movement formed from those following these things online, and made further contributions. Then the EA movement formed, and while it was originally focused on causes like global poverty, over time did a bunch of investigative work which led many EAs to become convinced that AI safety matters, and to start working on it, directly or indirectly (or to gain skills with the intent of doing such work).”
The three questions I am ultimately trying to answer are: a) how valuable is it to build up the EA movement? b) how much should I update when I learn that a given belief is a consensus in EA? and c) how much evidence do the opinions of other people provide in favour of AI safety being important?
To answer the first question, assuming that analysis of AI safety as a cause area is valuable, I should focus on contributions by people who were motivated or instigated by the EA movement itself. Here Nick doesn’t count (except insofar as EA made his book come out sooner or better).
To answer the second question, it helps to know whether the focus on AI safety in EA came about because many people did comprehensive due diligence and shared their findings, or whether there wasn’t much investigation and the ubiquity of the belief was driven via an information cascade. For this purpose, I should count work by people to the extent that they or people like them are likely to critically investigate other beliefs that are or will become widespread in EA. Being motivated to investigate AI safety by membership in the EA movement is the best evidence, but for the purpose of answering this question I probably should have used “motivated by the EA movement or motivated by very similar things to what EAs are motivated by”, and should partially count Nick.
To answer the third question, it helps to know whether the people who have become convinced that AI safety is important are a relatively homogenous group who might all have highly correlated biases and hidden motivations, or whether a wide range of people have become convinced. For this purpose, I should count work by people to the extent that they are dissimilar to the transhumanists and rationalists who came up with the original safety arguments, and also to the extent that they rederived the arguments for themselves rather than being influenced by the existing arguments. Here EAs who started off not being inclined towards transhumanism or rationalism at all count the most, and Nick counts very little.
Note that Nick is quite an outlier though, so while I’m using him as an illustrative example, I’d prefer engagement on the general points rather than this example in particular.
Then the EA movement formed, and while it was originally focused on causes like global poverty, over time did a bunch of investigative work which led many EAs to become convinced that AI safety matters, and to start working on it directly or indirectly.
Is this a statement that you’re endorsing, or is it part of what you’re questioning? Are you aware of any surveys or any other evidence supporting this? (I’d accept “most people in AI safety that I know started working in it because EA investigative work convinced them that AI safety matters” or something of that nature.)
b) how much should I update when I learn that a given belief is a consensus in EA?
Why are you trying to answer this, instead of “How should I update, given the results of all available investigations into AI safety as a cause area?” In other words, what is the point of dividing such investigations into “EA” and “not EA”, if in the end you just want to update on all of them to arrive at a posterior? Oh, is it because if a non-EA concludes that AI safety is not a worthwhile cause, it might just be because they don’t care much about the far future, so EA investigations are more relevant? But if so, why only “partially count” Nick?
Here EAs who started off not being inclined towards transhumanism or rationalism at all count the most, and Nick counts very little.
For this question then, it seems that Paul Christiano also needs to be discounted (and possibly others as well but I’m not as familiar with them).
Are you aware of any surveys or any other evidence supporting this? (I’d accept “most people in AI safety that I know started working in it because EA investigative work convinced them that AI safety matters” or something of that nature.)
I’m endorsing this, and I’m confused about which part you’re skeptical about. Is it the “many EAs” bit? Obviously the word “many” is pretty fuzzy, and I don’t intend it to be a strong claim. Mentally the numbers I’m thinking of are something like >50 people or >25% of committed (or “core”, whatever that means) EAs. Don’t have a survey to back that up though. Oh, I guess I’m also including people currently studying ML with the intention of doing safety. Will edit to add that.
Why are you trying to answer this, instead of “How should I update, given the results of all available investigations into AI safety as a cause area?”
There are other questions that I would like answers to, not related to AI safety, and if I trusted EA consensus, then that would make the process much easier.
For this question then, it seems that Paul Christiano also needs to be discounted (and possibly others as well but I’m not as familiar with them).
I’m endorsing this, and I’m confused about which part you’re skeptical about.
I think I interpreted your statement as saying something like most people in AI safety are EAs, because you started with “One very brief way you could describe the development of AI safety” which I guess made me think that maybe you consider this to be the main story of AI safety so far, or you thought other people considered this to be the main story of AI safety so far and you wanted to push against that perception. Sorry for reading too much / the wrong thing into it.
There are other questions that I would like answers to, not related to AI safety, and if I trusted EA consensus, then that would make the process much easier.
Ok I see. But there may not be that much correlation between the trustworthiness of EA consensus on different topics. It could easily be the case that EA has done a lot of good investigations on AI safety but very little or poor quality investigations on other topics. It seems like it wouldn’t be that hard to just look at the actual investigations for each topic, rather than rely on some sense of whether EA consensus is overall trustworthy.
Why doesn’t Superintelligence count?
To my knowledge it doesn’t meet the “Was motivated or instigated by EA” criterion, since Nick had been developing those ideas since well before the EA movement started. I guess he might have gotten EA money while writing the book, but even if that’s the case it doesn’t feel like a central example of what I’m interested in.
That moves the question to, why not count Nick as part of EA itself? :) It seems reasonable to count him given that he wrote Astronomical Waste which seems to be part of EA’s intellectual foundations, and he personally seems to be very interested in doing altruism effectively.
Or maybe you can explain more your motivations for writing this post, which would help me understand how best to interpret it.
Let me try answer the latter question (and thanks for pushing me to flesh out my vague ideas more!) One very brief way you could describe the development of AI safety is something like “A few transhumanists came up with some key ideas and wrote many blog posts. The rationalist movement formed from those following these things online, and made further contributions. Then the EA movement formed, and while it was originally focused on causes like global poverty, over time did a bunch of investigative work which led many EAs to become convinced that AI safety matters, and to start working on it, directly or indirectly (or to gain skills with the intent of doing such work).”
The three questions I am ultimately trying to answer are: a) how valuable is it to build up the EA movement? b) how much should I update when I learn that a given belief is a consensus in EA? and c) how much evidence do the opinions of other people provide in favour of AI safety being important?
To answer the first question, assuming that analysis of AI safety as a cause area is valuable, I should focus on contributions by people who were motivated or instigated by the EA movement itself. Here Nick doesn’t count (except insofar as EA made his book come out sooner or better).
To answer the second question, it helps to know whether the focus on AI safety in EA came about because many people did comprehensive due diligence and shared their findings, or whether there wasn’t much investigation and the ubiquity of the belief was driven via an information cascade. For this purpose, I should count work by people to the extent that they or people like them are likely to critically investigate other beliefs that are or will become widespread in EA. Being motivated to investigate AI safety by membership in the EA movement is the best evidence, but for the purpose of answering this question I probably should have used “motivated by the EA movement or motivated by very similar things to what EAs are motivated by”, and should partially count Nick.
To answer the third question, it helps to know whether the people who have become convinced that AI safety is important are a relatively homogenous group who might all have highly correlated biases and hidden motivations, or whether a wide range of people have become convinced. For this purpose, I should count work by people to the extent that they are dissimilar to the transhumanists and rationalists who came up with the original safety arguments, and also to the extent that they rederived the arguments for themselves rather than being influenced by the existing arguments. Here EAs who started off not being inclined towards transhumanism or rationalism at all count the most, and Nick counts very little.
Note that Nick is quite an outlier though, so while I’m using him as an illustrative example, I’d prefer engagement on the general points rather than this example in particular.
Is this a statement that you’re endorsing, or is it part of what you’re questioning? Are you aware of any surveys or any other evidence supporting this? (I’d accept “most people in AI safety that I know started working in it because EA investigative work convinced them that AI safety matters” or something of that nature.)
Why are you trying to answer this, instead of “How should I update, given the results of all available investigations into AI safety as a cause area?” In other words, what is the point of dividing such investigations into “EA” and “not EA”, if in the end you just want to update on all of them to arrive at a posterior? Oh, is it because if a non-EA concludes that AI safety is not a worthwhile cause, it might just be because they don’t care much about the far future, so EA investigations are more relevant? But if so, why only “partially count” Nick?
For this question then, it seems that Paul Christiano also needs to be discounted (and possibly others as well but I’m not as familiar with them).
I’m endorsing this, and I’m confused about which part you’re skeptical about. Is it the “many EAs” bit? Obviously the word “many” is pretty fuzzy, and I don’t intend it to be a strong claim. Mentally the numbers I’m thinking of are something like >50 people or >25% of committed (or “core”, whatever that means) EAs. Don’t have a survey to back that up though. Oh, I guess I’m also including people currently studying ML with the intention of doing safety. Will edit to add that.
There are other questions that I would like answers to, not related to AI safety, and if I trusted EA consensus, then that would make the process much easier.
Indeed, I agree.
I think I interpreted your statement as saying something like most people in AI safety are EAs, because you started with “One very brief way you could describe the development of AI safety” which I guess made me think that maybe you consider this to be the main story of AI safety so far, or you thought other people considered this to be the main story of AI safety so far and you wanted to push against that perception. Sorry for reading too much / the wrong thing into it.
Ok I see. But there may not be that much correlation between the trustworthiness of EA consensus on different topics. It could easily be the case that EA has done a lot of good investigations on AI safety but very little or poor quality investigations on other topics. It seems like it wouldn’t be that hard to just look at the actual investigations for each topic, rather than rely on some sense of whether EA consensus is overall trustworthy.