A judgement I’m attached to is that a person is either extremely confused or callous if they work in capabilities at a big lab. Is there some nuance I’m missing here?
Some of them have low p(doom from AI) and aren’t longtermists, which justifies the high-level decision of working on an existentially dangerous technology with sufficiently large benefits.
I do think their actual specific actions are not commensurate with the level of importance or moral seriousness that they claim to attach to their work, given those stated motivations.
Thanks for the comment Linch! Just to spell this out:
“Some of them have low p(doom from AI) and aren’t longtermists, which justifies the high-level decision of working on an existentially dangerous technology with sufficiently large benefits. ”
I would consider this acceptable if their p(doom) was ~ < 0.01%
I find this pretty incredulous TBH
“I do think their actual specific actions are not commensurate with the level of importance or moral seriousness that they claim to attach to their work, given those stated motivations.”
I’m a bit confused by this part. Are you saying you believe the importance / seriousness the person claims their work has is not reflected in the actions they actually take? In what way are you saying they do this?
I would consider this acceptable if their p(doom) was ~ < 0.01%
I find this pretty incredulous TBH
I dunno man, at least some people think that AGI is the best path to curing cancer. That seems like a big deal! If you aren’t a longtermist at all, speeding up the cure for cancer by a year is probably worth quite a bit of x-risk.
Are you saying you believe the importance / seriousness the person claims their work has is not reflected in the actions they actually take?
Yes.
In what way are you saying they do this?
Lab people shitpost, lab people take competition extremely seriously (if your actual objective is “curing cancer,” it seems a bit discordant to be that worried that Silicon Valley Startup #2 will cure cancer before you), people take infosec not at all seriously, the bizarre cultish behavior after the Sam Altman firing, and so forth.
In my experience, they’re mostly just impulsive and have already written their bottom line (“I want to work on AI projects that look cool to me”), and after that they come up with excuses to justify this to themselves.
Sorry, I will explain further for clarity. I am Buddhist and in that philosophy the phrase “confused” has an esoteric usage. E.g. “that person suffers because they’re confused about the nature of reality”. I think this lab person is confused about how much suffering they’re going to create for themselves and others (i.e. it hasn’t been seen or internalised).
Yes. I think most people working on capabilities at leading labs are confused or callous (or something similar, like greedy or delusional), but definitely not all. And personally, I very much hope there are many safety-concerned people working on capabilities at big labs, and am concerned about the most safety-concerned people feeling the most pressure to leave, leading to evaporative cooling.
Reasons to work on capabilities at a large lab:
To build career capital of the kind that will allow you to have a positive impact later. E.g. to be offered relevant positions in government
To positively influence the culture of capabilities teams or leadership at labs.
To be willing and able to whistleblow bad situations (e.g. seeing emerging dangerous capabilities in new models, the non-disparagement stuff).
[maybe] to earn to give (especially if you don’t think you’re contributing to core capabilities)
To be clear, I expect achieving the above to be infeasible for most people, and it’s important for people to not delude themselves into thinking they’re having a positive impact to keep enjoying a lucrative, exciting job. But I definitely think there are people for whom the above is feasible and extremely important.
Another way to phrase the question is “is it good for all safety-concerned people to shun capabilities teams, given (as seems to be the case) that those teams will continue to exist and make progress by default?” And for me the strong answer is “yes”. Which is totally consistent with wanting labs to pause and thinking that just contributing to capabilities (on frontier models) in expectation is extremely destructive.
Thanks so much for your thoughtful comment! I appreciate someone engaging with me on this rather than just disagree ticking. Some thoughts:
1. Build Career Capital for Later Impact:
I think this depends somewhat on what your AGI timelines are. If they’re short, you’ve wasted your time and possibly had a net-negative effect.
I think there is a massive risk of people entering an AGI lab for career capital building reasons, and then post rationalising their decision to stay. Action changes attitudes faster than attitude changes actions after all.
2. Influence the Culture:
Multiple board members attempted this at OpenAI got fired, what chance does a single lower level employee?
I have tried changing corporate culture at multiple organisations and it is somewhere between extremely hard and impossible (more on that below).
3. Be Prepared to Whistle-Blow:
I was a Whistle-Blower during my time at News Corp. It was extremely difficult. I simply do not expect a meaningful number of people to be able to do this. You have to be willing to turn your life upside down.
This can be terrible for a person’s mental health. We shouldn’t be vary careful openly recommending this as a reason to stay at a Lab.
As mentioned above, I expect a greater number of people to turn into converts than would-be Whistle-Blowers
I think if you’re the kind of person who goes into a Lab for Career Capital that makes you less likely to be a Whistle-Blower TBH.
(1) I agree if your timelines are super short, like <2yrs, it’s probably not worth it. I have a bunch of probability mass on longer timelines, though some on really short ones
Re (2), my sense is some employees already have had some of this effect (and many don’t. But some do). I think board members are terrible candidates for changing org culture; they have unrelated full-time jobs, they don’t work from the office, they have different backgrounds, most people don’t have cause to interact with them much. People who are full-time, work together with people all day every day, know the context, etc., seem more likely to be effective at this (and indeed, I think they have been, to some extent in some cases)
Re (3), seems like a bunch of OAI people have blown the whistle on bad behavior already, so the track record is pretty great, and I think them doing that has been super valuable. And 1 whistleblower seems much better than several converts is bad. I agree it can be terrible for mental health for some people, and people should take care of themselves.
Re (4), um, this is the EA Forum, we care about how good the money is. Besides crypto, I don’t think there are many for many of the relevant people to make similar amounts of money on similar timeframes. Actually I think working at a lab early was an effective way to make money. A bunch of safety-concerned people for example have equity worth several millions to tens of millions, more than I think they could have easily earned elsewhere, and some are now billionaires on paper. And if AI has the transformative impact on the economy we expect, that could be worth way more (and it being worth more is correlated with it being needed more, so extra valuable); we are talking about the most valuable/powerful industry the world has ever known here, hard to beat that for making money. I don’t think that makes it okay to lead large AI labs, but for joining early, especially doing some capabilities work that doesn’t push the most risky capabilities along much, I don’t think it’s obvious.
I agree that there are various risks related to staying too long, rationalizing, being greedy, etc., and in most cases I wouldn’t advice a safety-concerned person to do capabilities. But I think you’re being substantially too intense about the risk of speeding up AI relative to the benefits of seeing what’s happening on the inside, which seem like they’ve already been very substantial
A judgement I’m attached to is that a person is either extremely confused or callous if they work in capabilities at a big lab. Is there some nuance I’m missing here?
Some of them have low p(doom from AI) and aren’t longtermists, which justifies the high-level decision of working on an existentially dangerous technology with sufficiently large benefits.
I do think their actual specific actions are not commensurate with the level of importance or moral seriousness that they claim to attach to their work, given those stated motivations.
Thanks for the comment Linch! Just to spell this out:
“Some of them have low p(doom from AI) and aren’t longtermists, which justifies the high-level decision of working on an existentially dangerous technology with sufficiently large benefits. ”
I would consider this acceptable if their p(doom) was ~ < 0.01%
I find this pretty incredulous TBH
“I do think their actual specific actions are not commensurate with the level of importance or moral seriousness that they claim to attach to their work, given those stated motivations.”
I’m a bit confused by this part. Are you saying you believe the importance / seriousness the person claims their work has is not reflected in the actions they actually take? In what way are you saying they do this?
I dunno man, at least some people think that AGI is the best path to curing cancer. That seems like a big deal! If you aren’t a longtermist at all, speeding up the cure for cancer by a year is probably worth quite a bit of x-risk.
Yes.
Lab people shitpost, lab people take competition extremely seriously (if your actual objective is “curing cancer,” it seems a bit discordant to be that worried that Silicon Valley Startup #2 will cure cancer before you), people take infosec not at all seriously, the bizarre cultish behavior after the Sam Altman firing, and so forth.
In my experience, they’re mostly just impulsive and have already written their bottom line (“I want to work on AI projects that look cool to me”), and after that they come up with excuses to justify this to themselves.
Hi! Thanks for your comment :)
I basically file this under “confused”.
Interesting, in your ontology I’d much more straightforwardly characterize it as “callous”
Sorry, I will explain further for clarity. I am Buddhist and in that philosophy the phrase “confused” has an esoteric usage. E.g. “that person suffers because they’re confused about the nature of reality”. I think this lab person is confused about how much suffering they’re going to create for themselves and others (i.e. it hasn’t been seen or internalised).
Yes. I think most people working on capabilities at leading labs are confused or callous (or something similar, like greedy or delusional), but definitely not all. And personally, I very much hope there are many safety-concerned people working on capabilities at big labs, and am concerned about the most safety-concerned people feeling the most pressure to leave, leading to evaporative cooling.
Reasons to work on capabilities at a large lab:
To build career capital of the kind that will allow you to have a positive impact later. E.g. to be offered relevant positions in government
To positively influence the culture of capabilities teams or leadership at labs.
To be willing and able to whistleblow bad situations (e.g. seeing emerging dangerous capabilities in new models, the non-disparagement stuff).
[maybe] to earn to give (especially if you don’t think you’re contributing to core capabilities)
To be clear, I expect achieving the above to be infeasible for most people, and it’s important for people to not delude themselves into thinking they’re having a positive impact to keep enjoying a lucrative, exciting job. But I definitely think there are people for whom the above is feasible and extremely important.
Another way to phrase the question is “is it good for all safety-concerned people to shun capabilities teams, given (as seems to be the case) that those teams will continue to exist and make progress by default?” And for me the strong answer is “yes”. Which is totally consistent with wanting labs to pause and thinking that just contributing to capabilities (on frontier models) in expectation is extremely destructive.
Thanks so much for your thoughtful comment! I appreciate someone engaging with me on this rather than just disagree ticking. Some thoughts:
1. Build Career Capital for Later Impact:
I think this depends somewhat on what your AGI timelines are. If they’re short, you’ve wasted your time and possibly had a net-negative effect.
I think there is a massive risk of people entering an AGI lab for career capital building reasons, and then post rationalising their decision to stay. Action changes attitudes faster than attitude changes actions after all.
2. Influence the Culture:
Multiple board members attempted this at OpenAI got fired, what chance does a single lower level employee?
I have tried changing corporate culture at multiple organisations and it is somewhere between extremely hard and impossible (more on that below).
3. Be Prepared to Whistle-Blow:
I was a Whistle-Blower during my time at News Corp. It was extremely difficult. I simply do not expect a meaningful number of people to be able to do this. You have to be willing to turn your life upside down.
This can be terrible for a person’s mental health. We shouldn’t be vary careful openly recommending this as a reason to stay at a Lab.
As mentioned above, I expect a greater number of people to turn into converts than would-be Whistle-Blowers
I think if you’re the kind of person who goes into a Lab for Career Capital that makes you less likely to be a Whistle-Blower TBH.
4. Be Prepared to Whistle-Blow:
Sorry, you can make good money elsewhere.
(1) I agree if your timelines are super short, like <2yrs, it’s probably not worth it. I have a bunch of probability mass on longer timelines, though some on really short ones
Re (2), my sense is some employees already have had some of this effect (and many don’t. But some do). I think board members are terrible candidates for changing org culture; they have unrelated full-time jobs, they don’t work from the office, they have different backgrounds, most people don’t have cause to interact with them much. People who are full-time, work together with people all day every day, know the context, etc., seem more likely to be effective at this (and indeed, I think they have been, to some extent in some cases)
Re (3), seems like a bunch of OAI people have blown the whistle on bad behavior already, so the track record is pretty great, and I think them doing that has been super valuable. And 1 whistleblower seems much better than several converts is bad. I agree it can be terrible for mental health for some people, and people should take care of themselves.
Re (4), um, this is the EA Forum, we care about how good the money is. Besides crypto, I don’t think there are many for many of the relevant people to make similar amounts of money on similar timeframes. Actually I think working at a lab early was an effective way to make money. A bunch of safety-concerned people for example have equity worth several millions to tens of millions, more than I think they could have easily earned elsewhere, and some are now billionaires on paper. And if AI has the transformative impact on the economy we expect, that could be worth way more (and it being worth more is correlated with it being needed more, so extra valuable); we are talking about the most valuable/powerful industry the world has ever known here, hard to beat that for making money. I don’t think that makes it okay to lead large AI labs, but for joining early, especially doing some capabilities work that doesn’t push the most risky capabilities along much, I don’t think it’s obvious.
I agree that there are various risks related to staying too long, rationalizing, being greedy, etc., and in most cases I wouldn’t advice a safety-concerned person to do capabilities. But I think you’re being substantially too intense about the risk of speeding up AI relative to the benefits of seeing what’s happening on the inside, which seem like they’ve already been very substantial