Responding here for greater visibility—I’m responding to the idea in your short-form that the lesson from this is to hire for greater value alignment.
Epoch’s founder has openly stated that their company culture is not particularly fussed about most AI risk topics [edit: they only stated this today, making the rest of my comment here less accurate; see thread]. Key quotes from that post:
“on net I support faster development of AI, so we can benefit earlier from it.”
“I am not very concerned about violent AI takeover. I am concerned about concentration of power and gradual disempowerment.”
So I’m not sure this is that much of a surprise? It’s at least not totally obvious that Mechanize’s existence is contrary to those values.
As a result, I’m not sure the lesson is “EA orgs should hire for value alignment.” I think most EAs just didn’t understand what Epoch’s values were. If that’s right, the lesson is that the EA community shouldn’t assume that an organization that happens to work adjacent to AI safety actually cares about it. In part, that’s a lesson for funders to not just look at the content of the proposal in front of you, but also what the org as a whole is doing.
Epoch’s founder has openly stated that their company culture is not particularly fussed about most AI risk topics
To be clear, my personal views are different from my employees or our company. We have a plurality of views within the organisation (which I think it’s important for our ability to figure out what will actually happen!)
I co-started Epoch to get more evidence on AI and AI risk. As I learned more and the situation unfolded I have become more skeptical of AI Risk. I tried to be transparent about this, though I’ve changed my mind often and is time-consuming to communicate every update.
I also strive to make Epoch work relevant and useful to people regardless of their views. Eg both AI2027 and situational awareness rely heavily on Epoch work, even though I disagree with their perspectives. You don’t need to agree with what I believe to find our work useful!
That post was written today though—I think the lesson to be learned depends on whether those were always the values vs. a change from what was espoused at the time of funding.
Oh whoops, I was looking for a tweet they wrote a while back and confused it with the one I linked. I was thinking of this one, where he states that “slowing down AI development” is a mistake. But I’m realizing that this was also only in January, when the OpenAI funding thing came out, so doesn’t necessarily tell us much about historical values.
I suppose you could interpret some tweets like this or this in a variety of ways but it now reads as consistent with “don’t let AI fear get in the way of progress” type views. I don’t say this to suggest that EA funders should have been able to tell ages ago, btw, just trying to see if there’s any way to get additional past data.
Another fairly relevant thing to me is that their work is on benchmarking and forecasting potential outcomes, something that doesn’t seem directly tied to safety and which is also clearly useful to accelerationists. As a relative outsider to this space, it surprises me much less that Epoch would be mostly made up of folks interested in AI acceleration or at least neutral towards it, than if I found out that some group researching something more explicitly safety-focused had those values. Maybe the takeaway there is that if someone is doing something that is useful both to acceleration-y people and safety people, check the details? But perhaps that’s being overly suspicious.
And I guess also more generally, again from a relatively outside perspective, it’s always seemed like AI folks in EA have been concerned with both gaining the benefits of AI and avoiding X risk. That kind of tension was at issue when this article blew up here a few years back and seems to be a key part of why the OpenAI thing backfired so badly. It just seems really hard to combine building the tool and making it safe into the same movement; if you do, I don’t think stuff like Mechanize coming out of it should be that surprising, because your party will have guests who only care about one thing or the other.
I’ve written a short-form here as well.
Responding here for greater visibility—I’m responding to the idea in your short-form that the lesson from this is to hire for greater value alignment.
Epoch’s founder has openly stated that their company culture is not particularly fussed about most AI risk topics [edit: they only stated this today, making the rest of my comment here less accurate; see thread]. Key quotes from that post:
“on net I support faster development of AI, so we can benefit earlier from it.”
“I am not very concerned about violent AI takeover. I am concerned about concentration of power and gradual disempowerment.”
So I’m not sure this is that much of a surprise? It’s at least not totally obvious that Mechanize’s existence is contrary to those values.
As a result, I’m not sure the lesson is “EA orgs should hire for value alignment.” I think most EAs just didn’t understand what Epoch’s values were. If that’s right, the lesson is that the EA community shouldn’t assume that an organization that happens to work adjacent to AI safety actually cares about it. In part, that’s a lesson for funders to not just look at the content of the proposal in front of you, but also what the org as a whole is doing.
To be clear, my personal views are different from my employees or our company. We have a plurality of views within the organisation (which I think it’s important for our ability to figure out what will actually happen!)
I co-started Epoch to get more evidence on AI and AI risk. As I learned more and the situation unfolded I have become more skeptical of AI Risk. I tried to be transparent about this, though I’ve changed my mind often and is time-consuming to communicate every update.
I also strive to make Epoch work relevant and useful to people regardless of their views. Eg both AI2027 and situational awareness rely heavily on Epoch work, even though I disagree with their perspectives. You don’t need to agree with what I believe to find our work useful!
That post was written today though—I think the lesson to be learned depends on whether those were always the values vs. a change from what was espoused at the time of funding.
Oh whoops, I was looking for a tweet they wrote a while back and confused it with the one I linked. I was thinking of this one, where he states that “slowing down AI development” is a mistake. But I’m realizing that this was also only in January, when the OpenAI funding thing came out, so doesn’t necessarily tell us much about historical values.
I suppose you could interpret some tweets like this or this in a variety of ways but it now reads as consistent with “don’t let AI fear get in the way of progress” type views. I don’t say this to suggest that EA funders should have been able to tell ages ago, btw, just trying to see if there’s any way to get additional past data.
Another fairly relevant thing to me is that their work is on benchmarking and forecasting potential outcomes, something that doesn’t seem directly tied to safety and which is also clearly useful to accelerationists. As a relative outsider to this space, it surprises me much less that Epoch would be mostly made up of folks interested in AI acceleration or at least neutral towards it, than if I found out that some group researching something more explicitly safety-focused had those values. Maybe the takeaway there is that if someone is doing something that is useful both to acceleration-y people and safety people, check the details? But perhaps that’s being overly suspicious.
And I guess also more generally, again from a relatively outside perspective, it’s always seemed like AI folks in EA have been concerned with both gaining the benefits of AI and avoiding X risk. That kind of tension was at issue when this article blew up here a few years back and seems to be a key part of why the OpenAI thing backfired so badly. It just seems really hard to combine building the tool and making it safe into the same movement; if you do, I don’t think stuff like Mechanize coming out of it should be that surprising, because your party will have guests who only care about one thing or the other.
Agreed.
Interesting that you chose not to name the org in question—I guess you wanted to focus on the meta-level principle rather than this specific case
Maybe I should have. I honestly don’t know. I didn’t think deeply about it.