From my understanding of boards and governance structures, I think that few are actually very effective, and it’s often very difficult to tell this from outside the organization.
So, I think that the prior should be to expect these governance structures to be quite mediocre, especially in extreme cases, and wait for a significant amount of evidence otherwise.
I think some people think, “Sure, but it’s quite hard to provide a lot of public evidence, so instead we should give these groups the benefit of the doubt.” I don’t think this makes sense as an epistemic process.
If the prior is bad, then you should expect it to be bad. If it’s really difficult to convince a good epistemic process otherwise, don’t accept a worse epistemic process in order to make it seem “more fair for the evaluee”.
From my understanding of boards and governance structures, I think that few are actually very effective, and it’s often very difficult to tell this from outside the organization.
It seems valuable to differentiate between “ineffective by design” and “ineffective in practice”. Which do you think is more the cause for the trend you’re observing?
OP is concerned that Anthropic’s governance might fall into the “ineffective by design” category. Like, it’s predictable in advance that something could maybe go wrong here.
If yours is more of an “ineffective in practice” argument—that seems especially concerning, if the “ineffective in practice” point applies even when the governance appeared to be effective by design, ex ante.
In any case, I’d really like to see dedicated efforts to argue for ideal AI governance structures and documents. It feels like EA has overweighted the policy side of AI governance and underweighted the organizational founding documents side. Right now we’re in the peanut gallery, criticizing how things are going at OpenAI and now Anthropic, without offering much in the way of specific alternatives.
Events at OpenAI have shown that this issue deserves a lot more attention, in my opinion. Some ideas:
A big cash prize for best AI lab governance structure proposals. (In practice you’d probably want to pick and choose the best ideas across multiple proposals.)
Subsidize red-teaming novel proposals and testing out novel proposals in lower-stakes situations, for non-AI organiations. (All else equal, it seems better for AGI to be developed using an institutional template that’s battle-tested.) We could dogfood proposals by using them for non-AI EA startups or EA organizations focused on e.g. community-building.
Governance lit reviews to gather and summarize info, both empirical info and also theoretical models from e.g. economics. Cross-national comparisons might be especially fruitful if we don’t think the right structures are battle-tested in a US legal context.
At this point, I’m embarrassed that if someone asked me how to fix OpenAI’s governance docs, I wouldn’t really have a suggestion. On the other hand, if we had some really solid suggestions, it feels doable to either translate them into policy requirements, or convince groups like Anthropic’s trustees to adopt them.
If yours is more of an “ineffective in practice” argument—that seems especially concerning,
Good point here. You’re right that I’m highlighting “ineffective in practice”.
In any case, I’d really like to see dedicated efforts to argue for ideal AI governance structures and documents.
Yep, I’d also like to see more work here! A while back I helped work on this project, which investigated this area a bit, but I think it’s obvious there’s more work to do.
Thanks for investigating!
From my understanding of boards and governance structures, I think that few are actually very effective, and it’s often very difficult to tell this from outside the organization.
So, I think that the prior should be to expect these governance structures to be quite mediocre, especially in extreme cases, and wait for a significant amount of evidence otherwise.
I think some people think, “Sure, but it’s quite hard to provide a lot of public evidence, so instead we should give these groups the benefit of the doubt.” I don’t think this makes sense as an epistemic process.
If the prior is bad, then you should expect it to be bad. If it’s really difficult to convince a good epistemic process otherwise, don’t accept a worse epistemic process in order to make it seem “more fair for the evaluee”.
It seems valuable to differentiate between “ineffective by design” and “ineffective in practice”. Which do you think is more the cause for the trend you’re observing?
OP is concerned that Anthropic’s governance might fall into the “ineffective by design” category. Like, it’s predictable in advance that something could maybe go wrong here.
If yours is more of an “ineffective in practice” argument—that seems especially concerning, if the “ineffective in practice” point applies even when the governance appeared to be effective by design, ex ante.
In any case, I’d really like to see dedicated efforts to argue for ideal AI governance structures and documents. It feels like EA has overweighted the policy side of AI governance and underweighted the organizational founding documents side. Right now we’re in the peanut gallery, criticizing how things are going at OpenAI and now Anthropic, without offering much in the way of specific alternatives.
Events at OpenAI have shown that this issue deserves a lot more attention, in my opinion. Some ideas:
A big cash prize for best AI lab governance structure proposals. (In practice you’d probably want to pick and choose the best ideas across multiple proposals.)
Subsidize red-teaming novel proposals and testing out novel proposals in lower-stakes situations, for non-AI organiations. (All else equal, it seems better for AGI to be developed using an institutional template that’s battle-tested.) We could dogfood proposals by using them for non-AI EA startups or EA organizations focused on e.g. community-building.
Governance lit reviews to gather and summarize info, both empirical info and also theoretical models from e.g. economics. Cross-national comparisons might be especially fruitful if we don’t think the right structures are battle-tested in a US legal context.
At this point, I’m embarrassed that if someone asked me how to fix OpenAI’s governance docs, I wouldn’t really have a suggestion. On the other hand, if we had some really solid suggestions, it feels doable to either translate them into policy requirements, or convince groups like Anthropic’s trustees to adopt them.
Good point here. You’re right that I’m highlighting “ineffective in practice”.
Yep, I’d also like to see more work here! A while back I helped work on this project, which investigated this area a bit, but I think it’s obvious there’s more work to do.