IMO it makes much more sense to target AI developers who are training foundation models with huge amounts of compute. My understanding is that Cognition isn’t training foundation models, and is more of a “wrapper” in the sense that they are building on top of others’ foundation models to apply scaffolding, and/or fine-tuning with <~1% of the foundation model training compute. Correct me if I’m wrong.
Gesturing at some of the reasons I think that wrappers should be deprioritized:
Over time, I’d guess that wrapper companies working on AI R&D-relevant tasks like Cognition either get acquired or fade into irrelevancy since there will be pressure to make AI R&D agents internally (maybe this campaign is still useful if it gets acquired though?)
Good question! I basically agree with you about the relative importance of foundation model developers here (although I haven’t thought too much about the third point you mentioned. Thanks for bringing it up.)
I should say we are doing some other work to raise awareness about foundation model risks—especially at OpenAI, given recent events—but not at the level of this campaign.
The main constraint was starting (relatively) small. We’d really like to win these campaigns, and we don’t plan to let up until we have. The foundation model developers are generally some of the biggest companies in the world (hence the huge compute, as you mention), and the resources needed to win a campaign likely scale in proportion to the size of the target. We decided it’d be good to keep building our supporter base and reputation before taking the bigger players on. Cognition in particular seems to be in the center of the triple venn diagram between “making high-risk systems,” “way behind the curve on safety issues,” and “small enough that they can’t afford to ignore this.”
Btw, my background is in animal advocacy, and this is somewhat similar to how groups scaled there. i.e. they started by getting local restaurants to stop serving fois gras, and scaled up to getting McDonalds to phase out eggs from battery cages nationwide. Obviously we have less time with this issue—so I would like to scale quickly.
How did you decide to target Cognition?
IMO it makes much more sense to target AI developers who are training foundation models with huge amounts of compute. My understanding is that Cognition isn’t training foundation models, and is more of a “wrapper” in the sense that they are building on top of others’ foundation models to apply scaffolding, and/or fine-tuning with <~1% of the foundation model training compute. Correct me if I’m wrong.
Gesturing at some of the reasons I think that wrappers should be deprioritized:
Much of the risks from scheming AIs routes through internal AI R&D via internal foundation models
Over time, I’d guess that wrapper companies working on AI R&D-relevant tasks like Cognition either get acquired or fade into irrelevancy since there will be pressure to make AI R&D agents internally (maybe this campaign is still useful if it gets acquired though?)
Accelerating LM agent scaffolding has unclear sign for safety
Maybe the answer is that Cognition was way better than foundation model developers on other dimensions, in which case, fair enough.
Good question! I basically agree with you about the relative importance of foundation model developers here (although I haven’t thought too much about the third point you mentioned. Thanks for bringing it up.)
I should say we are doing some other work to raise awareness about foundation model risks—especially at OpenAI, given recent events—but not at the level of this campaign.
The main constraint was starting (relatively) small. We’d really like to win these campaigns, and we don’t plan to let up until we have. The foundation model developers are generally some of the biggest companies in the world (hence the huge compute, as you mention), and the resources needed to win a campaign likely scale in proportion to the size of the target. We decided it’d be good to keep building our supporter base and reputation before taking the bigger players on. Cognition in particular seems to be in the center of the triple venn diagram between “making high-risk systems,” “way behind the curve on safety issues,” and “small enough that they can’t afford to ignore this.”
Btw, my background is in animal advocacy, and this is somewhat similar to how groups scaled there. i.e. they started by getting local restaurants to stop serving fois gras, and scaled up to getting McDonalds to phase out eggs from battery cages nationwide. Obviously we have less time with this issue—so I would like to scale quickly.