Not disagreeing with your thesis necessarily, but I disagree that a startup can’t have a safety-focused culture. Most mainstream (i.e., not crypto) financial trading firms started out as a very risk-conscious startup. This can be hard to evaluate from the outside, though, and definitely depends on committed executives.
Regarding the actual companies we have, though, my sense is that OpenAI is not careful and I’m not feeling great about Anthropic either.
I agree that it’s possible for startups to have a safety-focused culture! The question that’s interesting to me is whether it’s likely / what the prior should be.
Finance is a good example of a situation where you often can get a safety culture despite no prior experience with your products (or your predecessor’s products, etc) killing people. I’m not sure why that happened? Some combination of 2008 making people aware of systemic risks + regulations successfully creating a stronger safety culture?
Oh sure, I’ll readily agree that most startups don’t have a safety culture. The part I was disagreeing with was this:
I think it’s hard to have a safety-focused culture just by “wanting it” hard enough in the abstract
Regarding finance, I don’t think this is about 2008, because there are plenty of trading firms that were careful from the outset that were also founded well before the financial crisis. I do think there is a strong selection effect happening, where we don’t really observe the firms that weren’t careful (because they blew up eventually, even if they were lucky in the beginning).
How do careful startups happen? Basically I think it just takes safety-minded founders. That’s why the quote above didn’t seem quite right to me. Why are most startups not safety-minded? Because most founders are not safety-minded, which in turn is probably due in part to a combination of incentives and selection effects.
How do careful startups happen? Basically I think it just takes safety-minded founders.
Thanks! I think this is the crux here. I suspect what you say isn’t enough but it sounds like you have a lot more experience than I do, so happy to (tentatively) defer.
Not disagreeing with your thesis necessarily, but I disagree that a startup can’t have a safety-focused culture. Most mainstream (i.e., not crypto) financial trading firms started out as a very risk-conscious startup. This can be hard to evaluate from the outside, though, and definitely depends on committed executives.
Regarding the actual companies we have, though, my sense is that OpenAI is not careful and I’m not feeling great about Anthropic either.
I agree that it’s possible for startups to have a safety-focused culture! The question that’s interesting to me is whether it’s likely / what the prior should be.
Finance is a good example of a situation where you often can get a safety culture despite no prior experience with your products (or your predecessor’s products, etc) killing people. I’m not sure why that happened? Some combination of 2008 making people aware of systemic risks + regulations successfully creating a stronger safety culture?
Oh sure, I’ll readily agree that most startups don’t have a safety culture. The part I was disagreeing with was this:
Regarding finance, I don’t think this is about 2008, because there are plenty of trading firms that were careful from the outset that were also founded well before the financial crisis. I do think there is a strong selection effect happening, where we don’t really observe the firms that weren’t careful (because they blew up eventually, even if they were lucky in the beginning).
How do careful startups happen? Basically I think it just takes safety-minded founders. That’s why the quote above didn’t seem quite right to me. Why are most startups not safety-minded? Because most founders are not safety-minded, which in turn is probably due in part to a combination of incentives and selection effects.
Thanks! I think this is the crux here. I suspect what you say isn’t enough but it sounds like you have a lot more experience than I do, so happy to (tentatively) defer.