But these should be matched by looking for cases where something good happened because people tried to accumulate power/influence within a system.
I think this is a significant percent of all good things that have ever happened.
I think you are right about this, you’ve changed my mind (toward greater uncertainty).
My gut feeling is that [...] the biggest difference between good outcomes and bad outcomes is how much work the big AI labs put into alignment during the middle of the intelligence explosion when progress moves fastest.
This seems to depend on a conjunction of several strong assumptions: (1) AI alignment is basically easy; (2) there will be a slow takeoff; (3) the people running AI companies are open to persuasion, and “make AI safety seem cool” is the best kind of persuasion.
But then again I don’t think pause protests are going to work, I’m just trying to pick whichever bad plan seems the least bad.
2. I agree I’m assuming there will be a slow takeoff (operationalized as let’s say a ~one year period where GPT-integer-increment-level-changes happen on a scale of months, before any such period where they happen on a scale of days).
3. AI companies being open to persuasion seems kind of trivial to me. They already have alignment teams. They already (I assume) have budget meetings where they discuss how many resources these teams should get. I’m just imagining inputs into this regular process. I agree that issues around politics could be a lesser vs. greater input.
1. I wouldn’t frame this as alignment is easy/hard, so much as “alignment is more refractory to 10,000 copies of GPT-6 working for a subjective century” vs. “alignment is more refractory to one genius, not working at a lab, coming up with a new paradigm using only current or slightly-above-current AIs as model organisms, in a sense where we get one roll at this per calendar year”.
Not to tell you what to do, but I’d love to see a longer ACX post making these arguments Scott :). Seems like it could be rich seed for discussion; almost all the writing I’ve seen in rationalist spaces around these issues has been anti-build-goodwill/influence-from-within, and I think your counterpoints here are strong
I think you are right about this, you’ve changed my mind (toward greater uncertainty).
This seems to depend on a conjunction of several strong assumptions: (1) AI alignment is basically easy; (2) there will be a slow takeoff; (3) the people running AI companies are open to persuasion, and “make AI safety seem cool” is the best kind of persuasion.
But then again I don’t think pause protests are going to work, I’m just trying to pick whichever bad plan seems the least bad.
2. I agree I’m assuming there will be a slow takeoff (operationalized as let’s say a ~one year period where GPT-integer-increment-level-changes happen on a scale of months, before any such period where they happen on a scale of days).
3. AI companies being open to persuasion seems kind of trivial to me. They already have alignment teams. They already (I assume) have budget meetings where they discuss how many resources these teams should get. I’m just imagining inputs into this regular process. I agree that issues around politics could be a lesser vs. greater input.
1. I wouldn’t frame this as alignment is easy/hard, so much as “alignment is more refractory to 10,000 copies of GPT-6 working for a subjective century” vs. “alignment is more refractory to one genius, not working at a lab, coming up with a new paradigm using only current or slightly-above-current AIs as model organisms, in a sense where we get one roll at this per calendar year”.
Not to tell you what to do, but I’d love to see a longer ACX post making these arguments Scott :). Seems like it could be rich seed for discussion; almost all the writing I’ve seen in rationalist spaces around these issues has been anti-build-goodwill/influence-from-within, and I think your counterpoints here are strong