Nice post! I basically agreed overall. Some rambly thoughts:
One reason to work on some “puntable” topics (which I expect you’re aware of, tbc) is if doing some of the work helps you understand how what good work on that question looks like, how to automate it well, etc — analogous to how it’s easier to hire for work that you know how to do well.
I feel a bit sceptical about work on “foundational and ontological” questions being worth it, though this might be kneejerk scepticism to the word “ontology” (sorry). It does seem like the work that’s slowest to do/hardest to evaluate.
(In general, it does seem hard to know how to prioritise slow, not urgent (?), hard-for-AIs work, other than by trying to make AIs better at such work)
Re: AI welfare, I agree that your quoted questions are very puntable, but I think other AI welfare work looks way better to do now. My guess is that the best consequentialist reason for working on AI welfare now is to avoid locking in bad norms, or (more ambitiously) trying to set low-cost good norms; I think this is a pretty reasonable position, and overall feel good about people researching e.g. cheap AI welfare interventions now. I think we probably disagree, so I figured I’d mention it.
Similarly, I think sometimes there’s a vibe of, “this will soon be salient to people, for maybe bad/misleading reasons, let’s try to make the discourse saner by being early to the topic”, which I find basically compelling as an argument.
I left this post feeling like, “man, so much of the plan/the hope is to punt the research to future AIs, or maybe future AI-assisted humans; it’s not even clear to me that Forethought should do strategy themselves at all, vs just trying really really hard to make sure handing off strategy to future AIs goes well”. (The tradeoff might in practice be small, e.g. maybe the ~best way for you to accomplish the latter is to do lots of the former to generate training data, but I think it’s >0.) How do you think about this? (Or, how much is Forethought doing strategy for strategy’s sake, vs instrumentally, to try to make automating strategy research go well?)
Nice post! I basically agreed overall. Some rambly thoughts:
One reason to work on some “puntable” topics (which I expect you’re aware of, tbc) is if doing some of the work helps you understand how what good work on that question looks like, how to automate it well, etc — analogous to how it’s easier to hire for work that you know how to do well.
I feel a bit sceptical about work on “foundational and ontological” questions being worth it, though this might be kneejerk scepticism to the word “ontology” (sorry). It does seem like the work that’s slowest to do/hardest to evaluate.
(In general, it does seem hard to know how to prioritise slow, not urgent (?), hard-for-AIs work, other than by trying to make AIs better at such work)
Re: AI welfare, I agree that your quoted questions are very puntable, but I think other AI welfare work looks way better to do now. My guess is that the best consequentialist reason for working on AI welfare now is to avoid locking in bad norms, or (more ambitiously) trying to set low-cost good norms; I think this is a pretty reasonable position, and overall feel good about people researching e.g. cheap AI welfare interventions now. I think we probably disagree, so I figured I’d mention it.
Similarly, I think sometimes there’s a vibe of, “this will soon be salient to people, for maybe bad/misleading reasons, let’s try to make the discourse saner by being early to the topic”, which I find basically compelling as an argument.
I left this post feeling like, “man, so much of the plan/the hope is to punt the research to future AIs, or maybe future AI-assisted humans; it’s not even clear to me that Forethought should do strategy themselves at all, vs just trying really really hard to make sure handing off strategy to future AIs goes well”. (The tradeoff might in practice be small, e.g. maybe the ~best way for you to accomplish the latter is to do lots of the former to generate training data, but I think it’s >0.) How do you think about this? (Or, how much is Forethought doing strategy for strategy’s sake, vs instrumentally, to try to make automating strategy research go well?)
Knowing these authors, my guess on ontology is that they might say that it could be instrumental in things like
motivating progress in safer paradigms of AI development
understanding ‘hybrid’ human-AI-org opportunities and threats
figuring out what types of ‘post early’ conditions look favourable for dealing with the next challenges
These all look like activities with bearing on how to tackle ‘early’ challenges.