Agreed! Indeed, I think AGI timelines research is even less useful than this post implies; I think just about all of the work to date didn’t help and shouldn’t have been a priority.
I disagree with Reason 6 as a thing that should influence our behavior; if we let our behavior be influenced by reputational risks as small as this, IMO we’ll generally be way too trigger-happy about hiding our honest views in order to optimize reputation, which is not a good way to make intellectual progress or build trust.
Regardless of timelines, there are many things we need to be making progress on as quickly as possible. These include improving discourse and practice around publication norms in AI; improving the level of rigor for risk assessment and management for developed and deployed AI systems;
Agreed.
improving dialogue and coordination among actors building powerful AI systems, to avoid reinvention of the wheel re: safety assessments and mitigations;
I’m not sure exactly what you have in mind here, but at a glance, this doesn’t sound like a high priority to me. I don’t think we have wheels to reinvent; the priority is to figure out how to do alignment at all, not to improve communication channels so we can share our current absence-of-ideas.
I would agree, however, that it’s very high-priority to get people on the same page about basic things like ‘we should be trying to figure out alignment at all’, insofar as people aren’t on that page.
getting competent, well-intentioned people into companies and governments to work on these things;
Getting some people into gov seems fine to me, but probably not on the critical path. Getting good people into companies seems more on the critical path to me, but this framing seems wrong to me, because of my background model that (e.g.) we’re hopelessly far from knowing how to do alignment today.
I think the priority should be to cause people to think about alignment who might give humanity a better idea of a realistic way we could actually align AGI systems, not to find nice smart people and reposition them to places that vaguely seem more important. I’d guess most ‘placing well-intentioned people at important-seeming AI companies’ efforts to date have been net-negative.
getting serious AI regulation started in earnest;
Seems like plausibly a bad idea to me. I don’t see a way this can realistically help outside of generically slowing the field down, and I’m not sure even this would be net-positive, given the likely effect on ML discourse?
I’d at least want to hear more detail, rather than just “let’s regulate AI, because something must be done, and this is something”.
and doing basic safety and policy research.
I would specifically say ‘figure out how to do technical alignment of AGI systems’. (Still speaking from my own models.)
Clarifying the kind of timelines work I think is low-importance:
I think there’s value in distinguishing worlds like “1% chance of AGI by 2100” versus “10+% chance”, and distinguishing “1% chance of AGI by 2050” versus “10+% chance”.
So timelines work enabling those updates was good.[1]
But I care a lot less about, e.g., “2/3 by 2050” versus “1/3 by 2050″.
And I care even less about distinguishing, e.g., “30% chance of AGI by 2030, 80% chance of AGI by 2050” from “15% chance of AGI by 2030, 50% chance of AGI by 2050″.
Though I think it takes very little evidence or cognition to rationally reach 10+% probability of AGI by 2100.
One heuristic way of seeing this is to note how confident you’d need to be in ‘stuff like the deep learning revolution (as well as everything that follows it) won’t get us to AGI in the next 85 years’, in order to make a 90+% prediction to that effect.
Notably, you don’t need a robust or universally persuasive 10+% in order to justify placing the alignment problem at or near the top of your priority list.
You just needs that to be your subjective probability at all, coupled with a recognition that AGI is an absurdly big deal and aligning the first AGI systems looks non-easy.
In retrospect I should have made a clearer distinction between “things that the author thinks are good and which are mostly timeline-insensitive according to his model of how things work” and “things that all reasonable observers would agree are good ideas regardless of their timelines.” The stuff you mentioned mostly relates to currently-existing-AI-systems and management of their risks, and while not consensus-y, are mostly agreed on by people in the trenches of language model risks—for example, there is a lot of knowledge to share and which is being shared already about language model deployment best practices. And one needn’t invoke/think one way or the other about AGI to justify government intervention in managing risks of existing and near-term systems given the potential stakes of failure (e.g. collapse of the epistemic commons via scaled misuse of increasingly powerful language/image generation; reckless deployment of such systems in critical applications). Of course one might worry that intervening on those things will detract resources from other things, but my view, which I can’t really justify concisely here but happy to discuss in another venue, is that overwhelmingly the synergies outweigh the tradeoffs (e.g. there are big culture/norm benefits at the organizational and industry level—which will directly increase the likelihood of good AGI outcomes if the same orgs/people are involved—of being careful about current technologies compared to not doing so, even if the techniques themselves are very different).
Yeah, I’m specifically interested in AGI / ASI / “AI that could cause us to completely lose control of the future in the next decade or less”, and I’m more broadly interested in existential risk / things that could secure or burn the cosmic endowment. If I could request one thing, it would be clarity about when you’re discussing “acutely x-risky AI” (or something to that effect) versus other AI things; I care much more about that than about you flagging personal views vs. consensus views.
I agree on regulations. Our general prior should look like public choice theory. Regulations have a tendency to drift toward unintended kinds, usually with a more rent-seeking focus than planned. They also tend to have more unintended consequences than people predict.
There probably are some that pass a cost-benefit, but as a general prior, we should be very reluctant, and have very high standards. Getting serious AI regulations started has a very high chance of misfiring or overshooting or backfiring.
Agreed! Indeed, I think AGI timelines research is even less useful than this post implies; I think just about all of the work to date didn’t help and shouldn’t have been a priority.
I disagree with Reason 6 as a thing that should influence our behavior; if we let our behavior be influenced by reputational risks as small as this, IMO we’ll generally be way too trigger-happy about hiding our honest views in order to optimize reputation, which is not a good way to make intellectual progress or build trust.
Agreed.
I’m not sure exactly what you have in mind here, but at a glance, this doesn’t sound like a high priority to me. I don’t think we have wheels to reinvent; the priority is to figure out how to do alignment at all, not to improve communication channels so we can share our current absence-of-ideas.
I would agree, however, that it’s very high-priority to get people on the same page about basic things like ‘we should be trying to figure out alignment at all’, insofar as people aren’t on that page.
Getting some people into gov seems fine to me, but probably not on the critical path. Getting good people into companies seems more on the critical path to me, but this framing seems wrong to me, because of my background model that (e.g.) we’re hopelessly far from knowing how to do alignment today.
I think the priority should be to cause people to think about alignment who might give humanity a better idea of a realistic way we could actually align AGI systems, not to find nice smart people and reposition them to places that vaguely seem more important. I’d guess most ‘placing well-intentioned people at important-seeming AI companies’ efforts to date have been net-negative.
Seems like plausibly a bad idea to me. I don’t see a way this can realistically help outside of generically slowing the field down, and I’m not sure even this would be net-positive, given the likely effect on ML discourse?
I’d at least want to hear more detail, rather than just “let’s regulate AI, because something must be done, and this is something”.
I would specifically say ‘figure out how to do technical alignment of AGI systems’. (Still speaking from my own models.)
Clarifying the kind of timelines work I think is low-importance:
I think there’s value in distinguishing worlds like “1% chance of AGI by 2100” versus “10+% chance”, and distinguishing “1% chance of AGI by 2050” versus “10+% chance”.
So timelines work enabling those updates was good.[1]
But I care a lot less about, e.g., “2/3 by 2050” versus “1/3 by 2050″.
And I care even less about distinguishing, e.g., “30% chance of AGI by 2030, 80% chance of AGI by 2050” from “15% chance of AGI by 2030, 50% chance of AGI by 2050″.
Though I think it takes very little evidence or cognition to rationally reach 10+% probability of AGI by 2100.
One heuristic way of seeing this is to note how confident you’d need to be in ‘stuff like the deep learning revolution (as well as everything that follows it) won’t get us to AGI in the next 85 years’, in order to make a 90+% prediction to that effect.
Notably, you don’t need a robust or universally persuasive 10+% in order to justify placing the alignment problem at or near the top of your priority list.
You just needs that to be your subjective probability at all, coupled with a recognition that AGI is an absurdly big deal and aligning the first AGI systems looks non-easy.
What about distinguishing 50% by 2050 vs. 50% by 2027?
In retrospect I should have made a clearer distinction between “things that the author thinks are good and which are mostly timeline-insensitive according to his model of how things work” and “things that all reasonable observers would agree are good ideas regardless of their timelines.” The stuff you mentioned mostly relates to currently-existing-AI-systems and management of their risks, and while not consensus-y, are mostly agreed on by people in the trenches of language model risks—for example, there is a lot of knowledge to share and which is being shared already about language model deployment best practices. And one needn’t invoke/think one way or the other about AGI to justify government intervention in managing risks of existing and near-term systems given the potential stakes of failure (e.g. collapse of the epistemic commons via scaled misuse of increasingly powerful language/image generation; reckless deployment of such systems in critical applications). Of course one might worry that intervening on those things will detract resources from other things, but my view, which I can’t really justify concisely here but happy to discuss in another venue, is that overwhelmingly the synergies outweigh the tradeoffs (e.g. there are big culture/norm benefits at the organizational and industry level—which will directly increase the likelihood of good AGI outcomes if the same orgs/people are involved—of being careful about current technologies compared to not doing so, even if the techniques themselves are very different).
Yeah, I’m specifically interested in AGI / ASI / “AI that could cause us to completely lose control of the future in the next decade or less”, and I’m more broadly interested in existential risk / things that could secure or burn the cosmic endowment. If I could request one thing, it would be clarity about when you’re discussing “acutely x-risky AI” (or something to that effect) versus other AI things; I care much more about that than about you flagging personal views vs. consensus views.
I agree on regulations. Our general prior should look like public choice theory. Regulations have a tendency to drift toward unintended kinds, usually with a more rent-seeking focus than planned. They also tend to have more unintended consequences than people predict.
There probably are some that pass a cost-benefit, but as a general prior, we should be very reluctant, and have very high standards. Getting serious AI regulations started has a very high chance of misfiring or overshooting or backfiring.
Could you please elaborate on this? The reasoning here seems non-obvious.