Error
Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?
There are very expensive interventions that are financially constrained and could use up ~all EA funds, and the cost-benefit calculation takes probability of powerful AGI in a given time period as an input, so that e.g. twice the probability of AGI in the next 10 years justifies spending twice as much for a given result by doubling the chance the result gets to be applied. That can make the difference between doing the intervention or not, or drastic differences in intervention size.
Could you give an example or two? I tend to think of “~all of EA funds”-level interventions as more like timeline-shifting interventions than things that would be premised on a given timeline (though there is a fine line between the two), and am skeptical of most that I can think of, but I agree that if such things exist it would count against what I’m saying.
The funding scale of AI labs/research, AI chip production, and US political spending could absorb billions per year, tens of billions or more for the first two. Philanthropic funding of a preferred AI lab at the cutting edge as model sizes inflate could take all EA funds and more on its own.
There are also many expensive biosecurity interventions that are being compared against an AI intervention benchmark. Things like developing PPE, better sequencing/detection, countermeasures through philanthropic funding rather than hoping to leverage cheaper government funding.
Thanks for elaborating—I haven’t thought much about the bio comparison and political spending things but on funding a preferred lab/compute stuff, I agree that could be more sensitive to timelines than the AI policy things I mentioned.
FWIW I don’t think it’s as sensitive to timelines as it may first appear (doing something like that could still make sense even with longer timelines given the potential value in shaping norms, policies, public attitudes on AI, etc., particularly if one expects sub-AGI progress to help replenish EA coffers, and if such an idea were misguided I think it’d probably be for non-timeline-related reasons like accelerating competition or speeding things up too much even for a favored lab to handle).
But if I were rewriting I’d probably mention it as a prominent counterexample justifying some further work along with some of the alignment agenda stuff mentioned below.
Oh, one more thing: AI timelines put a discount on other interventions. Developing a technology that will take 30 years to have its effect is less than half as important if your median AGI timeline is 20 years.
I assume Carl is thinking of something along the lines of “try and buy most new high-end chips”. See eg Sam interviewed by Rob.
I agree with a lot of this post. In particular, getting more precision in timelines is probably not going to help much with persuading most people, or in influencing most of the high-level strategic questions that Miles mentions. I also expect that it’s going to be hard to get much better predictions than we have now: much of the low-hanging fruit has been plucked. However, I’d personally find better timelines quite useful for prioritizing my technical research agenda problems to work on. I might be in a minority here, but I suspect not that small a one (say 25-50% of AI safety researchers).
There’s two main ways timelines influence what I would want to work on. First, it directly changes the “deadline” I am working towards. If I thought the deadline was 5 years, I’d probably work on scaling up the most promising approaches we have now—warts and all. If I thought it was 10 years away, I’d try and make conceptual progress that could be scaled in the future. If it was 20 years away, I’d focus more on longer-term field building interventions: clarifying what the problems are, helping develop good community epistemics, mentoring people, etc. I do think what matters here is something like the log-deadline more than the deadline itself (5 vs 10 is very decision relevant, 20 vs 25 much less so) which we admittedly have a better sense of, although there’s still some considerable disagreement.
The second way timelines are relevant is that my prediction on how AI is developed changes a lot conditioned on timelines. I think we should probably just try to forecast or analyze how-AI-is-developed directly—but timelines are perhaps easier to formalize. If timelines are less than 10 years I’d be confident we develop it within the current deep learning paradigm. More than that and possibilities open up a lot. So overall longer timelines would push me towards more theoretical work (that’s generally applicable across a range of paradigms) and taking bets on underdog areas of ML . There’s not much research into, say, how to align an AI built on top of a probabilistic programming language. I’d say that’s probably not a good use of resources right now—but if we had a confident prediction human-level AI was 50 years away, I might change my mind.
Is there an argument for a <10 years timeline that doesn’t go directly through the claim that it’s going to be achieved in the current paradigm?
You could argue from a “flash of insight” and scientific paradigm shifts generally giving rise to sudden progress. We certainly know contemporary techniques are vastly less sample and compute efficient than the human brain—so there does exist some learning algorithm much better than what we have today. Moreover there probably exists some learning algorithm that would give rise to AGI on contemporary (albeit expensive) hardware. For example, ACX notes there’s a supercomputer than can do $10^17$ FLOPS vs the estimated $10^16 needed for a human brain. These kinds of comparisons are always a bit apples to oranges, but it does seem like compute is probably not the bottleneck (or won’t be in 10 years) for a maximally-efficient algorithm.
The nub of course is whether such an algorithm is plausibly reachable by human flash of insight (and not via e.g. detailed empirical study and refinement of a less efficient but working AGI). It’s hard to rule out. How simple/universal we think the algorithm the human brain implements is one piece of evidence here—the more complex and laden with inductive bias (e.g. innate behavior), the less likely we are to come up with it. But even if the human brain is a Rube Goldberg machine, perhaps there does exist some more straightforward algorithm evolution did not happen upon.
Personally I’d put little weight on this. I have <10% probability on AGI in next 10 years, and think I put no more than 15% on AGI being developed ever by something that looks like a sudden insight than more continuous progress. Notably even if such an insight does happen soon, I’d expect it to take at least 3-5 years for it to gain recognition and be sufficiently scaled up to work. I do think it’s probable enough for us to actively keep an eye out for promising new ideas that could lead to AGI so we can be ahead of the game. I think it’s good for example that a lot of people working on AI safety were working on language models “before it was cool” (I was not one of these people), for example, although we’ve maybe now piled too much into that area.
Today, Miles Brundage published the following, referencing this post:
I do broadly agree with the direction and the sentiment: on the margin, I’d be typically interested in other forecasts than “year of AGI” much more.
For example: in time where we get “AGI” (according to your definition) … how large fraction of GDP are AI companies? … how big is AI as a political topic? … what does the public think?
I’m currently thinking about questions including “how big is AI as a political topic” and “what does the public think”; any recommended reading?
I agree there are diminishing returns; I think Ajeya’s report has done a bunch of what needed to be done. I’m less sure about timelines being decision-irrelevant. Maybe not for Miles, but it seems quite relevant for cause prioritisation, career-planning between causes, and prioritizing policies. I also think better timeline-related arguments could on-net improve, not worsen reputation, because improved substance and polish will actually convince some people.
On the other hand, one argument I might add is that researching timelines could shorten them, by motivating people to make AI that will be realised in their lifetimes, so timelines research can do harm.
On net, I guess I weakly agree—we seem not to be under-investing in timelines research, on the current margin. That said, AI forecasting more broadly—that considers when particular AI capabilities might arise—can be more useful than examining timelines alone, and seems quite useful overall.
+1. My intuition was that forecasts on more granular capabilities would happen automatically if you want to further improve overall timeline estimates. E.g. this is my impression of what a lot of AI timeline related forecasts on Metaculus look like.
It’s hard for me to agree or disagree with timeline research being overrated, since I don’t have a great sense of how many total research hours are going into it, but I think Reason #4 is pretty important to this argument and seems wrong. The goodness of these broad strategic goals is pretty insensitive to timelines, but lots of specific actions wind up seeming worth doing or not worth doing based on timelines. I find myself seriously saying something like “Ugh, as usual, it all depends on AI timelines” in conversations about community-building strategy or career decisions like once a week.
For example, in this comment thread about whether and when to do immediately impactful work versus career-capital building, both the shape and the median of the AI x-risk distribution winds up mattering. A more object-level consideration means that “back-loaded” careers like policy look worse relative to “front-loaded” careers like technical research insofar as timelines are earlier.
In community-building, earlier timelines generally supports outreach strategies more focused on finding very promising technical safety researchers; moderate timelines support relatively more focus on policy field-building; and long timelines support more MacAskill-style broad longtermism, moral circle expansion, etc.
Of course, all of this is moot if the questions are super intractable, but I do think additional clarity would turn out to be useful for a pretty broad set of decision-makers—not just top funders or strategy-setters but implementers at the “foot soldier” level of community-building, all the way down to personal career choice.
Figuring out AGI timelines might be overrated compared to other AI forecasting questions (for eg: things related to takeoff / AI takeover etc) because the latter are more neglected. However, it still seems likely to me that more people within EA should be thinking about AGI timelines because so many personal career decisions are downstream of your beliefs about timelines.
Some things look much less appealing if you think AGI is < 10 years away, such as getting credentials and experience working on something that is not AI safety-related or spending time on community building projects directed towards high schoolers. It also feels like a lot of other debates are very linked to the question about timelines. For example, lots of people disagree about the probability of AI causing an existential catastrophe but my intuition is that the disagreement around this probability conditioning on specific timelines would be a lot less (that is, people with higher p(doom) compared to the average mostly think that way because they think timelines are shorter).
More timelines discourse would be good for the reputation of the community because it will likely convince others of AI x-risk being a massive problem. Non-EA folks I know who were previously unconcerned about AI x-risk were much more convinced when they read Holden’s posts on AI forecasting and learned about Ajeya’s bioanchors model (more than when they simply read descriptions of the alignment problem). More discussion of timelines would also signal to outsiders that we take this seriously.
It feels like people who have timelines similar to most other people in the community (~20 years away) would be more likely to agree with this than people with much shorter or longer timelines because, for the latter group, it makes sense to put more effort into convincing the community of their position or just because they can defer less to the community when deciding what to do with their career.
Lots of people in the community defer to others (esp Ajeya/bioanchors) when it comes to timelines but should probably spend more time developing their own thoughts and thinking about the implications of that.
I agree with Miles that EA often over-emphasizes AGI time-lines, and that this has less utility than generally assumed. I’d just add two additional points, one about the historical context of machine learning and AI research, and one about the relative risks of domain-specific versus ‘general’ AI.
My historical perspective comes from having worked on machine learning since the late 1980s. My first academic publication in 1989 developed a method of using genetic algorithms to design neural network architectures, and has been cited about 1,100 times since then. There was a lot of excitement in the late 80s about the new back-propagation algorithm for supervised learning in multi-layer neural networks. We expected that it would yield huge breakthroughs in many domains of AI in the next decade, the 1990s. We also vaguely expected that AGI would be developed within a couple of decades after that—probably by 2020. Back-propagation led to lots of cool work, but practical progress was slow, and we eventually lapse into the ‘AI winter’ of the 1990s, until deep learning methods were developed in the 2005-2010 era.
In the last decade, based on deep learning plus fast computers plus huge training datasets, we’ve seen awesome progress in many domain-specific applications of AI, from face recognition to chatbots to visual arts. But have we really made much progress in understanding how to get from domain-specific AI to true AGI of the sort that would impose sudden and unprecedented existential risks on humanity? How we even learned enough to seriously update our AGI timelines compared to what we expected in the late 1980s? I don’t think so. AGI still seems about 15-30 years away—just as it always has since the 1950s.
Even worse, I don’t think the cognitive sciences have really made much serious progress on understanding what an AGI cognitive architecture would even look like—or how it would plausibly lead to existential risks. (I’ll write more about this in due course.)
My bigger concern is that a fixation on AGI timelines in relation to X risk can distract attention from domain-specific progress in AI that could impose much more immediate, plausible, and concrete global catastrophic risks on humanity.
I’d like to see AI timelines for developing cheap, reliable autonomous drone swarms capable of assassinating heads of state and provoking major military conflicts. Or AI timelines for developing automated financial technologies capable of hacking major asset markets or crypto protocols with severe enough consequences that they impose high risks of systemic liquidation cascades in the global financial system, resulting in mass economic suffering. Or AI timelines for developing good enough automated deepfake video technologies that citizens can’t trust any video news sources, and military units can’t trust any orders from their own commanders-in-chief.
There are so many ways that near-term, domain-specific AI could seriously mess up our lives, and I think they deserve more attention. An over-emphasis on fine-tuning our AGI timelines seems to have distracted quite a few talented EAs from addressing those issues.
(Of course, a cynical take would be that under-researching the near-term global catastrophic risks of domain-specific AI will increase the probability that those risks get realized in the next 10-20 years, and they will cause such social, economic, and technological disruption that AGI research is delayed by many decades. Which, I guess, could be construed as one clever but counter-intuitive way to reduce AGI X risk.)
I tend to think diversification in EA is important even if we think there’s a high chance of AGI by 2040. Working on other issues gives us better engagement with policy makers and the public, improves the credibility of the movement, and provides more opportunities to get feedback on what does or doesn’t work for maximizing impact. Becoming insular or obsessive about AI would be alienating to many potential allies and make it harder to support good epistemic norms. And there are other causes where we can have a positive effect without directly competing for resources, because not all participants and funders are willing or able to work on AI.
Agreed! Indeed, I think AGI timelines research is even less useful than this post implies; I think just about all of the work to date didn’t help and shouldn’t have been a priority.
I disagree with Reason 6 as a thing that should influence our behavior; if we let our behavior be influenced by reputational risks as small as this, IMO we’ll generally be way too trigger-happy about hiding our honest views in order to optimize reputation, which is not a good way to make intellectual progress or build trust.
Agreed.
I’m not sure exactly what you have in mind here, but at a glance, this doesn’t sound like a high priority to me. I don’t think we have wheels to reinvent; the priority is to figure out how to do alignment at all, not to improve communication channels so we can share our current absence-of-ideas.
I would agree, however, that it’s very high-priority to get people on the same page about basic things like ‘we should be trying to figure out alignment at all’, insofar as people aren’t on that page.
Getting some people into gov seems fine to me, but probably not on the critical path. Getting good people into companies seems more on the critical path to me, but this framing seems wrong to me, because of my background model that (e.g.) we’re hopelessly far from knowing how to do alignment today.
I think the priority should be to cause people to think about alignment who might give humanity a better idea of a realistic way we could actually align AGI systems, not to find nice smart people and reposition them to places that vaguely seem more important. I’d guess most ‘placing well-intentioned people at important-seeming AI companies’ efforts to date have been net-negative.
Seems like plausibly a bad idea to me. I don’t see a way this can realistically help outside of generically slowing the field down, and I’m not sure even this would be net-positive, given the likely effect on ML discourse?
I’d at least want to hear more detail, rather than just “let’s regulate AI, because something must be done, and this is something”.
I would specifically say ‘figure out how to do technical alignment of AGI systems’. (Still speaking from my own models.)
Clarifying the kind of timelines work I think is low-importance:
I think there’s value in distinguishing worlds like “1% chance of AGI by 2100” versus “10+% chance”, and distinguishing “1% chance of AGI by 2050” versus “10+% chance”.
So timelines work enabling those updates was good.[1]
But I care a lot less about, e.g., “2/3 by 2050” versus “1/3 by 2050″.
And I care even less about distinguishing, e.g., “30% chance of AGI by 2030, 80% chance of AGI by 2050” from “15% chance of AGI by 2030, 50% chance of AGI by 2050″.
Though I think it takes very little evidence or cognition to rationally reach 10+% probability of AGI by 2100.
One heuristic way of seeing this is to note how confident you’d need to be in ‘stuff like the deep learning revolution (as well as everything that follows it) won’t get us to AGI in the next 85 years’, in order to make a 90+% prediction to that effect.
Notably, you don’t need a robust or universally persuasive 10+% in order to justify placing the alignment problem at or near the top of your priority list.
You just needs that to be your subjective probability at all, coupled with a recognition that AGI is an absurdly big deal and aligning the first AGI systems looks non-easy.
What about distinguishing 50% by 2050 vs. 50% by 2027?
In retrospect I should have made a clearer distinction between “things that the author thinks are good and which are mostly timeline-insensitive according to his model of how things work” and “things that all reasonable observers would agree are good ideas regardless of their timelines.” The stuff you mentioned mostly relates to currently-existing-AI-systems and management of their risks, and while not consensus-y, are mostly agreed on by people in the trenches of language model risks—for example, there is a lot of knowledge to share and which is being shared already about language model deployment best practices. And one needn’t invoke/think one way or the other about AGI to justify government intervention in managing risks of existing and near-term systems given the potential stakes of failure (e.g. collapse of the epistemic commons via scaled misuse of increasingly powerful language/image generation; reckless deployment of such systems in critical applications). Of course one might worry that intervening on those things will detract resources from other things, but my view, which I can’t really justify concisely here but happy to discuss in another venue, is that overwhelmingly the synergies outweigh the tradeoffs (e.g. there are big culture/norm benefits at the organizational and industry level—which will directly increase the likelihood of good AGI outcomes if the same orgs/people are involved—of being careful about current technologies compared to not doing so, even if the techniques themselves are very different).
Yeah, I’m specifically interested in AGI / ASI / “AI that could cause us to completely lose control of the future in the next decade or less”, and I’m more broadly interested in existential risk / things that could secure or burn the cosmic endowment. If I could request one thing, it would be clarity about when you’re discussing “acutely x-risky AI” (or something to that effect) versus other AI things; I care much more about that than about you flagging personal views vs. consensus views.
I agree on regulations. Our general prior should look like public choice theory. Regulations have a tendency to drift toward unintended kinds, usually with a more rent-seeking focus than planned. They also tend to have more unintended consequences than people predict.
There probably are some that pass a cost-benefit, but as a general prior, we should be very reluctant, and have very high standards. Getting serious AI regulations started has a very high chance of misfiring or overshooting or backfiring.
Could you please elaborate on this? The reasoning here seems non-obvious.
I agree with the spirit of claim. Timeline information is probably not used for much.
One thing I disagree with:
But convincing smart people to work on alignment is also convincing those smart people to not work on something else, and there are large opportunity costs. It doesn’t seem true that it’s regardless of timelines, unless you assume the variability in plausible timelines is on the short side.
Also, the regulations that are actually value-adding seems at least somewhat timeline dependent.
Still, I think this essay makes an important point—there’s a lot of babble about timelines, which is extremely unlikely to have alpha on those predictions. And there’s a large opportunity cost to spending time talking about it. Smart people’s time are extremely valuable, but even ignoring that, life is short.
The best timeline estimates are far more likely to come from institutions that specialize into forecasting, who can take advantage of the most modern, best methods. Other people who aren’t using those methods can still talk about the topic if they want to, but it’s very unlikely they’ll come up with timelines that are better.
I don’t have time for a long reply, but I think the perspective in this post would be good to keep in mind: https://forum.effectivealtruism.org/posts/FpjQMYQmS3rWewZ83/effective-altruism-is-a-question-not-an-ideology
By putting an answer (reduce AI risk) ahead of the question (how can we do the most good?) we would be selling ourselves short.
Some people, maybe a lot of people, should probably choose to focus fully on AI safety and stop worrying about cause prioritization. But nobody should feel like they’re being pushed into that or like other causes are worthless. EA should be a big tent. I don’t agree that it’s easier to rally people around a narrow cause; on the contrary, single minded focus on AI would drive away all but a small fraction of potential supporters, and have an evaporative cooling effect on the current community too.
18 years is a marathon, not a sprint.
TL;DR: Some people care about whether AGI risk is “longtermism” or a threat to their own life [1] [2]