Superforecaster, former philosophy PhD, Giving What We Can member since 2012. Currently trying to get into AI governance.
David Mathersđ¸
Karma: 5,940
I guess Iâm just slightly confused about what economists actually think here since Iâd always thought they took the idea that markets and investors were mostly quite efficient most of the time fairly seriously.
I donât know if/âhow much EA money should go to AI safety either. EAs are trying to find the single best thing, and itâs very hard to know what that is, and many worthwhile things will fail that bar. Maybe David Thorstad is right, and small X-risk reductions have relatively low value because another X-risk will get us in the next few centuries anyway*. What I do think is that society as a whole spending some resources caring about the risk of AGI arriving in the next ten years is likely optimal, and that itâs not more silly to do so than to do many other obviously good things. I donât actually give to AI safety myself, and I only work on AI-related stuff-forecasting etc., Iâm not a techy person-because itâs what people are prepared to pay more for, and people being prepared to pay me to work on near-termist causes is less common, though it does happen. I myself give to animal welfare, not AI safety.
If you really believe that everyone putting money into Open AI etc. will only see returns if they achieve AGI that seems to me to be a point in favour of âthere is a non-negligible risk of AGI in the next 10 yearsâ. I donât believe that, but if I did I that alone would significantly raise the chance I give to AGI within the next 10 years. But yes, they have some incentive to lie here, or to lie to themselves, obviously. Nonetheless, I donât think that means their opinion should get zero weight. For it to actually have been some amazing strategy for them to talk up the chances of AGI, *because it attracted cash* youâd have to believe they can fool outsiders with serious money on the line, and that this will be profitable for them in the long term, rather than crashing and burning when AGI does not arrive. I donât think that is wildly unlikely or anything, indeed, I think it is somewhat plausible-though my guess is Anthropic in particular believe their own hype. But it does require a fairly high amount of foolishness on behalf of other quite serious actors. Iâm much more sure of âraising large amounts of money for stuff that obviously wonât work is relatively hardâ than I am of any argument about how far we are from AGI that looks at the direct evidence, since the latter sort of arguments are very hard to evaluate. Iâd feel very differently here if we were arguing about 50% chance of AI in ten years, or even 10% chance. Itâs common for people to invest in things that probably wonât work but have a high pay-off if they do. But what your saying is that Richard is wrong for thinking there is a non-negligible risk, because the chance is significantly under 1%. I doubt there are many takers for like â1 in 1000â chance of a big pay-off.
It is of course not THAT unlikely that they are fooling the serious money: serious investors make mistakes and even the stock market does. Nonetheless, being able to attract serious investment that is genuinely only investing because they think youâll achieve X, whilst simultaneously being under huge media attention and scrutiny is a credible signal that youâll eventually achieve X.
I donât think the argument Iâve just given is all that definitive, because they have other incentives to hype, like attracting top researchers (who I think it is probably eaiser to fool, because if they are fooled about AGI working at a big lab was probably good for them anyway; quite different from what happens to people funding the labs who are fooled who just lose money.) So itâs possible that the people pouring serious money in donât take any of the AGI stuff seriously. Nonetheless, I trust âserious organisations with technical prowess seem to be trying to do thisâ as a signal to take something minimally seriously, even if they have some incentive to lie.
Similarly, if you really think Microsoft and Google have taken decisions that will crash their stock if AGI doesnât arrive, I think a similar argument applies: Are you really sure youâre better at evaluating whether there is a non-negligible chance that a tech will be achieved by the tech industry than Microsoft and Google? Eventually, if AGI is not arriving from the huge training runs that are being planned in the near future, people will notice, and Microsoft and Google donât want to lose money in 5 years from now either. Again, itâs not THAT implausible that they are mistaken, mistakes happen. But you arenât arguing that there probably wonât be AGI in ten years-a claim I actually strongly agree with!-but rather that Richard was way out in saying that itâs a tail risk we should take seriously given how important it would be.
Slower progress on one thing than another does not mean no progress on the slower thing.
âdespite those benchmarks not really being related to AGI in any way.â This is your judgment, but clearly it is not the judgment of some of the worldâs leading scientific experts in the area. (Though there may well be other experts who agree with you.)
*Actually Thorstadâs opinion is more complicated than that, he says that this is true conditional on X-risk currently being non-negligible, but he doesnât himself endorse the view that it is currently non-negligible as far as I can tell.
METR has an official internal view on what time horizons correspond to âtakeover not ruled outâ?
Yeah, I am inclined to agree-for what my opinion is worth which on this topic is probably not that much-that there will be many things AIs canât do even once they have a METR 80% time-horizon of say 2 days. But I am less sure of that than I am of the meta-level point about this being an important crux.
Sure, but I I wasnât really thinking of people on LessWrong, but rather of the fact that at least some relevant experts outside of the LW milieu seem worried and/âor think that AGI is not THAT far. I.e. Hinton, Bengio, Stuart Russell (for danger) and even people often quoted as skeptical experts* like Gary Marcus or Yann LeCunn often give back of the envelope timelines of 20 years, which is not actually THAT long. Furthermore I actually do think the predictions of relatively near term AGI by Anthropic and the fact that DeepMind and OpenAI have building AGI as a goal to carry some weight here. Please donât misunderstand me, I am not saying that these orgs are trustworthy exactly: I expect them to lie in their own interests to some degree, including about how fast their models will get better, and also to genuinely overestimate how fast they will make progress. Nonetheless they are somewhat credible in the sense that a) they are at the absolute cutting edge of the science here and have made some striking advancements, and b) they have some incentive not to overpromise so much that no one ever believes anything they say ever again, and c) they are convincing enough to outsiders with money that they keep throwing large sums of money at them, which suggests those outsiders at least expect reasonably rapid advancement, whether or not they expect AGI itself, and which is also evidence that these are serious organizations.
Iâd also say that near-term AGI is somewhat disanalogous to Hinduism, ESP, Christianity, crystal healing etc. in that all these things are actually in conflict with a basic scientific worldview fairly directly, in that they describe things that would plausibly violate known laws of physics, or are clearly supernatural in a fairly straightforward sense. Thatâs not true of near-term AGI.
Having said that I certainly agree that it is not completely obvious that there is enough real expertise behind predictions of near-term AGI to treat them with deference. My personal judgment is that there is, but once we get away from obvious edge cases like textbook hard science on the one hand and âexpertsâ in plainly supernatural things on the other, it gets hard to tell how much deference people deserve.
Thereâs also an issue of priors here of course: I donât think âAGI will be developed in the next 100 yearsâ is an âextraordinaryâ claim in the same sense as supernatural claims, or even just something unlikely but possible like âScotland will win the next football world cupâ. We know it is possible in principle, and that technology can advance quickly over a timespan of decades-just compare where flight was in 1900 to 1960-and that trillions of dollars are going to be spent advancing AI in the near term, and while Mooreâs law is breaking down, we havenât actually hit theoretical in principle limits on how good chips can be, and that more money and labour is currently going into making advanced AI than ever before. If we say thereâs a 25% chance of it being developed in the next hundred years, an even divide per decade of that would say 2.5% chance of it arriving next decade. Even if we cut that 5x for the next decade, that would give a 0.5% chance which I think is worth worrying about given how dramatic the consequence of AGI would be. (Of course, you personally have lots of arguments against it being near, but I am just talking about what itâs reasonable to expect from broad features of the current situation before we get into the details.) But of course, forecasting technology 100 years out is extraordinarily hard. In general because forecasting is hard beyond the next few years, so I guess maybe 25% is way too high (although the uncertainty cuts both ways.)
I get that the worry here is that people can just say any possible technology might happen soon, so if it was very consequential we should worry about it now. But my response is just that if itâs a technology that several of the worldâs largest or, fastest growing or most innovative companies claim or hint to be building it, and a Nobel winning scientist in the area in question agree that they very well might be, probably this is right, whereas if no one serious is working towards a technology, a higher degree of skepticism is probably warranted. (Presumption could be overcome if almost all scientists with relevant expertise think that Bengio and Hinton are complete cranks on this, but I havenât seen strong evidence of that.)
*In fairness, my guess is that they are actually more bullish on AGI than many people in machine learning, but that is only a guess.
I donât think this is sufficient to explain EA disinterest, because there are also neartermist EAs who are skeptical about near-term AGI, or just donât incorporate it into their assessment of cause areas and interventions.
Somewhat surprised to hear that people can successfully pull that off.
It seems to me like the really important thing is interpreting what âMETR 80% time horizon goes to a yearâ, or whatever endpoint you have in mind actually means. Itâs important if that takes longer than AI2027 predicts, obviously, but it seems more crux-y to me whether getting to that point means transformative AI is near or not, since the difference between â3 years and 7 seven yearsâ say, while important seems less important to me than between âdefinitely in 7 yearsâ and âwho knows, could still be 20+ years awayâ.
I think part of the issue here probably is that EAs mostly donât think biodiversity is good in itself, and instead believe only humans and animals experiencing well-being is good, and that the impact on well-being of promoting biodiversity is complex, uncertain and probably varies a lot with how and where biodiversity is being promoted. Hard to try and direct biodiversity funding if you donât really clearly agree with raising biodiversity as a goal.
Oh, ok, I agree, if the number of deer is the same after as counterfactually, it seems plausibly net positive yes.
Also, itâs certainly not common sense that it is always better to have less beings with higher welfare. Itâs not common sense that a world with 10 incredibly happy people is better than one with a billion very slightly less happy people.
And not every theory that avoids the repugnant conclusion delivers this result, either.
I agree, it is unclear whether welfare is actually positive.
Those are fair point in themselves, but I donât think âless deer is fine, so long as they have a higher standard of livingâ has anything like the same commonsense standing as âwe should protect people from malaria with insecticide even if the insecticide hurts insectsâ.
And itâs not clear to me that assuming less deer is fine in itself even if their lives are good is avoiding taking a stance on the intractable philosophical debate, rather than just implicitly taking one side of it.
âA potentially lower-risk example might be the warble fly (Hypoderma), which burrows under the skin of cattle and deer, causing great discomfort, yet rarely kills its host. The warble fly is small in biomass, host-specific (so doesnât greatly affect other species), and has more limited interactions beyond its host-parasite relationship. Although it does reduce the grazing and reproductive activity of hosts, these effects are comparatively minor and could be offset with non-invasive fertility controlâ
Remember that itâs not uncontroversial that it is preferable to have less animals at higher welfare level, rather than more animals at lower welfare level. Where welfare is net positive either way, some population ethicists are going to say having more animals at a lower level of welfare can be better than less at a higher level of welfare. See for example: https://ââwww.cambridge.org/ââcore/ââjournals/ââutilitas/ââarticle/ââwhat-should-we-agree-on-about-the-repugnant-conclusion/ââEB52C686BAFEF490CE37043A0A3DD075 But also, even on critical level views designed to BLOCK the repugnant conclusion, it can sometimes be better to have more welfare subjects at a lower but still positive level of welfare, then less subjects at a higher level of welfare. So maybe itâs better to have more deer even when some of them have warble fly, than to have less deer, but none of them have warble fly.
And itâs not so much that I think I have zero evidence: I keep up with progress in AI to some degrees, I have some idea of what the remaining gaps are to general intelligence, Iâve seen the speed at which capabilities have improved in recent years etc. Itâs that how to evaluate that evidence is not obvious, and so simply presenting a skeptic with it probably wonât move them, especially as the skeptic-in this case you-probably already has most of the evidence I have anyway. If it was just some random person who had never heard of AI asking why I thought the chance of mildly-over-human level AI in 10 years was not far under 1%, there are things I could say. Itâs just you already know those things, probably, so thereâs not much point in my saying them to you.
Yeah, I agree that in some sense saying âwe should instantly reject a theory that recommends WDâ doesnât not combine super-well with belief in classical U, for the reasons you give. Thatâs compatible with classical Uâs problems with WD being less bad than NUâs problemâs with it, is all Iâm saying.
âIâm generally against this sort of appeal to authority. While Iâm open to hear the arguments of smart people, we should evaluate those arguments themselves and not the people giving them. So far, Iâve heard no argument that would change my opinion on this matter.â
I think this attitude is just a mistake if your goal is to form the most accurate credences you can. Obviously, it is always good practice to ask people for their arguments rather than only taking what they say on trust. But your evaluation of other peopleâs arguments is fallible, and you know it is fallible. So you should distribute some of your confidence to cases where your personal evaluations of credible peopleâs arguments are just wrong. This isnât the same as failing to question purported experts. I can question an expert, and even disagree with them overall, and still move my credences somewhat towards theirs. (Iâm much more confident about this general claim than I am about what credences in ASI in the next decade are or arenât reasonable, or how much credibility anyone saying ASI is coming in the next decade should get.)
âIt all comes down to the question of whether the current tech is relevant for ASI or not. In my estimation, it is not â something else entirely is required. The probability for us discovering that something else just now is low.â
I think Richardâs idea is that you shouldnât have *super-high* confidence in your estimation here, but should put some non-negligible credence on the idea that it is wrong, and current progress is relevant. Why be close to certainty about a question that you probably think is hard and that other smart people disagree about being the reasoning? And once you open yourself up to a small chance that current progress is in fact relevant, it then becomes at least somewhat unclear that you should be way below 1% in the chanc of AGI in the relatively near term or in current safety work being relevant. (Not necessarily endorsing the line of thought in this paragraph myself.)
It seems like if you find it incredible to deny and he doesnât, itâs very hard to make further progress :( Iâm on your side about the chance being over 1% in the next decade, I think, but I donât know how Iâd prove it to a skeptic, except to gesture and say that capabilities have improved loads in a short time, and it doesnât seem like the are >20 similar sized jumps before AGI. But when I ask myself what evidence I have for âthere are not >20 similar sized jumps before AGIâ I come up short. I donât necessarily think the burden of proof here is actually on people arguing that the chance of AGI in the next decade is non-negligible though: itâs a goal of some serious people within the relevant science, and they are not making zero progress, and some identifiable quantifiable individual capabilities have improved very fast. Plus the extreme difficulty of forecasting technological breakthroughs over more than a couple of years cuts both ways.
The report has many authors, some of whom maybe much less concerned or think the whole thing is silly. I never claimed that Bengio and Hintonâs views were a consensus, and in any case, I was citing their views as evidence for taking the idea that AGI may arrive soon seriously, not their views on how risky AI is. Iâm pretty sure Iâve seen them give relatively short time-lines when speaking individually, but I guess I could be misremembering. For what itâs worth Yann LeCunn seems to think 10 years is about right, and Gary Marcus seems to think a guess of 10-20 years is reasonable: https://ââhelentoner.substack.com/ââp/ââlong-timelines-to-advanced-ai-have