The AI bubble popping would be a strong signal that this [capabilities] optimism has been misplaced.
Are you presupposing that good practical reasoning involves (i) trying to picture the most-likely future, and then (ii) doing what would be best in that event (while ignoring other credible possibilities, no matter their higher stakes)?
It would be interesting to read a post where someone tries to explicitly argue for a general principle of ignoring credible risks in order to slightly improve most-probable outcomes. Seems like such a principle would be pretty disastrous if applied universally (e.g. to aviation safety, nuclear safety, and all kinds of insurance), but maybe thereās more to be said? But itās a bit frustrating to read takes where people just seem to presuppose some such anti-precautionary principle in the background.
To be clear: I take the decision-relevant background question here to not be the binary question Is AGI imminent? but rather something more degreed, like Is there a sufficient chance of imminent AGI to warrant precautionary measures? And I donāt see how the AI bubble popping would imply that answering āYesā to the latter was in any way unreasonable. (A bit like how you canāt say an election forecaster did a bad job just because their 40% candidate won rather than the one they gave a 60% chance to. Sometimes seeing the actual outcome seems to make people worse at evaluating othersā forecasts.)
Some supporters of AI Safety may overestimate the imminence of AGI. Itās not clear to me how much of a problem that is? (Many people overestimate risks from climate change. That seems important to correct if it leads them to, e.g., anti-natalism, or to misallocate their resources. But if it just leads them to pollute less, then it doesnāt seem so bad, and Iād be inclined to worry more about climate change denialism. Similarly, I think, for AI risk.) There are a lot more people who persist in dismissing AI risk in a way that strikes me as outrageously reckless and unreasonable, and so that seems by far the more important epistemic error to guard against?
That said, Iād like to see more people with conflicting views about AGI imminence arrange public bets on the topic. (Better calibration efforts are welcome. Iām just very dubious of the OPās apparent assumption that losing such a bet ought to trigger deep āsoul-searchingā. Itās just not that easy to resolve deep disagreements about what priors /ā epistemic practices are reasonable.)
Are you presupposing that good practical reasoning involves (i) trying to picture the most-likely future, and then (ii) doing what would be best in that event (while ignoring other credible possibilities, no matter their higher stakes)?
No, of course not.
I have written about this at length before, on multiple occasions (e.g. here and here, to give just two examples). I donāt expect everyone who reads one of my posts for the first time to know all that context and background ā why would they? ā but, also, the amount of context and background I have to re-explain every time I make a new post is already high because if I donāt, people will just raise the obvious objections I didnāt already anticipate and respond to in the post.
But, in, short: no.
Iām just very dubious of the OPās apparent assumption that losing such a bet ought to trigger deep āsoul-searchingā. Itās just not that easy to resolve deep disagreements about what priors /ā epistemic practices are reasonable.
I agree, but I didnāt say the AI bubble popping should settle the matter, only that I hoped it would motivate people to revisit the topic of near-term AGI with more open-mindedness and curiosity, and much less hostility toward people with dissenting opinions, given that there are already clear, strong objections ā and some quite prominently made, as in the case of Toby Ordās post on RL scaling ā to the majority view of the EA community that seem to have mostly escaped serious consideration.
You donāt need an external economic event to see that the made-up graphs in āSituational Awarenessā are ridiculous or that AI 2027 could not rationally convince anyone of anything who is not already bought-in to the idea of near-term AGI for other reasons not discussed in AI 2027. And so on. And if the EA community hasnāt noticed these glaring problems, what else hasnāt it noticed?
These are examples that anyone can (hopefully) easily understand with a few minutes of consideration. Anyone can click on one of the āSituational Awarenessā graphs and very quickly see that the numbers and lines are just made-up, or that the y-axis has an ill-defined unit of measurement (āeffective computeā, which is relative the tasks/āproblems compute is used for) or no unit of measurement (just āorders of magnitudeā, but orders of magnitude of what?) and also no numbers. Plus other ridiculous features, such as claiming that GPT-4 is an AGI.
With AI 2027, it takes more like 10-20 minutes to see that the whole thing is just based on a few guysā gut intuitions and nothing else. There are other glaring problems in EA discourse around AGI that take more time to explain, such as objections around benchmark construct validity or criterion validity. Even in cases where errors are clear, straightforward, objective, and relatively quick and simple to explain (see below), people often just ignore it when someone points them out. More complex or subtle errors will probably never be considered, even if they are consequential.
The EA community doesnāt have any analogue of peer review ā or it just barely does ā where people play the role of rigorously scrutinizing work to catch errors and make sure it meets a certain quality threshold. Some people in the community (probably a minority, but a vocal and aggressive minority) are disdainful of academic science in general and peer review in particular, and donāt think peer review or an analogue of it would actually be helpful. This makes things a little more difficult.
I recently caught two methodological errors in a survey question asked by the Forecasting Research Institute. Pointing them out was an absolutely thankless task and was deeply unpleasant. I got dismissed and downvoted, and if not for titotalās intervention one of the errors probably never would have gotten fixed. This is very discouraging.
Iām empathetic to the fact that producing research or opinion writing and getting criticized to death also feels deeply unpleasant and thankless, and Iām not entirely sure on the nuances of how to make both sides of the coin feel rewarded rather than punished, but surely there must be a way. Iāve seen it work out well before (and itās not like this is a new problem no one has dealt with before).
The FRI survey is one example, but one of many. In my observation, people in the EA community are not receptive to the sort of scrutiny that is commonplace in academic contexts. This could be anything from correcting someone on a misunderstanding of the definitions of technical terms used in machine learning or pointing out that Waymo vehicles still have a human in the loop (Waymo calls it āfleet responseā). The community pats itself on the back for āloving criticismā. I donāt think anybody really loves criticism ā only rarely ā and maybe the best we can hope for is to begrudgingly accept criticism. But that involves setting up a social and maybe even institutional process of criticism that currently doesnāt exist in the EA community.
When I say ānot receptiveā, I donāt just mean that people hear the scrutiny and just disagree ā thatās not inherently problematic, and could be what being receptive to scrutiny looks like ā I mean that, for example, they downvote posts/ācomments and engage in personal insults or accusations (e.g. explicit accusations of ābad faithā, of which there is one in the comments on this very post), or other hostile behaviour that discourages the scrutiny. Only my masochism allows me to continue posting and commenting on the EA Forum. I honestly donāt know if I have the stomach to do this long-term. Itās probably a bad idea to try.
The Unjournal seems like it could be a really promising project in the area of scrutiny and sober second thought. I love the idea of commissioning outside experts to review EA research. I think for organizations with the money to pay for this, this should be the default.
Iāll say just a little bit more on the topic of the precautionary principle for now. I have a complex multi-part argument on this, which will take some explaining that I wonāt try to do here. I have covered a lot of this in some previous posts and comments. The main three points Iād make in relation to the precautionary principle and AGI risk are:
Near-term AGI is highly unlikely, much less than a 0.05% chance in the next decade
We donāt have enough knowledge of how AGI will be built to usefully prepare now
As knowledge of how to build AGI is gained, investment into preparing for AGI becomes vastly more useful, such that the benefits of investing resources into preparation at higher levels of knowledge totally overwhelm the benefits of investing resources at lower levels of knowledge
In principle, of course, but how? There are various practical obstacles such as:
Are such bets legal?
How do you compel people to pay up?
Why would someone on the other side of the bet want to take it?
I donāt have spare money to be throwing at Internet stunts where thereās a decent chance that, e.g. someone will just abscond with my money and Iāll have no recourse (or at least nothing cost-effective)
If itās a bet that takes a form where if AGI isnāt invented by January 1, 2036, people have to pay me a bunch of money (and vice versa), of course Iāll accept such bets gladly in large sums.
I would also be willing to take bets of that form for good intermediate proxies for AGI, which would take a bit of effort to figure out, but that seems doable. The harder part is figuring out how to actually structure the bet and ensure payment (if this is even legal in the first place).
From my perspective, itās free money, and Iāll gladly take free money (at least from someone wealthy enough to have money to spare ā I would feel bad taking it from someone who isnāt financially secure). But even though similar bets have been made before, people still donāt have good solutions to the practical obstacles.
I wouldnāt want to accept an arrangement that would be financially irrational (or illegal, or not legally enforceable), though, and that would amount to essentially burning money to prove a point. That would be silly, I donāt have that kind of money to burn.
Also, if I were on the low probability end of a bet, Iād be more worried about the risk of measurement or adjudicator error where measuring the outcome isnāt entirely clear cut. Maybe a ruleset could be devised that is so objective and so well captures whether AGI exists that this concern isnāt applicable. But if thereās an adjudication/āerror error risk of (say) 2 percent and the error is equally likely on either side, itās much more salient to someone betting on (say) under 1 percent odds.
Some supporters of AI Safety may overestimate the imminence of AGI. Itās not clear to me how much of a problem that is?
It seems plausible that there could be significant adverse effects on AI Safety itself. Thereās been an increasing awareness of the importance of policy solutions, whose theory of impact requires support outside the AI Safety community. I think thereās a risk that AI Safety is becoming linked in the minds of third parties with a belief in AGI imminence in a way that will seriously if not irrevocably damage the formerās credibility in the event of a bubble /ā crash.
One might think that publicly embracing imminence is worth the risk, of course. For example, policymakers are less likely to endorse strong action for anything that is expected to have consequences many decades in the future. But being perceived as crying wolf if a bubble pops is likely to have some consequences.
@Richard Y Chappellšø, would you please do me the courtesy of acknowledging that you misunderstood my argument? I think this was a rather uncharitable reading on your part and would have been fairly easy to avoid. Your misreading was not explicitly forestalled by the text but not supported by the text, either, and there was much in the text to suggest I did not hold the view that you took to be the thesis or argument. I found your misreading discourteous for that reason.
Much of the post is focused on bad intellectual practices, such as:
Not admitting you got a prediction wrong after you got it wrong
Repeating the same prediction multiple times in a row and repeatedly getting it wrong, and seemingly not learning anything
Making fake graphs with false data, no data, dubious units of measurement, no units of measurements, and other problems or inaccuracies
Psychological or social psychological biases like millennialist cognitive bias, bias resulting from the intellectual and social insularity of the EA community, and possible confirmation bias (e.g. why hasnāt Toby Ordās RL scaling post gotten much more attention?)
Acceptance or tolerance of arguments and assertions that are really weak, unsupported, or sometimes just bad
I donāt interpret your comment as a defense or endorsement of any of these practices (although I could if I wanted to be combative and discourteous). Iām assuming you donāt endorse these practices and your comment was not intended as a defense of them.
So, why reply to a post that is largely focused on those things as if the thesis or argument or thrust of the post is something other than that, and which was not said in the text?
On the somewhat more narrow point of AI capabilities optimism, I think the AI bubble popping within the next 5 years or so would be strong evidence that the EA communityās AI capabilities optimism has been misplaced. If the large majority of people in the EA community only thought thereās a 0.1% chance or a 1% chance of AGI within a decade, then the AI bubble popping might not be that surprising from their point of view. But the actual majority view seems to be more like a 50%+ chance of AGI within a decade. My impression from discussions with various people in the EA community is that many of them would find it surprising if the AI bubble popped.
The difference between a 50%+ chance of AGI within a decade and a 0.1% chance is a lot from an epistemic perspective, even if, just for the sake of argument, it makes absolutely no difference for precautionary arguments about AI safety. So, I think misestimating the probability by that much would be worthy of discussion, even if ā for the sake of argument ā it doesnāt change the underlying case for AI safety.
It is especially worthy of discussion if the misestimation is influenced by bad intellectual practices, such as those listed above. All the information needed to diagnose those intellectual practices as bad is available today, so the AI bubble popping isnāt necessary. However, people in the EA community may be reluctant to give a hard look at them without some big external event like an AI bubble popping shaking them up. As I said in the post, Iām pessimistic that even after the AI bubble pops, people in the EA community will, even then, be willing to examine these intellectual practices and acknowledge that theyāre bad. But itās worth a shot for me to say something about it anyway.
There are many practical reasons to worry about bad intellectual practices. For example, people in AI safety should worry about whether theyāre making existential risk from AGI better or worse, and having bad intellectual practices on a systemic or widespread level will make it more likely theyāll screw this up. Or, given that, according to Denkenberger in another comment on this post, funding around existential risk from AGI has significantly taken away funding around other existential risks, overestimating existential risk from AGI based on bad intellectual practices might (counterfactually) increase total existential risk just by causing funding to be less wisely allocated. And, of course, there are many other reasons to worry about bad intellectual practices, especially if they are prevalent in a community and culturally supported by that community.
We both could list reasons on and on why thinking badly might lead to doing badly. Just one more example Iāll bring up is that, in practice, most AI safety work seems to make rather definite, specific assumptions about the underlying technical nature of AGI. If AI safety has (by and large) identified an implausible AI paradigm to underlie AGI out of at least several far more plausible and widely-known candidates (largely as a result of the bad intellectual practices listed above), then AI safety will be far less effective at achieving its goals. There might still be a strong precautionary argument for doing AI safety work on even that implausible AI paradigm, but given that AI safety, has, in practice, for the most part, bet on one specific horse and not the others, it is a problem to pick the wrong paradigm. You could maybe argue for an allocation of resources weighted to different AI paradigms based on their perceived plausibility, but that would still result in a large reallocation of resources if the paradigm AI safety is betting on is highly implausible and there are several other candidates that are much more plausible. So, I think this is a fair line of argument.
What matters is not just some unidimensional measure of the EA communityās beliefs like the median year of AGI or the probability of AGI within a certain timeframe or the probability of global catastrophe from AGI (conditional on its creation, or within a certain timeframe). If bad intellectual practices make that number go up too high, itās not necessarily just fine on precautionary grounds, it can mean existential risk is increased.
Honestly, I still think my comment was a good one! I responded to what struck me as themost cruxy claim in your post, explaining why I found it puzzling and confused-seeming. I then offered what I regard as an important corrective to a bad style of thinking that your post might encourage, whatever your intentions. (I made no claims about your intentions.) Youāre free to view things differently, but I disagree that there is anything ādiscourteousā about any of this.
Going back to the OPās claims about what is or isnāt āa good way to argue,ā I think itās important to pay attention to the actual text of what someone wrote. Thatās what my blog post did, and itās annoying to be subject to criticism (and now downvoting) from people who arenāt willing to extend the same basic courtesy to me.
You misunderstood my argument based on a misreading of the text. Simple as that.
Please extend the same courtesy to others that you request for yourself. Otherwise, itās just ārules for thee but for not meā.
As I see it, I responded entirely reasonably to the actual text of what you wrote. (Maybe what you wrote gave a misleading impression of what you meant or intended; again, I made no claims about the latter.)
Is there a way to mute comment threads? Pursuing this disagreement further seems unlikely to do anyone any good. For what itās worth, I wish you well, and Iām sorry that I wasnāt able to provide you with the agreement that youāre after.
Itās a bad habit to make up a ridiculous assertion that isnāt in the text youāre responding to, and then respond as if that assertion is in the text. Iāve been gravely concerned about low-probability, high-impact events since before the term āeffective altruismā existed. Iāve discussed this publicly for at least a decade. Iāve discussed it many times on this forum. I donāt need you to tell me to worry about low-probability, high-impact events, and to pretend that my post above is arguing that we should disregard those is uncharitable and discourteous. I donāt you need to explain the concept of expected value. Thatās absurd.
I started writing about existential risk from AI in 2015, based on Nick Bostromās Superintelligence book and other sources. Bostrom, of course, helped popularize the argument that very low probabilities of existential catastrophe have very high expected value. As mentioned in the post, Iāve been writing about AGI for a long time, and Iām clearly familiar with the basic arguments. Itās frustrating that your response was: well, wait, have you ever considered that worrying about low-probability events might be important? Yeah, of course I have. How could anyone familiar with this topic not have considered that?
The majority of people involved in EA currently seem to believe that thereās a 50%+ chance of AGI by 2035 ā with many setting their median year at or before 2032 ā and if that probability is 3-8 orders of magnitude too high, or something like that, because people in EA accept bad arguments and evidence without a reasonable minimum of scientific skepticism or critical appraisal, then thatās worthy of discussion. Thatās the main thing this post is about. Coming to wrong conclusions typically has negative consequences, as does having bad intellectual practices. Those negative consequences may include a failure to guard against existential risk wisely and well (e.g., it could mean the wrong kind of AI safety work is getting funded, rather than AI safety work getting funded too much overall). There may be other important negative consequences as well, such as reputational harm to EA, alienation of longstanding community members (like me), misallocation of funding away from places where itās direly needed (such as non-existentially dangerous pandemics or global poverty or animal welfare), or serious physical or psychological harm to individuals due to a few people holding apocalyptic or millennialist views ā just to name a few. Why not respond to that argument about the EA communityās intellectual practices, which is what is emphasized again and again and again in the post, rather than something that isnāt in the text?
This misunderstanding was a pretty avoidable mistake on your part ā which is okay, mistakes are forgivable ā and it would have been easy to simply acknowledge your mistake after I pointed it out. Why canāt you extend that basic courtesy?
I donāt think you should complain about David Thorstad or anyone else allegedly misrepresenting your arguments when youāre not upholding the principle here that you asked Thorstad to respect. You could have engaged in a more generous disagreement and maybe we could have had a constructive discussion. I was polite and patient the first time I pointed out you got my view wrong, more confrontational the second time, and now Iām pointing it out again in a state of complete expasteration. You had multiple chances to come to the table. What gives?
Are you presupposing that good practical reasoning involves (i) trying to picture the most-likely future, and then (ii) doing what would be best in that event (while ignoring other credible possibilities, no matter their higher stakes)?
It would be interesting to read a post where someone tries to explicitly argue for a general principle of ignoring credible risks in order to slightly improve most-probable outcomes. Seems like such a principle would be pretty disastrous if applied universally (e.g. to aviation safety, nuclear safety, and all kinds of insurance), but maybe thereās more to be said? But itās a bit frustrating to read takes where people just seem to presuppose some such anti-precautionary principle in the background.
To be clear: I take the decision-relevant background question here to not be the binary question Is AGI imminent? but rather something more degreed, like Is there a sufficient chance of imminent AGI to warrant precautionary measures? And I donāt see how the AI bubble popping would imply that answering āYesā to the latter was in any way unreasonable. (A bit like how you canāt say an election forecaster did a bad job just because their 40% candidate won rather than the one they gave a 60% chance to. Sometimes seeing the actual outcome seems to make people worse at evaluating othersā forecasts.)
Some supporters of AI Safety may overestimate the imminence of AGI. Itās not clear to me how much of a problem that is? (Many people overestimate risks from climate change. That seems important to correct if it leads them to, e.g., anti-natalism, or to misallocate their resources. But if it just leads them to pollute less, then it doesnāt seem so bad, and Iād be inclined to worry more about climate change denialism. Similarly, I think, for AI risk.) There are a lot more people who persist in dismissing AI risk in a way that strikes me as outrageously reckless and unreasonable, and so that seems by far the more important epistemic error to guard against?
That said, Iād like to see more people with conflicting views about AGI imminence arrange public bets on the topic. (Better calibration efforts are welcome. Iām just very dubious of the OPās apparent assumption that losing such a bet ought to trigger deep āsoul-searchingā. Itās just not that easy to resolve deep disagreements about what priors /ā epistemic practices are reasonable.)
No, of course not.
I have written about this at length before, on multiple occasions (e.g. here and here, to give just two examples). I donāt expect everyone who reads one of my posts for the first time to know all that context and background ā why would they? ā but, also, the amount of context and background I have to re-explain every time I make a new post is already high because if I donāt, people will just raise the obvious objections I didnāt already anticipate and respond to in the post.
But, in, short: no.
I agree, but I didnāt say the AI bubble popping should settle the matter, only that I hoped it would motivate people to revisit the topic of near-term AGI with more open-mindedness and curiosity, and much less hostility toward people with dissenting opinions, given that there are already clear, strong objections ā and some quite prominently made, as in the case of Toby Ordās post on RL scaling ā to the majority view of the EA community that seem to have mostly escaped serious consideration.
You donāt need an external economic event to see that the made-up graphs in āSituational Awarenessā are ridiculous or that AI 2027 could not rationally convince anyone of anything who is not already bought-in to the idea of near-term AGI for other reasons not discussed in AI 2027. And so on. And if the EA community hasnāt noticed these glaring problems, what else hasnāt it noticed?
These are examples that anyone can (hopefully) easily understand with a few minutes of consideration. Anyone can click on one of the āSituational Awarenessā graphs and very quickly see that the numbers and lines are just made-up, or that the y-axis has an ill-defined unit of measurement (āeffective computeā, which is relative the tasks/āproblems compute is used for) or no unit of measurement (just āorders of magnitudeā, but orders of magnitude of what?) and also no numbers. Plus other ridiculous features, such as claiming that GPT-4 is an AGI.
With AI 2027, it takes more like 10-20 minutes to see that the whole thing is just based on a few guysā gut intuitions and nothing else. There are other glaring problems in EA discourse around AGI that take more time to explain, such as objections around benchmark construct validity or criterion validity. Even in cases where errors are clear, straightforward, objective, and relatively quick and simple to explain (see below), people often just ignore it when someone points them out. More complex or subtle errors will probably never be considered, even if they are consequential.
The EA community doesnāt have any analogue of peer review ā or it just barely does ā where people play the role of rigorously scrutinizing work to catch errors and make sure it meets a certain quality threshold. Some people in the community (probably a minority, but a vocal and aggressive minority) are disdainful of academic science in general and peer review in particular, and donāt think peer review or an analogue of it would actually be helpful. This makes things a little more difficult.
I recently caught two methodological errors in a survey question asked by the Forecasting Research Institute. Pointing them out was an absolutely thankless task and was deeply unpleasant. I got dismissed and downvoted, and if not for titotalās intervention one of the errors probably never would have gotten fixed. This is very discouraging.
Iām empathetic to the fact that producing research or opinion writing and getting criticized to death also feels deeply unpleasant and thankless, and Iām not entirely sure on the nuances of how to make both sides of the coin feel rewarded rather than punished, but surely there must be a way. Iāve seen it work out well before (and itās not like this is a new problem no one has dealt with before).
The FRI survey is one example, but one of many. In my observation, people in the EA community are not receptive to the sort of scrutiny that is commonplace in academic contexts. This could be anything from correcting someone on a misunderstanding of the definitions of technical terms used in machine learning or pointing out that Waymo vehicles still have a human in the loop (Waymo calls it āfleet responseā). The community pats itself on the back for āloving criticismā. I donāt think anybody really loves criticism ā only rarely ā and maybe the best we can hope for is to begrudgingly accept criticism. But that involves setting up a social and maybe even institutional process of criticism that currently doesnāt exist in the EA community.
When I say ānot receptiveā, I donāt just mean that people hear the scrutiny and just disagree ā thatās not inherently problematic, and could be what being receptive to scrutiny looks like ā I mean that, for example, they downvote posts/ācomments and engage in personal insults or accusations (e.g. explicit accusations of ābad faithā, of which there is one in the comments on this very post), or other hostile behaviour that discourages the scrutiny. Only my masochism allows me to continue posting and commenting on the EA Forum. I honestly donāt know if I have the stomach to do this long-term. Itās probably a bad idea to try.
The Unjournal seems like it could be a really promising project in the area of scrutiny and sober second thought. I love the idea of commissioning outside experts to review EA research. I think for organizations with the money to pay for this, this should be the default.
Iāll say just a little bit more on the topic of the precautionary principle for now. I have a complex multi-part argument on this, which will take some explaining that I wonāt try to do here. I have covered a lot of this in some previous posts and comments. The main three points Iād make in relation to the precautionary principle and AGI risk are:
Near-term AGI is highly unlikely, much less than a 0.05% chance in the next decade
We donāt have enough knowledge of how AGI will be built to usefully prepare now
As knowledge of how to build AGI is gained, investment into preparing for AGI becomes vastly more useful, such that the benefits of investing resources into preparation at higher levels of knowledge totally overwhelm the benefits of investing resources at lower levels of knowledge
Is this something youāre willing to bet on?
In principle, of course, but how? There are various practical obstacles such as:
Are such bets legal?
How do you compel people to pay up?
Why would someone on the other side of the bet want to take it?
I donāt have spare money to be throwing at Internet stunts where thereās a decent chance that, e.g. someone will just abscond with my money and Iāll have no recourse (or at least nothing cost-effective)
If itās a bet that takes a form where if AGI isnāt invented by January 1, 2036, people have to pay me a bunch of money (and vice versa), of course Iāll accept such bets gladly in large sums.
I would also be willing to take bets of that form for good intermediate proxies for AGI, which would take a bit of effort to figure out, but that seems doable. The harder part is figuring out how to actually structure the bet and ensure payment (if this is even legal in the first place).
From my perspective, itās free money, and Iāll gladly take free money (at least from someone wealthy enough to have money to spare ā I would feel bad taking it from someone who isnāt financially secure). But even though similar bets have been made before, people still donāt have good solutions to the practical obstacles.
I wouldnāt want to accept an arrangement that would be financially irrational (or illegal, or not legally enforceable), though, and that would amount to essentially burning money to prove a point. That would be silly, I donāt have that kind of money to burn.
Also, if I were on the low probability end of a bet, Iād be more worried about the risk of measurement or adjudicator error where measuring the outcome isnāt entirely clear cut. Maybe a ruleset could be devised that is so objective and so well captures whether AGI exists that this concern isnāt applicable. But if thereās an adjudication/āerror error risk of (say) 2 percent and the error is equally likely on either side, itās much more salient to someone betting on (say) under 1 percent odds.
It seems plausible that there could be significant adverse effects on AI Safety itself. Thereās been an increasing awareness of the importance of policy solutions, whose theory of impact requires support outside the AI Safety community. I think thereās a risk that AI Safety is becoming linked in the minds of third parties with a belief in AGI imminence in a way that will seriously if not irrevocably damage the formerās credibility in the event of a bubble /ā crash.
One might think that publicly embracing imminence is worth the risk, of course. For example, policymakers are less likely to endorse strong action for anything that is expected to have consequences many decades in the future. But being perceived as crying wolf if a bubble pops is likely to have some consequences.
@Richard Y Chappellšø, would you please do me the courtesy of acknowledging that you misunderstood my argument? I think this was a rather uncharitable reading on your part and would have been fairly easy to avoid. Your misreading was not explicitly forestalled by the text but not supported by the text, either, and there was much in the text to suggest I did not hold the view that you took to be the thesis or argument. I found your misreading discourteous for that reason.
Much of the post is focused on bad intellectual practices, such as:
Not admitting you got a prediction wrong after you got it wrong
Repeating the same prediction multiple times in a row and repeatedly getting it wrong, and seemingly not learning anything
Making fake graphs with false data, no data, dubious units of measurement, no units of measurements, and other problems or inaccuracies
Psychological or social psychological biases like millennialist cognitive bias, bias resulting from the intellectual and social insularity of the EA community, and possible confirmation bias (e.g. why hasnāt Toby Ordās RL scaling post gotten much more attention?)
Acceptance or tolerance of arguments and assertions that are really weak, unsupported, or sometimes just bad
I donāt interpret your comment as a defense or endorsement of any of these practices (although I could if I wanted to be combative and discourteous). Iām assuming you donāt endorse these practices and your comment was not intended as a defense of them.
So, why reply to a post that is largely focused on those things as if the thesis or argument or thrust of the post is something other than that, and which was not said in the text?
On the somewhat more narrow point of AI capabilities optimism, I think the AI bubble popping within the next 5 years or so would be strong evidence that the EA communityās AI capabilities optimism has been misplaced. If the large majority of people in the EA community only thought thereās a 0.1% chance or a 1% chance of AGI within a decade, then the AI bubble popping might not be that surprising from their point of view. But the actual majority view seems to be more like a 50%+ chance of AGI within a decade. My impression from discussions with various people in the EA community is that many of them would find it surprising if the AI bubble popped.
The difference between a 50%+ chance of AGI within a decade and a 0.1% chance is a lot from an epistemic perspective, even if, just for the sake of argument, it makes absolutely no difference for precautionary arguments about AI safety. So, I think misestimating the probability by that much would be worthy of discussion, even if ā for the sake of argument ā it doesnāt change the underlying case for AI safety.
It is especially worthy of discussion if the misestimation is influenced by bad intellectual practices, such as those listed above. All the information needed to diagnose those intellectual practices as bad is available today, so the AI bubble popping isnāt necessary. However, people in the EA community may be reluctant to give a hard look at them without some big external event like an AI bubble popping shaking them up. As I said in the post, Iām pessimistic that even after the AI bubble pops, people in the EA community will, even then, be willing to examine these intellectual practices and acknowledge that theyāre bad. But itās worth a shot for me to say something about it anyway.
There are many practical reasons to worry about bad intellectual practices. For example, people in AI safety should worry about whether theyāre making existential risk from AGI better or worse, and having bad intellectual practices on a systemic or widespread level will make it more likely theyāll screw this up. Or, given that, according to Denkenberger in another comment on this post, funding around existential risk from AGI has significantly taken away funding around other existential risks, overestimating existential risk from AGI based on bad intellectual practices might (counterfactually) increase total existential risk just by causing funding to be less wisely allocated. And, of course, there are many other reasons to worry about bad intellectual practices, especially if they are prevalent in a community and culturally supported by that community.
We both could list reasons on and on why thinking badly might lead to doing badly. Just one more example Iāll bring up is that, in practice, most AI safety work seems to make rather definite, specific assumptions about the underlying technical nature of AGI. If AI safety has (by and large) identified an implausible AI paradigm to underlie AGI out of at least several far more plausible and widely-known candidates (largely as a result of the bad intellectual practices listed above), then AI safety will be far less effective at achieving its goals. There might still be a strong precautionary argument for doing AI safety work on even that implausible AI paradigm, but given that AI safety, has, in practice, for the most part, bet on one specific horse and not the others, it is a problem to pick the wrong paradigm. You could maybe argue for an allocation of resources weighted to different AI paradigms based on their perceived plausibility, but that would still result in a large reallocation of resources if the paradigm AI safety is betting on is highly implausible and there are several other candidates that are much more plausible. So, I think this is a fair line of argument.
What matters is not just some unidimensional measure of the EA communityās beliefs like the median year of AGI or the probability of AGI within a certain timeframe or the probability of global catastrophe from AGI (conditional on its creation, or within a certain timeframe). If bad intellectual practices make that number go up too high, itās not necessarily just fine on precautionary grounds, it can mean existential risk is increased.
Honestly, I still think my comment was a good one! I responded to what struck me as the most cruxy claim in your post, explaining why I found it puzzling and confused-seeming. I then offered what I regard as an important corrective to a bad style of thinking that your post might encourage, whatever your intentions. (I made no claims about your intentions.) Youāre free to view things differently, but I disagree that there is anything ādiscourteousā about any of this.
To quote you from another thread:
You misunderstood my argument based on a misreading of the text. Simple as that.
Please extend the same courtesy to others that you request for yourself. Otherwise, itās just ārules for thee but for not meā.
As I see it, I responded entirely reasonably to the actual text of what you wrote. (Maybe what you wrote gave a misleading impression of what you meant or intended; again, I made no claims about the latter.)
Is there a way to mute comment threads? Pursuing this disagreement further seems unlikely to do anyone any good. For what itās worth, I wish you well, and Iām sorry that I wasnāt able to provide you with the agreement that youāre after.
Itās a bad habit to make up a ridiculous assertion that isnāt in the text youāre responding to, and then respond as if that assertion is in the text. Iāve been gravely concerned about low-probability, high-impact events since before the term āeffective altruismā existed. Iāve discussed this publicly for at least a decade. Iāve discussed it many times on this forum. I donāt need you to tell me to worry about low-probability, high-impact events, and to pretend that my post above is arguing that we should disregard those is uncharitable and discourteous. I donāt you need to explain the concept of expected value. Thatās absurd.
I started writing about existential risk from AI in 2015, based on Nick Bostromās Superintelligence book and other sources. Bostrom, of course, helped popularize the argument that very low probabilities of existential catastrophe have very high expected value. As mentioned in the post, Iāve been writing about AGI for a long time, and Iām clearly familiar with the basic arguments. Itās frustrating that your response was: well, wait, have you ever considered that worrying about low-probability events might be important? Yeah, of course I have. How could anyone familiar with this topic not have considered that?
The majority of people involved in EA currently seem to believe that thereās a 50%+ chance of AGI by 2035 ā with many setting their median year at or before 2032 ā and if that probability is 3-8 orders of magnitude too high, or something like that, because people in EA accept bad arguments and evidence without a reasonable minimum of scientific skepticism or critical appraisal, then thatās worthy of discussion. Thatās the main thing this post is about. Coming to wrong conclusions typically has negative consequences, as does having bad intellectual practices. Those negative consequences may include a failure to guard against existential risk wisely and well (e.g., it could mean the wrong kind of AI safety work is getting funded, rather than AI safety work getting funded too much overall). There may be other important negative consequences as well, such as reputational harm to EA, alienation of longstanding community members (like me), misallocation of funding away from places where itās direly needed (such as non-existentially dangerous pandemics or global poverty or animal welfare), or serious physical or psychological harm to individuals due to a few people holding apocalyptic or millennialist views ā just to name a few. Why not respond to that argument about the EA communityās intellectual practices, which is what is emphasized again and again and again in the post, rather than something that isnāt in the text?
This misunderstanding was a pretty avoidable mistake on your part ā which is okay, mistakes are forgivable ā and it would have been easy to simply acknowledge your mistake after I pointed it out. Why canāt you extend that basic courtesy?
I donāt think you should complain about David Thorstad or anyone else allegedly misrepresenting your arguments when youāre not upholding the principle here that you asked Thorstad to respect. You could have engaged in a more generous disagreement and maybe we could have had a constructive discussion. I was polite and patient the first time I pointed out you got my view wrong, more confrontational the second time, and now Iām pointing it out again in a state of complete expasteration. You had multiple chances to come to the table. What gives?