The recent anthology Essays on Longtermism, which is open access and free to read here, has several essays with good criticisms of longtermism. You might find some of those essays interesting. The authors included in that anthology are a mix of proponents of longtermism and critics of longtermism.
This is not necessarily to disagree with any of your specific arguments or your conclusion, but I think for people who have not been extremely immersed in effective altruist discourse for years, what has been happening with effective altruism over the last 5-10 years can easily be mis-diagnosed.
In the last 5-10 years, has EA shifted significantly toward prioritizing very long-term outcomes (i.e. outcomes more than 1,000 years in the future) over relatively near-term outcomes (i.e. outcomes within the next 100 years)? My impression is no, not really.
Instead, what has happened is that a large number of people in EA have come to believe that thereās more than a 50% chance of artificial general intelligence being created within the next 20 years, with many thinking thereās more than a 50% chance of it being created within 10 years. If AGI is created, many people in EA believe there is a significant risk of human extinction (or another really, really bad outcome). āSignificant riskā could mean anywhere from 10% to over 50%. People vary on that.
This is not really about the very long-term future. Itās actually about the near-term future: what happens within the next 10-20 years. Itās not a pivot from the near-term to the very long-term, itās a pivot from global poverty and factory farming to near-term AGI. So, itās not really about longtermism at all.
The people who are concerned about existential risk from near-term AGI donāt think itās only a justified worry if you account for lives in the distant future. They think itās a justified worry if you only account for people who already alive right now. The shift in opinion is not anything to do with arguments about longtermism, but about people thinking AGI is much more likely much sooner than they previously did, and also them accepting arguments that AGI would be incredibly dangerous if created.
The pivot in EA over the last 5-10 years has also not, in my observation, been a pivot from global poverty and factory farming to existential risk in general, but a pivot to only specifically existential risk from near-term AGI.
To put my cards on the table, my own personal view is:
Longtermism is, in principle, correct (i.e. all else being equal, future lives matter as much as present lives or past lives and, if we can, we should try to ensure there are a lot of good future lives), but even after years of discussion, it seems really hard for anyone to give a good example of what actions longtermism justifies that we wouldnāt have already been doing anyway ā other than actions related to existential risk, the discussion of which predates the discussion of longtermism by many years.
Whereas many people in EA seem to think the probability of AGI being created within the next 7 years is 50% or more, I think that probability is significantly less than 0.1%.
The arguments that AGI would be extremely dangerous rely on assumptions about the underlying technologies used to create AGI. I donāt think we know yet what those technologies will be, so I donāt accept these arguments. Iām not necessarily saying I know for sure AGI wonāt be dangerous, either, Iām just saying knowing for sure would require information we donāt have.
I miss when EA was much more focused on global poverty and factory farming. (So, I agree with you there.)
Some existential risks are probably neglected, but not existential risk from near-term AGI. I think the world most likely still underinvests in global catastrophic risks of the humans vs. nature variety: asteroids, large volcanos, and natural pandemics. These risks are easier to calculate rigorous probabilities for.
The x-risks you discussed in your post are humans vs. humans risks: nuclear war, bioweapons, and the humans creating AGI. These are far more complex. Asteroids donāt respond to our space telescopes by attempting to disguise themselves to evade detection. But with anything to do with humans, humans will always respond to what we do, and that response is always at least somewhat unpredictable.
I still think we should do things to reduce the risk from nuclear war and bioweapons. Iām just saying that these risks are more complex and uncertain than risks from nature. So, itās more harder to do the cost-effectiveness math that shows spending to reduce these risks is justified. However, so much in the world canāt be rigorously analyzed with that kind of math, so thatās not necessarily an argument against it!
As for climate change, I agree itās important, and maybe some people in EA have done some good work in this area ā I donāt really know ā but it seems like thereās already so much focus on it from so many people, many of whom are extremely competent, itās hard to see what EA would contribute by focusing on it. By contrast, global poverty charity effectiveness wasnāt a topic many people outside of international development thought about ā or at least felt they could do anything about ā before GiveWell and effective altruism. Moreover, there wasnāt any social movement advocating for people to donate 10% of their income to help the global poor.
Whereas many people in EA seem to think the probability of AGI being created within the next 7 years is 50% or more, I think that probability is significantly less than 0.1%.
In principle, yes, but in a typical bet structure, there is no upside for the person taking the other side of that bet, so what would be the point of it for them? I would gladly accept a bet where someone has to pay me an amount of money on January 1, 2033 if AGI isnāt created by then (and vice versa), but why would they accept that bet? Thereās only downside for them.
Sometimes these bets are structured as loans. As in, I would loan someone money and they would promise to pay me that money back plus a premium after 7 years. But I donāt want to give a stranger from another country a 7-year loan that I wouldnāt be able to compel them to repay once the time is up. From my point of view, that would just be me giving a cash gift to a stranger for no particularly good reason.
There is Long Bets, which is a nice site, but since everything goes to charity, itās largely symbolic. (Also, the money is paid up by both sides in advance, and the Long Now Foundation just holds onto it until the bet is resolved. So, itās a little bit wasteful in that respect. The money is tied up for the duration of the bet and there is a time value of money.)
Right, Iād forgotten that betting on this is hard. I was thinking if one could do a sort of cross-over between an end-of-the-world bet crossed with betting a specific on a proportion of oneās net worth. This is the most fleshed-out proposal Iāve seen so far.
But I donāt want to give a stranger from another country a 7-year loan that I wouldnāt be able to compel them to repay once the time is up.
I wonder if this could be solved via a trusted third person who knows both bettors. (I think there are possible solutions here via blockchains, e.g. the ability to unilaterally destroy an escrow, but I guess thatās going to become quite complicated, not worth the setup, and using a technology I guess youāre skeptical of anyway)
Are you referring to the length of tasks that LLMs are able to complete with a 50% success rate? I donāt see that as a meaningful indicator of AGI. Indeed, I would say itās practically meaningless. It truly just doesnāt make sense an indicator of progress toward AGI. I find it strange that anyone thinks otherwise. Why should we see that metric as indicating AGI progress anymore than, say, the length of LLMsā context windows?
I think a much more meaningful indicator from METR would be the rate at which AI coding assistants speeds up coding tasks for human coders. Currently, METRās finding is that it slows them down by 19%. But this is asymmetric. Failing to clear a low bar like being an unambiguously useful coding assistant in such tests is strong evidence against models nearing human-level capabilities, but clearing a low bar is not strong evidence for models nearing human-level capabilities. By analogy, we might take an AI system being bad at chess as evidence that it has much less than human-level general intelligence. But we shouldnāt take an AI system (such as Deep Blue or AlphaZero) being really good at chess as evidence that it has human-level or greater general intelligence.
If I wanted to settle for an indirect proxy for progress toward AGI, I could short companies like Nvidia, Microsoft, Google, or Meta (e.g. see my recent question about this), but, of course, those companies stock pricesā donāt directly measure AGI progress. Conversely, someone who wanted to take the other side of the bet could take a long position in those stocks. But then this isnāt much of an improvement on the above. If LLMs became much more useful coding assistants, then this could help justify these companiesā stock prices, but it wouldnāt say much about progress toward AGI. Likewise for other repetitive, text-heavy tasks, like customer support via web chat.
It seems like the flip side should be different: if you do think AGI is very likely to be created within 7 years, shouldnāt that imply a long position in stocks like Nvidia, Microsoft, Google, or Meta would be lucrative? In principle, you could believe that LLMs are some number of years away from being able to make a lot of money and at most 7 years away from progressing to AGI, and that the market will give up on LLMs making a lot of money just a few years too soon. But I would find this to be a strange and implausible view.
So, to be clear, you think that if LLMs continue to complete software engineering tasks of exponentially increasing lengths at exponentially decreasing risk of failure, then that tell us nothing about whether LLMs will reach AGI?
I expect most EAs who have enough money to consider investing them to already be investing them in index funds, which, by design, long the Magnificent Seven already.
Iām not sure if youāre asking about the METR graph on task length or about the practical use of AI coding assistants, which the METR study found is currently negative.
If I understand it correctly, the METR graph doesnāt measure an exponentially decreasing failure rate, just a 50% failure rate. (Thereās also a version of the graph with a 20% failure rate, but thatās not the one people typically cite.)
I also think automatically graded tasks used in benchmarks donāt usually deserve to be called āsoftware engineeringā or anything that implies that the actual tasks the LLM is doing are practically useful, economically valuable, or could actually substitute for tasks that humans get paid to do.
I think many of these LLM benchmarks are trying to measure such narrow things and such toy problems, which seem to be largely selected so as to make the benchmarks easier for LLMs, that they arenāt particularly meaningful.
In terms of studies of real world performance like METRās study on human coders using an AI coding assistant, thatās much more interesting and important. Although I find most LLM benchmarks practically meaningless for measuring AGI progress, I think practical performance in economically valuable contexts is much more meaningful.
My point in the above comment was just that an unambiguously useful AI coding assistant would not by itself be strong evidence for near-term AGI. AI systems mastering games like chess and go is impressive and interesting and probably tells us some information about AGI progress, but if someone pointed to AlphaGo beating Lee Seedol as strong evidence that AGI would have been created within 7 years of that point, they would have been wrong.
In other words, progress in AI probably tells us something about AGI progress, but just taking impressive results in AI and saying that implies AGI within 7 years isnāt correct, or at least itās unsupported. Why 7 years and not 17 years or 77 years or 177 years?
If you assume whatever rate of progress you like, that will support any timeline you like based on any evidence you like, but, in my opinion, thatās no way to make an argument.
On the topic of betting and investing, itās true that index funds have exposure to AI, and indeed personally I worry about how much exposure the S&P 500 has (global index funds that include small-cap stocks have less, but I donāt know how much less). My argument in the comment above is simply that if someone thought it was rational to bet some amount of money on AGI arriving within 7 years, then surely it would be rational to invest that same amount of money in a 100% concentrated investment in AI and not, say, the S&P 500.
The people who are concerned about existential risk from near-term AGI donāt think itās only a justified worry if you account for lives in the distant future. They think itās a justified worry if you only account for people who already alive right now.
The recent anthology Essays on Longtermism, which is open access and free to read here, has several essays with good criticisms of longtermism. You might find some of those essays interesting. The authors included in that anthology are a mix of proponents of longtermism and critics of longtermism.
This is not necessarily to disagree with any of your specific arguments or your conclusion, but I think for people who have not been extremely immersed in effective altruist discourse for years, what has been happening with effective altruism over the last 5-10 years can easily be mis-diagnosed.
In the last 5-10 years, has EA shifted significantly toward prioritizing very long-term outcomes (i.e. outcomes more than 1,000 years in the future) over relatively near-term outcomes (i.e. outcomes within the next 100 years)? My impression is no, not really.
Instead, what has happened is that a large number of people in EA have come to believe that thereās more than a 50% chance of artificial general intelligence being created within the next 20 years, with many thinking thereās more than a 50% chance of it being created within 10 years. If AGI is created, many people in EA believe there is a significant risk of human extinction (or another really, really bad outcome). āSignificant riskā could mean anywhere from 10% to over 50%. People vary on that.
This is not really about the very long-term future. Itās actually about the near-term future: what happens within the next 10-20 years. Itās not a pivot from the near-term to the very long-term, itās a pivot from global poverty and factory farming to near-term AGI. So, itās not really about longtermism at all.
The people who are concerned about existential risk from near-term AGI donāt think itās only a justified worry if you account for lives in the distant future. They think itās a justified worry if you only account for people who already alive right now. The shift in opinion is not anything to do with arguments about longtermism, but about people thinking AGI is much more likely much sooner than they previously did, and also them accepting arguments that AGI would be incredibly dangerous if created.
The pivot in EA over the last 5-10 years has also not, in my observation, been a pivot from global poverty and factory farming to existential risk in general, but a pivot to only specifically existential risk from near-term AGI.
To put my cards on the table, my own personal view is:
Longtermism is, in principle, correct (i.e. all else being equal, future lives matter as much as present lives or past lives and, if we can, we should try to ensure there are a lot of good future lives), but even after years of discussion, it seems really hard for anyone to give a good example of what actions longtermism justifies that we wouldnāt have already been doing anyway ā other than actions related to existential risk, the discussion of which predates the discussion of longtermism by many years.
Whereas many people in EA seem to think the probability of AGI being created within the next 7 years is 50% or more, I think that probability is significantly less than 0.1%.
The arguments that AGI would be extremely dangerous rely on assumptions about the underlying technologies used to create AGI. I donāt think we know yet what those technologies will be, so I donāt accept these arguments. Iām not necessarily saying I know for sure AGI wonāt be dangerous, either, Iām just saying knowing for sure would require information we donāt have.
I miss when EA was much more focused on global poverty and factory farming. (So, I agree with you there.)
Some existential risks are probably neglected, but not existential risk from near-term AGI. I think the world most likely still underinvests in global catastrophic risks of the humans vs. nature variety: asteroids, large volcanos, and natural pandemics. These risks are easier to calculate rigorous probabilities for.
The x-risks you discussed in your post are humans vs. humans risks: nuclear war, bioweapons, and the humans creating AGI. These are far more complex. Asteroids donāt respond to our space telescopes by attempting to disguise themselves to evade detection. But with anything to do with humans, humans will always respond to what we do, and that response is always at least somewhat unpredictable.
I still think we should do things to reduce the risk from nuclear war and bioweapons. Iām just saying that these risks are more complex and uncertain than risks from nature. So, itās more harder to do the cost-effectiveness math that shows spending to reduce these risks is justified. However, so much in the world canāt be rigorously analyzed with that kind of math, so thatās not necessarily an argument against it!
As for climate change, I agree itās important, and maybe some people in EA have done some good work in this area ā I donāt really know ā but it seems like thereās already so much focus on it from so many people, many of whom are extremely competent, itās hard to see what EA would contribute by focusing on it. By contrast, global poverty charity effectiveness wasnāt a topic many people outside of international development thought about ā or at least felt they could do anything about ā before GiveWell and effective altruism. Moreover, there wasnāt any social movement advocating for people to donate 10% of their income to help the global poor.
Are you willing to bet on this?
In principle, yes, but in a typical bet structure, there is no upside for the person taking the other side of that bet, so what would be the point of it for them? I would gladly accept a bet where someone has to pay me an amount of money on January 1, 2033 if AGI isnāt created by then (and vice versa), but why would they accept that bet? Thereās only downside for them.
Sometimes these bets are structured as loans. As in, I would loan someone money and they would promise to pay me that money back plus a premium after 7 years. But I donāt want to give a stranger from another country a 7-year loan that I wouldnāt be able to compel them to repay once the time is up. From my point of view, that would just be me giving a cash gift to a stranger for no particularly good reason.
There is Long Bets, which is a nice site, but since everything goes to charity, itās largely symbolic. (Also, the money is paid up by both sides in advance, and the Long Now Foundation just holds onto it until the bet is resolved. So, itās a little bit wasteful in that respect. The money is tied up for the duration of the bet and there is a time value of money.)
Right, Iād forgotten that betting on this is hard. I was thinking if one could do a sort of cross-over between an end-of-the-world bet crossed with betting a specific on a proportion of oneās net worth. This is the most fleshed-out proposal Iāve seen so far.
I wonder if this could be solved via a trusted third person who knows both bettors. (I think there are possible solutions here via blockchains, e.g. the ability to unilaterally destroy an escrow, but I guess thatās going to become quite complicated, not worth the setup, and using a technology I guess youāre skeptical of anyway)
You could bet on shorter-term indicators e.g. whether the METR trend will stop or accelerate.
Are you referring to the length of tasks that LLMs are able to complete with a 50% success rate? I donāt see that as a meaningful indicator of AGI. Indeed, I would say itās practically meaningless. It truly just doesnāt make sense an indicator of progress toward AGI. I find it strange that anyone thinks otherwise. Why should we see that metric as indicating AGI progress anymore than, say, the length of LLMsā context windows?
I think a much more meaningful indicator from METR would be the rate at which AI coding assistants speeds up coding tasks for human coders. Currently, METRās finding is that it slows them down by 19%. But this is asymmetric. Failing to clear a low bar like being an unambiguously useful coding assistant in such tests is strong evidence against models nearing human-level capabilities, but clearing a low bar is not strong evidence for models nearing human-level capabilities. By analogy, we might take an AI system being bad at chess as evidence that it has much less than human-level general intelligence. But we shouldnāt take an AI system (such as Deep Blue or AlphaZero) being really good at chess as evidence that it has human-level or greater general intelligence.
If I wanted to settle for an indirect proxy for progress toward AGI, I could short companies like Nvidia, Microsoft, Google, or Meta (e.g. see my recent question about this), but, of course, those companies stock pricesā donāt directly measure AGI progress. Conversely, someone who wanted to take the other side of the bet could take a long position in those stocks. But then this isnāt much of an improvement on the above. If LLMs became much more useful coding assistants, then this could help justify these companiesā stock prices, but it wouldnāt say much about progress toward AGI. Likewise for other repetitive, text-heavy tasks, like customer support via web chat.
It seems like the flip side should be different: if you do think AGI is very likely to be created within 7 years, shouldnāt that imply a long position in stocks like Nvidia, Microsoft, Google, or Meta would be lucrative? In principle, you could believe that LLMs are some number of years away from being able to make a lot of money and at most 7 years away from progressing to AGI, and that the market will give up on LLMs making a lot of money just a few years too soon. But I would find this to be a strange and implausible view.
So, to be clear, you think that if LLMs continue to complete software engineering tasks of exponentially increasing lengths at exponentially decreasing risk of failure, then that tell us nothing about whether LLMs will reach AGI?
I expect most EAs who have enough money to consider investing them to already be investing them in index funds, which, by design, long the Magnificent Seven already.
Iām not sure if youāre asking about the METR graph on task length or about the practical use of AI coding assistants, which the METR study found is currently negative.
If I understand it correctly, the METR graph doesnāt measure an exponentially decreasing failure rate, just a 50% failure rate. (Thereās also a version of the graph with a 20% failure rate, but thatās not the one people typically cite.)
I also think automatically graded tasks used in benchmarks donāt usually deserve to be called āsoftware engineeringā or anything that implies that the actual tasks the LLM is doing are practically useful, economically valuable, or could actually substitute for tasks that humans get paid to do.
I think many of these LLM benchmarks are trying to measure such narrow things and such toy problems, which seem to be largely selected so as to make the benchmarks easier for LLMs, that they arenāt particularly meaningful.
In terms of studies of real world performance like METRās study on human coders using an AI coding assistant, thatās much more interesting and important. Although I find most LLM benchmarks practically meaningless for measuring AGI progress, I think practical performance in economically valuable contexts is much more meaningful.
My point in the above comment was just that an unambiguously useful AI coding assistant would not by itself be strong evidence for near-term AGI. AI systems mastering games like chess and go is impressive and interesting and probably tells us some information about AGI progress, but if someone pointed to AlphaGo beating Lee Seedol as strong evidence that AGI would have been created within 7 years of that point, they would have been wrong.
In other words, progress in AI probably tells us something about AGI progress, but just taking impressive results in AI and saying that implies AGI within 7 years isnāt correct, or at least itās unsupported. Why 7 years and not 17 years or 77 years or 177 years?
If you assume whatever rate of progress you like, that will support any timeline you like based on any evidence you like, but, in my opinion, thatās no way to make an argument.
On the topic of betting and investing, itās true that index funds have exposure to AI, and indeed personally I worry about how much exposure the S&P 500 has (global index funds that include small-cap stocks have less, but I donāt know how much less). My argument in the comment above is simply that if someone thought it was rational to bet some amount of money on AGI arriving within 7 years, then surely it would be rational to invest that same amount of money in a 100% concentrated investment in AI and not, say, the S&P 500.
The argument AI safety work is more cost-effective than AMF when considering only the next few generations is pretty weak.
Doesnāt that still depend on how much risk you think there is, and how tractable you think interventions are?
I think itās still accurate to say that those concerned with near term AI risk think it is likely more cost effective than AMF.
This is, of course, sensitive to your assumptions.