Pronouns: she/āher or they/āthem.
I got interested in effective altruism back before it was called effective altruism, back before Giving What We Can had a website. Later on, I got involved in my university EA group and helped run it for a few years. Now Iām trying to figure out where effective altruism can fit into my life these days and what it means to me.
Yarrow Bouchard šø
Karma: 1,353
Is AI risk classified as a longtermist cause? If so, why?
It seems like a lot people in EA think that AI risk is a relevant concern within the next 10 years, let alone the next 100 years. My impression is most of the people who think so believe that the near term is enough to justify worrying about AI risk, and that you donāt need to invoke people who wonāt be born for another 100 years to make the case.
Welcome to the EA Forum!
You know what, I donāt mean to discourage you from your project. Go for it.
I think the EA communityās belief in the EA communityās amazing āepistemicsā ā āepistemicsā is a made-up word, by the way, which is ironic, the very word that people in the EA community use to talk about the communityās knowledge and critical thinking skills is itself highly questionable ā is part of why people accept dubious or ridiculous arguments and evidence for imminent AGI. I think a lot of people believe the community wouldnāt accept arguments or evidence that have clear holes, so if anyone points out what they see as clear holes in the arguments/āevidence, theyāre most likely just wrong.
A lot of people believe and want to continue believing that the EA community is exceptional in some big ways. Examples of this Iāve heard include the belief that people in the EA community understand important methodological issues in social science better than social scientists and caught onto these issues sooner, and generally have better thinking skills than academics or experts, and are more likely to be right than academics or experts when the community disagrees with them.
I imagine people who think like this must be resistant to arguments along the lines of āthe EA community has been making all kinds of ridiculous, obvious errors around this topic and if you just begin to scratch the surface, you start digging up more and more things that just donāt make a lick of senseā. On one hand, people may be less receptive to messages that are blunt and confronting like that. On the other hand, I donāt like participating in a dynamic where people nod politely or tiptoe around things when the situation is this dire.
The vast majority of people in EA seem stable and like theyāre doing okay psychologically, but I catch some glimpses of people who seem to be losing touch with reality in a concerning way. So, the harm is not just philanthropists making some bad calls and wasting a lot of money that could have gone to help the global poor (or do something else more useful), itās also that it seems like some people are getting harmed by these ideas in a more direct way.
The LessWrong community is a complete mess in that regard ā there are the scary cults, people experiencing psychosis, a lot of paranoia and conspiratorial thinking (toward each other, toward EA, toward Silicon Valley, toward the government, toward liberalism, science, and journalism), and a lot of despair. One post that stuck out to me on LessWrong was someone who expressed their sense of hopelessness because, as they saw it, even if all the safety and alignment problems around AGI could be solved, that would still be bad because it makes the world a weird and unsettling place, where humansā role would be unclear. The view of Eliezer Yudkowsky and other transhumanists going back to the 1990s ā before Yudkowsky started worrying about the alignment stuff ā was that inventing AGI would be the best thing that ever happened, and Yudkowsky wrote about how it gave him hope despite all the suffering and injustice in the world. Yudkowsky is depressed (ostensibly) because he thinks thereās a 99.5% chance AGI would cause human extinction. Itās worrying and sad to see people also feel despair about the AGI scenario playing out in the way that Yudkowsky was hopeful about all those years ago.
The EA community has done a much better job than LessWrong at staying grounded and stable, but I still see signs that a few people here and there are depressed, panicked, hopeless, vengeful toward those they see as enemies or ādefectorsā, and sometimes come across as detached from reality in an eerie, unsettling way. Itās horrible to see the human cost of bad ideas that make no sense. Probably the people who are worst affected have other psychological risk factors (that typically seems to be the case in these sorts of situations), but that doesnāt mean the ideas donāt make things worse.You make a good point that practically everyone has probabilistic getouts. If you assign a probability to AGI by 2033 (or whatever) of anywhere from 10% to 90%, if 2034 rolls around and thereās still no AGI, you can plausibly say, retrospectively, you still think you assigned the right probability to AGI. (Especially if your probability is more like 60% than 90%.)
This sort of thing makes perfect sense with something rigorous and empirical like FiveThirtyEightās election forecast models. The difference is that FiveThirtyEight can do a post-mortem and scrutinize the model and the polls, and check things like how much the polls missed the actual vote margins. FiveThirtyEight can open source their model code, list the polls they use as inputs, publicly describe their methodology, and invite outside scrutiny. Thatās where FiveThirtyEightās credibility comes from. (Or came from ā sadly, itās no longer.)In the case of AGI forecasts, there are so few opportunities to test the forecasting āmodelā (i.e. a personās gut intuition). One of the few pieces of external confirmation/ādisconfirmation that could mean something, i.e. whether AGI happens by the year predicted or not, is easily brushed aside. So, itās not that the probabilistic getout is inherently illegitimate, itās that these views are so unempirical in the first place, and this move conveniently avoids one of the few ways these views could be empirically tested.
The reason I think the AI bubble popping should surprise people (enough to hopefully motivate them to revisit their overall views on a deep level) is that the AI bubble popping seems incompatible with the story many people in EA are telling about AI capabilities. Itās hard to square the hype over AI capabilities with the reality that there are hardly profitable applications of generative AI (profitable for the end customer), it doesnāt seem to help much with workersā productivity in most cases (coding might be an important exception, although still less so than e.g. the hype around the METR time horizons would suggest), and that not many people find chatbots useful enough to pay for a premium subscription. It seems hard to square that reality with AGI by 2033. Of course, they can always just kick the can down the road to AGI by 2038 or whatever. But if the bubble pops in 2026 or 2027 or 2028, I donāt see how people could keep thinking 2033 is the right year for AGI and not push this back some.
I agree that most people who think thereās a realistic chance of AGI killing us all before 2035 will probably just feel jubilant and relieved if an AI bubble pops. Thatās a bit worrying to me too, since re-examining their views on a deep level would mean letting go of that good feeling. (Or ā I just thought of this ā maybe they might like the taste of not worrying about dying, and would invite the deeper reflection. I donāt know. I think itās hard to predict how people will think or feel about something like this.)
I donāt know if I believe that. It seems like thereās a whole lot of random guessing going on among the most senior people in EA.
You might be better served to look into general psychology/āsocial science research and advice into how to make difficult personal decisions under conditions of high irreducible uncertainty. There are a few tricks I like. One is the regret minimization trick: which path would you regret least, even if it didnāt work out? Another trick is the coin flip or random number generator trick: randomly pick an option as if the coin flip (or other random event) is going to settle the decision for you. Notice whether you feel good or bad about that.
I think we all wish there was some kind of scientific formula we could use to make career decisions and so on. But I think thereās an inevitable mix of things like intuition, gut feeling, traditional wisdom, advice from people you know and trust, and so on.
I was responding to this part:
But letās imagine that I am a newly minted hundred millionaire. If there were Qualy, a well trusted LLM that would link to pages in the EA corpus and answer questions, I might chat to it a bit?
I agree chatbots are not to be trusted to do research or analysis, but I was imagining someone using a chatbot as a search engine, to just get a list of charities that they could then read about.
I think 15 different lists or rankings or aggregations would be fine too.
If a millionaire asked me right now for a big list of EA-related charities, I would give them the donation election list. And if they wanted to know the EA communityās ranking, I guess I would show them the results of the donation election. (Although I think those are hidden for the moment and weāre waiting on an announcement post.)
Maybe a bunch of people should write their own personal rankings.
Someone should make a Tier List template for EA charities. Something like this:
It seems like many senior EAs know an advanced internal reasoning process (for example, how to estimate when earning to give is actually worth it)in their brain that I havenāt yet learned. This worries me, because without these tools my independent career reasoning may be systematically inaccurate, and I would need frequent corrections from more EA people. However, I currently donāt know many EAs, and the ones I do know are often very busy.
No, these problems are fundamentally unsolvable, and everybody is just as confused and as uncertain as you. There is simply no way to know or predict these kinds of things. All you can do is make your best guess.
If someone wants to use a chatbot to research EA-endorsed charities they might be interested in donating to, they can just describe to ChatGPT or Claude what they want to know and ask the bot to search the EA Forum and other EA-related websites for info.
What percentage probability would you assign to your ability to accurately forecast this particular question?
Iām not sure why youāre interested in getting me to forecast this. I havenāt ever made any forecasts about AI systemsā ability to do math research. I havenāt made any statements about AI systemsā current math capabilities. I havenāt said that evidence of AI systemsā ability to do math research would affect how I think about AGI. So, whatās the relevance? Does it have a deeper significance, or is it just a random tangent?
If there is a connection to the broader topic of AGI or AI capabilities, I already gave a bunch of examples of evidence I would consider to be relevant and that would change my mind. Math wasnāt one of them. I would be happy to think of more examples as well.
I think a potentially good counterexample to your argument about FrontierMath ā original math research is natural language processing ā replacing human translators. Surely you would agree that LLMs have mastered the basic building blocks of translation? So, 2-3 years after GPT-4, why is demand for human translators still growing? One analysis claims that growth is counterfactually less that it would have been without the increase in the usage of machine translation, but demand is still growing.
I think this points to the difficulty in making these sorts of predictions. If back in 2015, someone had described to you the capabilities and benchmark performance of GPT-4 in 2023, as well as the rate of scaling of new models and progress on benchmarks, would you have thought that demand for human translators would continue to grow for at least the next 2-3 years?
I donāt have any particular point other than what seems intuitively obvious in the realm of AI capabilities forecasting may in fact be false, and I am skeptical of hazy extrapolations.The most famous example of a failed prediction of this sort is Geoffrey Hintonās prediction in 2016 that radiologistsā jobs would be fully automated by 2021. Almost ten years after this prediction, the number of radiologists is still growing and radiologistsā salaries are growing. AI tools that assist in interpreting radiology scans exist, but evidence is mixed on whether they actually help or hinder radiologists (and possibly harm patients).
You might learn useful stuff about current models from research on current models, but not necessarily anything useful about AGI (except maybe in the slightest, most indirect way). For example, I donāt know if anyone thinks if we had invested 100x or 1,000x more into research on symbolic AI systems 30 years ago, that we would know meaningfully more about AGI today. So, as you anticipated, the relevance of this research to AGI depends on an assumption about the similar between a hypothetical future AGI and current models.
However, even if you think AGI will be similar to current models, or it might be similar, there might be no cost to delaying research related to alignment, safety, control, preparedness, value lock-in, governance, and so on until more fundamental research progress on capabilities has been made. If in five or ten or fifteen years or whatever we understand much better how AGI will be built, then a single $1 million grant to a few researchers might produce more useful knowledge about alignment, safety, etc. than Dustin Moskovitzās entire net worth would produce today if it were spend on research into the same topics.My argument about ādoing basic scienceā vs. āmitigating existential riskā is that these collapse into the same thing unless you make very specific assumptions about which theory of AGI is correct. I donāt think those assumptions are justifiable.
Put it this way: letās say we are concerned that, for reasons due to fundamental physics, the universe might spontaneously end. But we also suspect that, if this is true, there may be something we can do to prevent it. What we want to know is a) if the universe is in danger in the first place, b) if so, how soon, and c) if so, what we can do about it.
To know any of these three things, (a), (b), or (c), we need to know which fundamental theory of physics is correct, and what the fundamental physical properties of our universe are. Problem is, there are half a dozen competing versions of string theory, and within those versions, the number of possible variations that could describe our universe is astronomically large, 10^500, or 10^272,000, or possibly even infinite. We donāt know which variation correctly describes our universe.
Plus, a lot of physicists say string theory is a poorly conceived theory in the first place. Some offer competing theories. Some say we just donāt know yet. Thereās no consensus. Everybody disagrees.
What does the āexistential riskā framing get us? What action does it recommend? How does the precautionary principle apply? Letās say you have a $10 billion budget. How do you spend it to mitigate existential risk?
I donāt see how this doesnāt just loop all the way back around to basic science. Whether thereās an existential risk, and if so, when we need to worry about it, and if when the time comes, what we can do about it, are all things we can only know if we figure out the basic science. How do we figure out the basic science? By doing the basic science. So, your $10 billion budget will just go to funding basic science, the same physics research that is getting funded anyway.
The space of possible theories about how the mind works is at least six, plus a lot of people saying we just donāt know yet, and there are probably silly but illustrative ways to formulate it where you get very large numbers.
For instance, if we think the correct theory can be summed up in just 100 bits of information, then the number of possible theories is 10,000.
Or we could imagine what would happen if we paid a very large number of experts from various relevant fields (e.g. philosophy, cognitive science, AI) a lot of money to spend a year coming up with a one-to-two-page description of as many original, distinct, even somewhat plausible or credible theories as they could think of. Then we group together all the submissions that were similar enough and counted them as the same theory. How many distinct theories would we end up with? A handful? Dozens? Hundreds? Thousands?
Iām aware these thought experiments are ridiculous, but Iām trying to emphasize the point that the space of possible ideas seems very large. At the frontier of knowledge in a domain like the science of the mind, which largely exists in a pre-scientific or protoscientific or pre-paradigmatic state, trying to actually map out the space of theories that might possibly be correct is a daunting task. Doing that well, to a meaningful extent, ultimately amounts to actually doing the science or advancing the frontier of knowledge yourself.
What is the right way to apply the precautionary principle in this situation? I would say the precautionary principle isnāt the right way to think about it. We would like to be precautionary, but we donāt know enough to know how to be. Weāre in a situation of fundamental, wide-open uncertainty, at the frontier of knowledge, in a largely pre-scientific state of understanding about the nature of the mind and intelligence. So, we donāt know how to reduce risk ā for example, our ideas on how to reduce risk might do nothing or they might increase risk.
If we donāt know which of infinite or astronomically many possible theories about AGI are more likely to be correct than the others, how can we prepare?
Maybe alignment techniques conceived based on our current wrong theory make otherwise benevolent and safe AGIs murderous and evil on the correct theory. Or maybe theyāre just inapplicable. Who knows?
What I just said: AI systems acting like a toddler or a cat would make me think AGI might be developed soon.
Iām not sure FrontierMath is any more meaningful than any other benchmark, including those on which LLMs have already gotten high scores. But I donāt know.
I donāt really know all the specifics of all the different projects and grants, but my general impression is that very little (if any) of the current funding makes sense or can be justified if the goal is to do something useful about AGI (as opposed to, say, make sure Claude doesnāt give risky medical advice). Absent concerns about AGI, I donāt know if Coefficient Giving would be funding any of this stuff.
To make it a bit concrete, there at least five different proposed pathways to AGI, and I imagine the research Coefficient Giving is only relevant to one of the five pathways, if itās even relevant to that one. But the number five is arbitrary here. The actual decision-relevant number might be a hundred, or a thousand, or a million, or infinity. It just doesnāt feel meaningful or practical to try to map out the full space of possible theories of how the mind works and apply the precautionary principle against the whole possibility space. Why not just do science instead?
By word count, I think Iāve written significantly more about object-level technical issues relevant to AGI than directly about AGI forecasts or my subjective guesses of timelines or probabilities. The object-level technical issues are what Iāve tried to emphasize. Unfortunately, commenters seem fixated on surveys, forecasts, and bets, and donāt seem to be as interested in the object-level technical topics. I keep trying to steer the conversation in a technical direction. But people keep wanting to steer it back toward forecasting, subjective guesses, and bets.
For example, I wrote a 2,000-word post called āUnsolved research problems on the road to AGIā. There are two top-level comments. The one with the most karma proposes a bet.
My post āFrozen skills arenāt general intelligenceā mainly focuses on object-level technical issues, including some of the research problems discussed in the other post. You have the top comment on that post (besides SummaryBot) and your comment is about a forecasting survey.
People on the EA Forum are apparently just really into surveys, bets, and forecasts.
I am not breaking new ground by saying it would be far more interesting to see an AI system behave like a playful, curious toddler or a playful, curious cat than a mathematician. That would be a sign of fundamental, paradigm-shifting capabilities improvement and would make me think maybe AGI is coming soon.
I agree that IQ tests were designed for humans, not machines, and thatās a reason to think itās a poor test for machines, but what about all the other tests that were designed for machines? GPT-4 scored quite high on a number of LLM benchmarks in March 2023. Has enough time passed that we can say LLM benchmark performance doesnāt meaningfully translate into real world capabilities? Or do we have to reserve judgment for some number of years still?
If your argument is that math as a domain is uniquely well-suited to the talents of LLMs, that could be true. I donāt know. Maybe LLMs will become an amazing AI tool for math, similar to AlphaFold for protein structure prediction. That would certainly be interesting, and would be exciting progress for AI.
I would say this argument is highly irreducibly uncertain and approaches the level of uncertainty of something like guessing whether the fundamental structure of physical reality matches the fundamental mathematical structure of string theory. Iām not sure itās meaningful to assign probabilities to that.
It also doesnāt seem like it would be particularly consequential outside of mathematics, or outside of things that mathematical research directly affects. If benchmark performance in other domains doesnāt generalize to research, but benchmark performance in math does generalize to math research, well, then, that affects math research and only math research. Which is really interesting, but would be a breakthrough akin to AlphaFold ā consequential for one domain and not others.
You said that my argument against accepting FrontierMath performance as evidence for AIs soon being able to perform original math research is overly general, such that a similar argument could be used against any evidence of progress. But what you said about that is overly general and similar reasoning could be used against any argument about not accepting a certain piece of evidence about current AI capabilities to support a certain conclusion about AI capabilities forecasting.
I suppose looking at the general contours of arguments from 30,000 feet in the air rather than their specifics and worrying āwhat ifā is not particularly useful.
Strong upvoted. Thank you for clarifying your views. Thatās helpful. We might be getting somewhere.
With regard to AI 2027, I get the impression that a lot of people in EA and in the wider world were not initially aware that AI 2027 was an exercise in judgmental forecasting. The AI 2027 authors did not sufficiently foreground this in the presentation of their āresultsā. I would guess there are still a lot of people in EA and outside it who think AI 2027 is something more rigorous, empirical, quantitative, and/āor scientific than a judgmental forecasting exercise.
I think this was a case of some people in EA being fooled or tricked (even if that was not the authorsā intention). They didnāt evaluate the evidence they were looking at properly. You were quick to agree with my characterization of AI 2027 as a forecast based on subjective intuitions. However, in one previous instance on the EA Forum, I also cited nostalgebraistās eloquent post and made essentially the same argument I just made, and someone strongly disagreed. So, I think people are just getting fooled, thinking that evidence exists that really doesnāt.
What does the forecasting literature say about long-term technology forecasting? Iāve only looked into it a little bit, but generally technology forecasting seems really inaccurate, and the questions forecasters/āexperts are being asked in those studies seem way easier than forecasting something like AGI. So, Iām not sure there is a credible scientific basis for the idea of AGI forecasting.
I have been saying from the beginning and Iāll say once again that my forecast of the probability and timeline of AGI is just a subjective guess and thereās a high level of irreducible uncertainty here. I wish that people would stop talking so much about forecasting and their subjective guesses. This eats up an inordinate portion of the conversation, despite its low epistemic value and credibility. For months, I have been trying to steer the conversation away from forecasting toward object-level technical issues.
Initially, I didnāt want to give any probability, timeline, or forecast, but I realized the only way to be part of the conversation in EA is to āplay the gameā and say a number. I had hoped that would only be the beginning of the conversation, not the entire focus of the conversation forever.
You canāt squeeze Bayesian blood from a stone of uncertainty. You canāt know what you canāt know by an act of sheer will. Most discussion of AGI forecasting is wasted effort. Most of it is mostly pointless.
What is not pointless is understanding the object-level technical issues better. If anything helps with AGI forecasting accuracy (and thatās a big āifā), this will. But it also has other important advantages, such as:
Helping us understand what risks AGI might or might not pose
Helping us understand what we might be able to do, if anything, to prepare for AGI, and what we would need to know to usefully prepare
Getting a better sense of what kinds of technical or scientific research might be promising to fund in order to advance fundamental AI capabilities
Understanding the economic impact of generative AI
Possibly helping to inform a better picture of how the human mind works
And more topics besides these.
I would consider it a worthy contribution to the discourse to play some small part in raising the overall knowledge level of people in EA about the object-level technical issues relevant to the AI frontier and to AGI. Based on track records, technology forecasting may be mostly forlorn, but, based on track records, science certainly isnāt forlorn. Focusing on the science of AI rather than on an Aristotelian approach would be a beautiful return to Enlightenment values, away from the anti-scientific/āanti-Enlightenment thinking that pervades much of this discourse.
By the way, in case itās not already clear, saying there is a high level of irreducible uncertainty does not support funding whatever AGI-related research program people in EA might currently feel inclined to fund. The number of possible ways the mind could work and the number of possible paths the future could take is large, perhaps astronomically large, perhaps infinite. To arbitrarily seize on one and say thatās the one, pour millions of dollars into that ā that is not justifiable.
I have no idea when AI systems will be able to do math research and generate original, creative ideas autonomously, but it will certainly be very interesting if/āwhen they do.
It seems like thereās not much of a connection between the FrontierMath benchmark and this, though. LLMs have been scoring well on question-and-answer benchmarks in multiple domains for years and havenāt produced any original, correct ideas yet, as far as Iām aware. So, why would this be different?
LLMs have been scoring above 100 on IQ tests for years and yet canāt do most of the things humans who score above 100 on IQ tests can do. If an LLM does well on math problems that are hard for mathematicians or math grad students or whatever, that doesnāt necessarily imply it will be able to do the other things, even within the domain of math, that mathematicians or math grad students do.
We have good evidence for this because LLMs as far back as GPT-4 nearly 3 years ago have done well on a bunch of written tests. Despite there being probably over 1 billion regular users of LLMs and trillions of queries put to LLMs, thereās no indication Iām aware of an LLM coming up with a novel, correct idea of any note in any academic or technical field. Is there a reason to think performance on the FrontierMath benchmark would be different than the trend weāve already seen with other benchmarks over the last few years?
The FrontierMath problems may indeed require creativity from humans to solve them, but that doesnāt necessarily mean solving them is a sign of creativity from LLMs. By analogy, playing grandmaster-level chess may require creativity from humans, but not from computers.
This is related to an old idea in AI called Moravecās paradox, which warns us not to assume what is hard for humans is hard for computers, or what is easy for humans is easy for computers.
a) Iām not sure all of those count as someone who would necessarily be an outsider to EA (e.g. Will MacAskill only assigns a 50% probability to consequentialism being correct, and he and others in EA have long emphasized pluralism about normative ethical theories; thereās been an EA system change group on Facebook since 2015 and discourse around systemic change has been happening in EA since before then)
b) Even if you do consider people in all those categories to be outsiders to EA or part of āthe out-groupā, us/āthem or in-group/āout-group thinking seems like a bad idea, possibly leading to insularity, incuriosity, and overconfidence in wrong views
c) Itās especially a bad idea to not only think in in-group/āout-group terms and seek to shut down perspectives of āthe out-groupā but also to cast suspicion on the in-group/āout-group status of anyone in an EA context who you happen to disagree with about something, even something minor ā that seems like a morally, subculturally, and epistemically bankrupt approach
Beautiful comment. I wholeheartedly agree that fun and friendship are not an extra or a nice-to-have, but are the lifeblood of communities and movements.
I understand the general concept of ingroup/āoutgroup, but what specifically does that mean in this context?
My much belated reply! On why I think short-form social media like Twitter and TikTok are good money chasing after bad, i.e., the medium is so broken and ill-designed in these cases, I think the best option is to just quit these platforms and focus on long-form stuff like YouTube, podcasts, blogs/ānewsletters (e.g. Medium, Substack), or what-have-you.
The most eloquent critic of Twitter is Ezra Klein. An from a transcript of his podcast, an episode recorded in December 2022:
My life immediately improved after I quit Twitter in early 2021. In retrospect, I see Twitter as a harmful addiction. On the extremely rare few occasions where Iāve dipped into looking at Twitter since then, itās always made me feel really yucky and frazzled afterward. But I still feel why itās addictive.
The same overall critique can be applied to TikTok without many modifications. Serious discourse on TikTok suffers in the same ways as on Twitter, for the same reasons.
And any Twitter copycat, such as Bluesky, or TikTok copycat, such as Instagram Reels, has the same problems, since theyāve deliberately copied those platforms as closely as possible, including what makes them bad.