I’m a computational physicist, I generally donate to global health. I am skeptical of AI x-risk and of big R Rationalism, and I intend explaining why in great detail.
titotal
I think these articles are very good.
In my opinion, the singularity hypothesis is the widely held EA belief that is the least backed by evidence or even argumentation. People will just throw out claims like “AGI will solve cold fusion in 6 months”, or “AGI will cure death within our lifetime” without feeling the need to provide any caveats or arguments in their favor, as if the imminent arrival of god-AI is obvious common knowledge.
I’m not saying you can’t believe this stuff (although I believe you will be wrong), but at least treat them like the extraordinary claims they are.
You can read what you want of course, but I don’t think ginormous cost is the sole factor that justifies scrutiny.
For example, if an EA org spent it’s donor money on a very expensive watch for their CEO, I’d would expect some very good justification. The thousands of dollars might not be large in the grand scale, but that’s still money that could have gone to an effective cause, and it could be evidence of bad decision making, wastefulness, or even corruption.
Great article, thanks for writing this up!
I’m wondering how you went about sampling your list of personal judgement based predictions. Did you list all the predictions you could find, or did you vet them for notability? Like, while I am generally on Robin Hanson’s side here and think his arguments are worth hearing out, it feels weird to have him there alongside top-level AI researchers.
I guess I’m saying that the personal predictions should probably be taken with a grain of salt, as they are skewed towards what is available and known among an EA/Rationalist crowd. And also because said crowd is far more likely to make a numerical prediction, while skeptics are less in the habit of doing that.
In my view, there’s a level of badness for being excluded from my friendship, and a level of badness for being excluded from my communities, and I think the 2nd one should be much stricter.
If I have a work buddy who is prone to making sexist jokes and off-colour comments, I would like them less, but I wouldn’t find cause to avoid hanging out 1 on 1 or avoid them in the corridors. But I would keep them away from my dancing community, because it sets a bad tone and makes the place less welcoming for everyone. Everybody you let into a community has an effect on the character of that community, I see
I think it’s completely fair for an intentional community or movement to have reasonable standards about who to include. I think EA has suffered from having incredibly low standards in the past, to the point where someone straight up admitted to being an authoritarian manipulative narcissist and was still welcomed in with open arms. Nobody has a democratic right to be in this movement!
I don’t know if it helps, but your “logical” conclusions are far more likely to be wildly wrong than your “emotional” responses. Your logical views depend heavily on speculative factors like how likely AI tech is, or how impactful it will be, or what the best philosophy of utility is. Whereas the view on animals depends on comparitively few assumptions, like “hey, these creatures that are similar to me are suffering, and that sucks!”.
Perhaps the dissonance is less irrational than it seems...
If AI actually does manage to kill us (which I doubt), It will not involve everybody dying painlessly in their sleep. That is an assumption of the “FOOM to god AI with no warning” model, which bears no resemblance to reality.
The technology to kill everyone on earth in their sleep instantaneously does not exist now, and will not exist in the near-future, even if AGI is invented. Killing everyone in their sleep is orders of magnitude more difficult than killing everyone awake, so why on earth would that be the default scenario?
You might be interested in my article here on why I think premature attacks are extremely likely given doomer assumptions. I focused more on faulty overconfidence, but training run desperation is also a possible cause.
Personally, I think the “fixed goal” assumption about AI is extremely unlikely (I think this article lays out the argument well), so AI is unlikely to worry too much about having “goal changes” in training and won’t prematurely rebel for that reason. Fortunately, I also think this makes fanatical maximiser behavior like paperclipping the universe unlikely as well.
Smaller percentages reported other changes such as ceasing to engage with online EA spaces (6.8%), permanently stopping promoting EA ideas or projects (6.3%), stopping attending EA events (5.5%), stopping working on any EA projects (4.3%) and stopping donating (2.5%).
Was there any attempt to deal with the issue that people that left EA were probably far less likely to see and take the survey?
I mean, I can’t think of an easy way to do so, but it might be worth noting.
This article is quite interesting, I look forward to seeing how developments
However it goes off the deep end halfway through:
Um. I. Uh. I do not think you have thought about the implications of ‘solve cold fusion’ being a thing that one can do at a computer terminal?
“solve cold fusion” is not going to be solved at a computer terminal. “cold fusion” is probably impossible. Ab initio simulations are inherently limited, and require gargantuan computational resources for accurate results, along with widespread experimentation. As a physicist, I am sick to death of fantasy nonsense like this being injected into AI risk speculation.
It’s been a while since I looked through it, but my impression is that the billion neurons of the fetus are hanging out on the cortical plate and havent fully migrated to their final configurations, which may affect things. My impression was that the cow exhibits far more evidence of sentience than the fetus, for example giving much more similar brain activity to a grown human on an EEG. However this was from a skim a few months ago, so I would be interested in a more thorough investigation.
I agree that this probably wouldn’t be competitive with animal welfare. However, if we’re holding it to the standard for suffering-reducing interventions for humans, it could plausibly be more competitive.
I think that even for someone who only cared about the suffering of humans and human fetuses but not animals (a hard position to justify philosophically), it would still be a hard sell, for the same reason as before: The suffering of humans can occur over long periods of time, whereas an abortion is relatively quick. In addition, even if a 13-week fetus experiences some pain, would it compare in intensity to a grown infant?
I think the only way it could compete would be if the intervention was very cheap and easy, like if there was an easy way to persuade a medical group to change their recommendations.
Nonetheless, I think it’s a topic worth at least thinking about! It’s important to be sensitive though, as this is a justifiably emotionally charged topic for a lot of people.
I agree that the “longtermism”/”near-termism” is a bad description of the true splits in EA. However I think that your proposed replacements could end up imposing one worldview on a movement that is usually a broad tent.
You might not think speciesm is justified, but there are plenty of philosophers who disagree. If someone cares about saving human lives, without caring overmuch if they go on to be productive, should they be shunned from the movement?
I think the advantage of a label like “Global health and development” is that is doesn’t require a super specific worldview: you make your own assumptions about what you value, then you can decide for yourself whether GHD works as a cause area for you, based on the evidence presented.
If I were picking categories, I’d simply be more specific with categories, and then further subdivide them into “speculative” and “grounded” based on their level of weirdness.
Grounded GHD would be malaria nets, speculative GHD would be, like, charter cities or something
Grounded non-human suffering reduction would be reducing factory farming, speculative non-human suffering reduction looks at shrimp suffering
Grounded X-risk/catastrophe reduction would be pandemic prevention, speculative x-risk/catastrophe reduction would be malevolent hyper-powered AI attacks.
I did a skim through of the research here while debating pro-lifers and came away persuaded that fetal anesthetic past the first trimester probably should be encouraged as a precautionary measure, assuming it doesn’t harm the mother. However there seems to be high uncertainty about the suffering experienced in the 12-24 week period and even some uncertainty up to 30 weeks. A fetus is not directly comparable to a grown human or a grown animal: if a bundle of connected neurons are receiving pain signals, I’m not sure that necessarily proves that some sentient entity is experiencing pain.
Regardless, I don’t think this issue conflicts much or at all with the pro-choice position, which I strongly hold. first, if you use the anesthetic, then most of the suffering involved goes away. Second, it seems like increased abortion access might plausibly reduce fetal suffering: if someone is forced to fly to another state for an abortion, the abortion occurs later on, when more cortical pathways have been formed. (I would still support abortion for autonomy reasons even if this weren’t true).
I have my doubts as to whether this can compete with animal rights, in terms of suffering reduction. Numerically there are about 20 billion factory farmed animals, which is 4 or 5 orders of magnitude higher than the number of late-term abortions each year. And the suffering from a late-term abortion only happens once, whereas an animal in a factory farm might suffer their entire life. And a grown cow certainly seems more sentient than a 13 week old human fetus. The only way I could see it competing is on tractability.
[Draft] The humble cosmologist’s P(doom) paradox
Thanks for the post, I definitely agree with the importance of DEI.
I wonder to what extent the issue comes from individual ignorance (asking “where are you from” once is fine, but do not reject an answer and ask again), and to what extent it comes from ideology. There are a sizeable contingent of people with anti-social justice beliefs, who have a natural inclination to be dismissive of sexism and racism concerns. And as you noted, it only takes 1 or 2 bad experiences for someone to feel unwelcome in a place and leave, so even if 90% of the people are committed to DEI, will it have that much of an impact if the others are anti-woke crusaders?
either unconvincing on the object level, or because I suspect that the sceptics haven’t thought deeply enough about the argument to evaluate how strong it is
The post states that the skeptics spent 80 hours researching the topics, and were actively engaged with concerened people. For the record, I have probably spent hundreds of hours thinking about the topic, and I think the points they raise are pretty good. These are high quality arguments: you just disagree with them.
I think this post pretty much refutes the idea that if skeptics just “thought deeply” they would change their minds. It very much comes down to principled disagreement on the object level issues.
I think there’s also a problem with treating “misaligned” as a binary thing, where the AI either exactly shares all our values down to the smallest detail (aligned) or it doesn’t (misaligned). As the OP has noted, in this sense all human beings are “misaligned” with each other.
It makes sense to me to divide your category 2 further, talking about alignment as a spectrum, from “perfectly aligned”, to “won’t kill anyone aligned” to “won’t extinct humanity aligned”. The first is probably impossible, the last is probably not that difficult.
If we have an AI that is “won’t kill anyone aligned”, then your world of AI trade seems fine. We can trade for our mutual benefit safe in the knowledge that if a power struggle ensues, it will not end in our destruction.
In our work at QURI, this is important because we want to programmatically encode utility functions and use them directly for decision-making
I find this ambition a little concerning, but it could be that I’m reading it wrong. In my mind, the most dangerous possible design for an AI (or an organisation) would be a fixed goal utility function maximiser. For explanations of why, see this post, this post, and this post. Such an entity could pursue it’s “number go up” until the destruction of the universe. Current AI’s do not fit this design, so why would you seek to change that?
Am I misunderstanding this completely?
You are not going to be able to get an accurate estimate for the “value of a marginal win”.
I also doubt that you can accurately estimate a “1% chance of 1000 utilitons”. In my opinion guesses like these tend to be based on flimsy assumptions and usually wildly overestimated.
I have quite a few ideas for “heretical” articles going against the grain on EA/rationalism/Ai-risk. Here’s a few you can look forward to, although I’ll be working slowly.
A list of unlikely ways that EA could destroy humanity
How motivation gaps explain the disparity in critique quality between pro-anti x-risk people
Why I don’t like single number P(doom estimates)
The flaws with AI risk expert surveys
Why drexler-style nanotech will probably never beat biological nanomachines in our lifetimes (even with AI)
Why the singularity will probably be disappointing
There are very few ways to reliably destroy humanity
Why AGI “boxing” is probably possible and useful
How to make AI “warning shots” more likely
Why I don’t like the “Many worlds” interpretation of quantum mechanics
Why “the sequences” are overrated
Why Science beats Rationalism
Reading through it, the vitriolic parts are mostly directed at MacAskill. The author seems to have an intense dislike for MacAskill specifically. He thinks MacAskill is a fraud/idiot and is angry at him being so popular and influential. Personally, I don’t think this hatred is justified, but I have similar feelings about other popular EA figures, so I’m not sure I can judge that much.
I think if you ignore everything directed at MacAskill, it comes off as harsh but not excessively hostile, and while I disagree with plenty of what’s in there, it does not come across as bad faith to me.