In the most recent episode of the 80,000 Hours podcast, Rob Wiblin and Ajeya Cotra from Open Phil discuss āthe challenge Open Phil faces striking a balance between taking big ideas seriously, and not going all in on philosophical arguments that may turn out to be barking up the wrong tree entirely.
āThey also discuss:
Which worldviews Open Phil finds most plausible, and how it balances them
Which worldviews Ajeya doesnāt embrace but almost does
How hard it is to get to other solar systems
The famous āsimulation argumentā
When transformative AI might actually arrive
The biggest challenges involved in working on big research reports
What itās like working at Open Phil
And much moreā
Iām creating this thread so that anyone who wants to share their thoughts on any of the topics covered in this episode can do so. This is in the spirit of MichaelAās suggestion of posting all EA-relevant content here.
Thanks for making this linkpost, Evelyn! I did have some thoughts on this episode, which Iāll split into separate comments so itās easier to keep discussion organised. (A basic point is that the episode was really interesting, and Iād recommend others listen as well.)
A bundle of connected quibbles:
Ajeya seems to use the term āexistential riskā when meaning just āextinction riskā
She seems to imply totalism is necessary for longtermism
She seems to imply longtermism is only/ānecessarily focused on existential risk reduction
(And I disagree with those things.)
An illustrative quote from Ajeya:
But āexistential risksā includes not just extinction risk but also includes risks of unrecoverable collapse, unrecoverable dystopia, and some (but not all) s-risks/āsuffering catastrophes. (See here.)
And my understanding is that, if we condition on rejecting the totalism:
Risk of extinction does becomes way less important
Though itād still matter due to its effects on the present generation
Risk of unrecoverable collapse probably becomes way less important (though this is a bit less clear)
Risk of unrecoverable dystopia and s-risks still retain much of their importance
(See here for some discussion relevant to those points.)
So one can reasonably be a non-totalist yet still prioritise reducing existential riskāespecially risk of unrecoverable dystopias.
Relatedly, a fair number of longtermists are suffering-focused and/āor prioritise s-risk reduction, sometimes precisely because they reject the idea that making more happy beings is good but do think making more suffering beings is bad.
Finally, one can be a longtermist without prioritising reducing either reduction of extinction risk or reducing of other existential risks. In particular, one could prioritise work on what Iām inclined to call ānon-existential trajectory changesā. From a prior post of mine:
(Relatedly, my impression from a couple videos or podcasts is that Will MacAskill is currently interested in thinking more about a broad set of trajectory changes longtermists could try to cause/āprevent, including but not limited to existential catastrophes.)
I expect Ajeya knows all these things. And I think itās reasonable for a person to think that extinction risks are far more important than other existential risks, that the strongest argument for longtermism rests on totalism, and that longtermists should only/āalmost only prioritise existential/āextinction risk reduction. (My own views are probably more moderate versions of those stances.) But it seems to me that itās valuable to not imply that those things are necessarily true or true by definition.
(Though itās of course easy to state things in ways that are less than perfectly accurate or nuanced when speaking in an interview rather than producing edited, written content. And I did find a lot of the rest of that section of the interview quite interesting and useful.)
Somewhat relatedly, Ajeya seems to sort-of imply that āthe animal-inclusive worldviewā is necessarily neartermist, and that āthe longtermist worldviewā is necessarily human-centric. For example, the above quote about longtermism focuses on āpeopleā, which I think would typically be interpreted as just meaning humans, and as very likely excluding at least some beings that might be moral patients (e.g., insects). And later she says:
But I think the questions of neartermism vs longtermism and animal-inclusivity vs human-centrism are actually fairly distinct. Indeed, I consider myself an animal-inclusive longtermist.
I do think itās reasonable to be a human-centric longtermist. And I do tentatively think that even animal-inclusive longtermism should still prioritise existential risks, and still with extinction risks as a/āthe main focus within that.
But I think animal-inclusivity makes at least some difference (e.g., pushing a bit in favour of prioritising reducing risks of unrecoverable dystopias). And it might make a larger difference. And in any case, it seems worth avoiding implying that all longtermists must be focused only or primarily on benefitting humans, since that isnāt accurate.
(But as with my above comment, I expect that Ajeya knows these things, and that the fact she was speaking rather than producing edited written content is relevant here.)
I havenāt finished listening to the podcast episode yet but I picked up on a few of these inaccuracies and was disappointed to hear them. As you say I would be surprised if Ajeya isnāt aware of these things. Anyone who has read Greaves and MacAskillās paper The Case for Strong Longtermism should know that longtermism doesnāt necessarily mean a focus on reducing x-risk, and that it is at least plausible that longtermism is not conditional on a total utilitarianism population axiology*.
However, given that many people listening to the show might not have read that paper, I feel these inaccuracies are important and might mislead people. If longtermism is robust to different views (or at least if this is plausible), then it is very important for EAs to be aware of this. I think that it is important for EAs to be aware of anything that might be important in deciding between cause areas, given the potentially vast differences in value between them.
*Even the importance of reducing extinction risk isnāt conditional on total utilitarianism. For example, it could be vastly important under average utilitarianism if we expect the future to be good, conditional on humans not going extinct. That said, Iām not sure how many people take average utilitarianism seriously.
Update: I sort-of adapted this comment into a question for Ajeyaās AMA, and her answer clarifies her views. (It seems like her and I do in fact basically agree on all of these points.)
Thank you for writing this critique, it was a thought I had while listening as well. In my experience many EAs make the same mistake, not just Ajeya.
Is eventually expanding beyond our solar system necessary for achieving a long period with very low extinction risk?
As part of the discussion of āEffective size of the long-term futureā, Ajeya and Rob discussed the barriers to and likelihood of various forms of space colonisation. I found this quite interesting.
During that section, I got the impression that Ajeya was implicitly thinking that a stable, low-extinction-risk future would require some kind of expansion beyond our solar system. (Though I donāt think she said that explicitly, so maybe Iām making a faulty inference. Perhaps what she actually had in mind was just that such expansion could be one way to get a stable, low-extinction-risk future, such that the likelihood of such expansion was one important question in determining whether we can get such a future, and a good question to start with.)
If she does indeed think that, that seems a bit surprising to me. I havenāt really thought about this before, but I think Iād guess that we could have a stable, low-extinction-risk futureāfor, letās says, hundreds of millions of yearsāwithout expanding beyond our solar system. Such expansion could of course help[1], both because it creates ābackupsā and because there are certain astronomical extinction events that would by default happen eventually to Earth/āour solar system. But it seems to me plausible that the right kind of improved technologies and institutions would allow us to reduce extinction risks to negligible levels just on Earth for hundreds of millions of years.
But Iāve never really directly thought about this question before, so I could definitely be wrong. If anyone happens to have thoughts on this, Iād be interested to hear them.
[1] Iām not saying itād definitely helpāthere are ways it could be net negative. And Iām definitely not saying that trying to advance expansion beyond our solar system is an efficient way to reduce extinction risk.
In my view, the AI timelines work is exactly the kind of research longtermists should do more of to persuade those more skeptical, and test their own assumptions.
Another is Christian Tarsneyās āThe Epistemic Challenge to Longtermismā.
What Iād like to see next are grounded estimates of the causal effects of longtermist interventions, like research and policy, and including risks of backfire (e.g. accelerating AI development).
The sections āBiggest challenges with writing big reportsā and āWhat itās like working at Open Philā were interesting and relatable
A lot of what was said in these sections aligned quite a bit with my own experiences from researching/āwriting about EA topics, both as part of EA orgs and independently.
For example, Ajeya said:
I think most of the EA-related things Iāve started looking into and writing up, except those that I deprioritised very early on, ending up growing and spawning spinoff tangent docs/āposts. And then those spinoffs often ended up spawning their own spinoffs, and so on. And I think this was usually actually productive, and sometimes the spinoffs were more valuable than the original thing, but it definitely meant a lot of missed deadlines, changed plans, and uncertainties about when to just declare something finished and move on.
I donāt have a lot of experience with research/āwriting on non EA-related topics, so maybe this is just a matter of my own (perhaps flawed) approach, or maybe itās just fairly normal. (One thing that comes to mind here is thatāif I recall correctlyāJoe Henrich says in his newest book, The WEIRDest People in the World, that his previous bookāSecret of Our Successāwas all basically just meant to be introductory chapters to WEIRDest People. And the prior book is itself quite long and quite fascinating!)
But I did do ~0.5FTE years of academic psychology research during my Honours year. There I came up with the question and basic design before even starting, and the final product really had stuck pretty closely to that, and on schedule, with no tangents. So thereās at least weak evidence that my more recent tangent-heavy approach (which I think I actually endorse) isnāt just an approach Iād adopt even in more established fields.
A few other things Ajeya said in those sections that resonated with me:
It was interestingāand sort of nice, in a weird way! - to hear that even someone with a relatively senior role at one of the most prominent and well-resourced EA orgs has those experiences and perceptions.
(To be clear, Iāve overall been very happy with the EA-related roles Iāve worked in! Ajeya also talked about a bunch of stuff about her job thatās really positive and that also resonated with me.)
One other part of those sections that feels worth highlighting:
I sort-of effectively followed similar advice, and have been very happy with the apparent results for my own career. And I definitely agree that there are a remarkable number of open questions (e.g., here and here) which it seems like a variety of people could just independently have a crack at, thereby testing their fit and/āor directly providing useful insights.
The doomsday argument, the self-sampling assumption (SSA), and the self-indication assumption (SIA)
The interview contained an interesting discussion of those ideas. I was surprised to find that, during that discussion, I felt like I actually understood what the ideas of SSA and SIA were, and why that mattered. (Whereas thereāve been a few previous times when I tried to learn about those things, but always ended up mostly still feeling confused. That said, itās very possible I currently just have an illusion of understanding.)
While listening, I felt like maybe that section of the interview could be summarised as follows (though note that I may be misunderstanding things, such that this summary might be misleading):
āWe seem to exist āearlyā in the sequence of possible humans. Weāre more likely to observe that if the sequence of possible humans will actually be cut off relatively early than if more of the sequence will occur. This should update us towards thinking the sequence will be cut off relatively earlyāi.e., towards thinking there will be relatively few future generations. This is how the SSA leads to the doomsday argument.
But, we also just seem to exist at all. And weāre more likely to observe that (rather than observing nothing at all) the more people will exist in totalāi.e., the more of the sequence of possible humans will occur. This should update us towards thinking the sequence wonāt be cut off relatively early. This is how the SIA pushes against the doomsday argument.
Those two updates might roughly cancel out [Iām not actually sure if theyāre meant to exactly, roughly, or only very roughly cancel out]. Thus, these very abstract considerations have relatively little bearing on how large we should estimate the future will be.ā
(Iād be interested in peopleās thoughts on whether my attempted summary seems accurate, as well as on whether it seems relatively clear and easy to follow.)
One other thing on this section of the interview: Ajeya and Rob both say that the way the SSA leads to the doomsday argument seems sort-of āsuspiciousā. Ajeya then says that, on the other hand, the way the SIA causes an opposing update also seems suspicious.
But I think all of her illustrations of how updates based on the SIA can seem suspicious involved infinities. And we already know that loads of things involving infinities can seem counterintuitive or suspicious. So it seems to me like this isnāt much reason to feel that SIA in particular can cause suspicious updates. In other words, it seems like maybe the āactive ingredientā causing the suspiciousness in the examples she gives is infinity, not SIA. Whereas the way the SSA leads to the doomsday argument doesnāt have to involve infinity, so there it seems like SSA is itself suspicious.
Iām not sure whether this is a valid or important point, but maybe it is? (I obviously donāt think we should necessarily dismiss things just because they feel āsuspiciousā, but it could make sense to update a bit away from them for that reason, and, to the extent that thatās true, a difference in the suspiciousness of SSA vs SIA could matter.)
I like Ajeyaās metaphor of different āsectsā of the EA movement being different stops on the ātrain to crazy townā:
The idea of neartermism and longtermism reflecting economist-like and philosopher-like ways of thinking struck a chord with me. I feel as if there are conflicting parts of me that want to follow different approaches to saving the world.
Iām not finished yet with the whole episode, but I didnāt understand the part about fairness agreements and the veil of ignorance that Rob and Ajeya were talking about as a way to figure out how much money to allocate per worldview. This was the part from 00:27:50 to 00:41:05. I think I understood the outlier opportunities principle though.
Iāve re-read the transcript once to try and understand it more but I still donāt. I also googled about the Veil of Ignorance, and it started to make more sense, but I still donāt understand the fairness agreements part. Is there a different article that explains what Ajeya meant by that? Or can someone explain it in a better way? Thanks!
(It turns out I was slightly mistaken in my other commentāthere actually are a few public written paragraphs on the idea of fairness agreements in one section of a post by Holden Karnofsky in 2018.)
A separate comment I had been writing about that section of the interview:
Ajeya and Rob discussed āFairness agreementsā. This seemed to me like a novel and interesting approach that could be used for normative/āmoral uncertainty (though Open Phil seem to be using it for worldview uncertainty, which is related but a bit different)
I currently feel more inclined towards some other approaches to moral uncertainty
But at this stage where the topic of moral uncertainty has received so little attention, it seems useful to come up with additional potential approaches
And it may be that, for a while, it remains useful to have multiple approaches one can bring to bear on the same question, to see where their results converge and diverge
On a meta level, I found it interesting that the staff of an organisation primarily focused on grantmaking appear to have come up with what might be a novel and interesting approach to normative/āmoral uncertainty
That seems like the sort of abstract theoretical philosophy work that one might expect to only be produced by academic philosophers, rather than people at a more āappliedā org
A more direct response to your comment:
I havenāt heard of the idea before, and had read a decent amount on moral uncertainty around the start of 2020. That, plus the way the topic was introduced in this episode, makes me think that this might be a new idea that hasnāt been publicly written up yet.
(See also the final bullet point here)
[Update: I was slightly mistaken; there are a few written paragraphs on the idea here]
I think itās understandable to have been a bit confused by that part; I donāt think I fully understood the idea myself, and I got the impression that it was still at a somewhat fuzzy stage
(Iād guess that with an hour of effort I could re-read that part of the transcript and write an ok explainer, but unfortunately I donāt have time right now. But hopefully someone else will be able to do that, ideally better and more easily than I could!)
No worries that you donāt have the time to explain it Michael! Iām glad to hear that others havenāt heard of the idea before and that this is a new topic. Hopefully someone else can explain it in more depth. I think sometimes concepts featured in 80K podcast episodes or other EA content can be really hard to grasp, and maybe others can create visuals, videos, or better explanations to help.
An example of another hard to grasp topic in 80Kās past episodes is complex cluelessness. I think Hilary Greaves and Arden did a good/āokay job in explaining it, and I kinda get the idea, but it would be hard for me to explain without looking up the paper, reading the transcript, or listening to the podcast again.
I also still find the concept of complex cluelessness slippery, and am under the impression that many EAs misunderstand and misuse the term compared to Greavesā intention. But if you havenāt seen it already, you may find this talk from Greavesā helpful.
I really enjoyed this podcast. But regarding space colonization, I do not think that uploading humans to computers is the only alternative to avoid transporting human colonies in space ships. For instance, we could send facilities for producing nutrition, oxygen and human artificial wombs there, plus two tiny test tubes of undamaged egg and sperm cells. Of course, once synthetic biology gives us the ability to create cells ourselves, we can also upload the human (epi)genome to a storage medium and synthetize the DNA and zygotes on the new planet.
Maybe we can do that eventually, but doesnāt that just seem unnecessarily hard?
I would have thought that this is magnitudes easier, because (with exception of my last sentence) this uses existing technology (although, AFAIK the artificial ecosystems we tried to create on earth failed after some time, so maybe there is a bit more fine-tuning needed). Whereas we still seem to be far away to understand humans or upload them to computers. But in the end, perhaps we would not want to colonise space with a rocket like structure, but with the lightest stuff we can possibly built do to relativistic mass increase. Who knows. The lightweight argument would certainly work in favour of the upload to computer solution.