Independent researcher on SRM and GCR/XRisk and on pluralisms in existential risk studies
GideonF
Karma: 1,793
Some important questions for the EA Leadership
Beyond Simple Existential Risk: Survival in a Complex Interconnected World
I’d be pretty interested in you laying out in depth why you have basically decides to dismiss these very varied and large set of arguments.(Full disclosure: I don’t agree with all of them, but in general I think there pretty good) A self admitted EA leader posting a response poo-pooing a long thought out criticism with very little argumentation, and mostly criticising it on tangential ToC grounds (which you don’t think or want to succeed anyway?) seems like it could be construed to be pretty bad faith and problematic. I don’t normally reply like this, but I think your original replied has essentially tried to play the man and not the ball, and I would expect better from a self-identified ‘central EA’ (not saying this is some massive failing, and I’m sure I’ve done similar myself a few times)
We are fighting a shared battle (a call for a different approach to AI Strategy)
Thank you for doing this and congratulations!
I haven’t managed to read the full report yet unfortunately, but I have a few questions/criticisms already- sorry to move onto these so quickly, but nonetheless I do think its important. (I tried to write these in a more friendly way, but I keep on failing to do, so please don’t take the tone as too aggressive, I am really not intending it to be, it just keeps coming across that way ! Sorry (: ) :
There are no mentions of systemic or cascading risks in the report. Why is this?
You don’t seem to engage with much of the peer-reviewed literature already written on climate change and GCRs. For example: Beard et al 2021, Kemp et al 2022, Richards et al 2021. Don’t get me wrong, you might disagree or have strong arguments against these papers, but it seems to some degree like you have failed to engage with them
You don’t seem to engage with much of the more complex systems aspects of civilisation collapse/ existential risk theory. Why is this?
There are no mentions of existential vulnerabilities and exposures, and you seem to essentially buy into a broadly hazard based account. The subdivision into direct and indirect effects further seems to support this idea. In this way you seem to ignore complex risk analysis. Why is this?
You seem to broadly ignore the work that went on around “sexy vs unsexy risks” and “boring apocalypses” and the more expansive work done to diversify views of how X-Risks may come about. Why is this?
Thanks for the report, and I am sure I will have more questions the more I go through it. I guess my major concern with this sort of stuff is it is likely that this work will go down (unrelated to its quality, and I am not saying its bad) as a “canonical” work in EA, so I think you perhaps have a responsibility, even if you in the end reject some of this scholarship, to engage in a lot of this (peer reviewed) scholarship on GCRs and X-Risks that has occurred in the “third wave” research paradigm of Existential Risk Studies, and I am slightly concerned that you appear not to have engaged with this literature!
If leading AI labs (OpenAI/Deepmind) shut down, or took voluntary policies to slow down progress to AGI (limits on number of parameters on models etc), would Anthropic follow (I assume you would?) Do you see yourself in a race that you must win, such that if the other parties dropped out, you’d still want to achieve AGI, or are you only at the frontier to keep doing up to date research?
Not going to lie, I understand you’re emotional, but being referred to as “all of the worst folks” in EA, and your previous comment before you edited wich was exceptionally rude, is honestly one of the worst interactions I’ve had in this community.
I have volunteered hours of my time running seminar programmes, writing curricula, helping organise speaker events and Q&As. I have spent literally 100s of hours consuming EA content and trying to compile EA content for people tgo use. I have orientated my research around X-Risk, spent an entire summer at CERI, taken a year off from university to research X-Risk, spoken at EAGx Rotterdam, and tried to make sense of the world to reduce X-Risk. Sure, I’m not perfect, and critiques of my approach and my criticisms I am sure I am valid, and understand this is emotional and hard for you, and I apologise if my critiques have made it harder- I still tink its time to have such difficult conversations, but I do apologise if this is hard. To be called “one of the worst folks in EA” and being accused essentially of “not actually doing stuff” is really upsetting when I really am trying my best, so I do hope you will apologise
It seems that the successful opposition to previous technologies was indeed explicitly against that technology, and so I’m not sure the softening of the message you suggest is actually necessarily a good idea. @charlieh943 recent case study into GM crops highlighted some of this (https://forum.effectivealtruism.org/posts/6jxrzk99eEjsBxoMA/go-mobilize-lessons-from-gm-protests-for-pausing-ai—he suggests emphasising the injustice of the technology might be good); anti-SRM activists have been explictly against SRM (https://www.saamicouncil.net/news-archive/support-the-indigenous-voices-call-on-harvard-to-shut-down-the-scopex-project), anti-nuclear activists are explicitly against nuclear energy and many more. Essentially, I’m just unconvinced that ‘its bad politics’ is necessarily supported by case studies that are most relevant to AI.
Nonetheless, I think there are useful points here both about what concrete demands could look like, or who useful allies could be, and what more diversified tactics could look like. Certainly, a call for a morotorium is not necessarily the only thing that could be useful in pushing towards a pause. Also, I think you make a point that a ‘pause’ might not be the best message that people can rally behind, although I reject the opposition. I think, in a similar way to @charlieh943 that emphasising injustice may be one good message that can be rallied around. I also think a more general ‘this technology is dangerous and allowing companies to make it are dangerous’ may also be a useful rallying message, which I have argued for in the past https://forum.effectivealtruism.org/posts/Q4rg6vwbtPxXW6ECj/we-are-fighting-a-shared-battle-a-call-for-a-different
I think the point I have been trying to make in criticising the leadership is something like this:
EA has this slightly weird, very opaque and untransparent, sort of unaccountable leadership which we trust with a lot of power. We treat them, in many ways, like “philosopher-kings” of the community
We do this on the premise that these people are uniquely good at leading the community and uniquely good at seeing risks that might be posed to the EA enterprise, and if they were more accountable and public, this would reduce their ability to do this, so reduce the ability of EA to do good in the world
However, the FTX scandal has shown us that these EA leaders are just humans, with the same flaws and biases as the rest of us (as I think many of us suspected), and aren’t these superhuman philosopher kings
Thus, we probably shouldn’t centralise so much power and trust in such a small unaccountable opaque group of people.
Its not that these people aren’t worthy of compassion, or that I would have done better (I wouldn’t have done). Its that we defer dso much power to them on the assumption that they are so much better than us at this, and I just think they’re not.
What 80000 Hours gets wrong about solar geoengineering
Statement on Pluralism in Existential Risk Studies
RESILIENCER Workshop Report on Solar Radiation Modification Research and Existential Risk Released
I guess I’m a bit skeptical of this, given that Buck has said this to weeatquince “I would prefer an EA Forum without your critical writing on it, because I think your critical writing has similar problems to this post (for similar reasons to the comment Rohin made here), and I think that posts like this/yours are fairly unhelpful, distracting, and unpleasant. In my opinion, it is fair game for me to make truthful comments that cause people to feel less incentivized to write posts like this one (or yours) in future”.
Hi John
Given the degree to which you have highlighted how experts have commented and reviewed the piece, will you, for the sake of intellectual transparency, commit to publishing all this expert feedback like the IPCC does. I think this may really help.
I said this in a subcomment, and it (worryingly) got significantly downvoted. It is a worrying sign for a community if a call for intellectual transparency (which is a key norm in EA) is downvoted just because the writer (ie me) has been critical of the piece.
I have great respect for you as an academic and an EA, and I trust that you will agree that such intellectual transparency is a useful norm, and if possible commit to publishing the commentsand reviews that those who reviewed the publication sent! The worry in the above paragraph is certainly not directed at you, and I have all the confidence that you are and will remain committed to maintaining EA as an as transparent space as possible
All the best
Gideon
I obviously think we need more time to flesh out real cruxes but I think our differences are cruxes are probably a few fold:
I think I am considerably less confident than you in the capacity of the research we have done thus far to confidently suggest climate’s contribute to existential risk. To some degree, I think the sort of evidence your happier relying on to make negative claims (ie not a major contributor to existential risk) I am much less happy with doing, as I think they often (and maybe always will) fail to account for plausible major contributors to the complexity of a system. This is both an advantage of the simple approach as Toby lays out earlier, but I’m more skeptical at its usage to make negative rather than positive claims.
I think you are looking for much better thought out pathways to catastrophe than I think is appropriate. I see climate acting as something acting to promote serious instability in a large number of aspects of a complex system, which should give us serious reasons to worry. This probably means my priors on climate are higher than yours immediately, as I’m of the impression you don’t hold this “risk emerges from an inherently interconnected world” ontology. This is why I’ve often put our differences down to our ontology and how we view risk in the real world
Because of my ontology and epistemology, I think I’m happier to put more credence on things like past precedent (collapses trigger by climate change, mass extinctions etc.), and decently formulated theory (planetary boundaries for GCR (although I recognise their real inherent flaws!), the sort of stuff laid out in Avin et al 2018, whats laid out in Beard et al 2021 and Kemp et al 2022). I’m also happier to take on board a broader range of evidence, and look more at things like how risk spreads, vulnerabilities/exposures, feedbacks, responses (and the plausible negatives therin) etc, which I don’t find your report convincing deals with, partially because they are really hard to deal with and partially because, particularly for the heavy tails of warming and other factors, there is a very small amount of research as Kemp et al lays out. Correct me if I’m wrong, but you see the world as a bit more understandable than I do, so simpler, quantitative, more rational models are seen as more important to be able to make any positive epistemic claim, and so you would somewhat reject the sort of analysis that I’m citing.
I’m also exceptionally skeptical of your claim that if direct risks are lower than indirect risks are lower; although I would reject the use of that language full stop
I also think its important to note that I make these claims in (mostly) the context of X-Risk. I think in “normal” scenarios, I would fall much closer to you than to disagreeing with you on a lot of things. But I think I have both a different ontology of existential risk (emerging mostly out of complex systems, so more like whats laid out in Beard et al 2021 and Kemp et al 2022) and perhaps more importantly a more pessimistic epistemology. As (partially) laid out when I discuss Existential Risk, Creativity and Well Adapted Science in the talk, I think that with Existential Risk negative statements (this won’t do this) actually have a higher evidentiary burden than positive statements of a certain flavour (it is plausible that this could happen). Perhaps this is because my priors of existential risk from most things are pretty low (owing I think in part to my pessimistic epistemology) that it just does take much more evidence to cause me to update downwards than to be like “huh, this could be a contributor to risk actually!”
Does this answer our cruxes? I know this doesn’t go into object level aspects of your report, but I think this may do a better job at explaining why we disagree, even when I do think your analysis is top-notch, albeit with a methodology that I disagree with on existential risk.
I also think its important that you know that I’m still not quite sure if I’m using the right language to explain myself here, and that my answer here is why I find your analysis unconvincing, rather than it being wrong. Perhaps as my views evolve I will look back and think differently. Anyway, I really would like to talk to you more about this at some point in the future.
Does this sound right to you?
Hi John, thanks for the comment, I’ve DM’d you about it. I think it may be easier if we did the discussion in person before putting something out on the forum, as there is probably quite a lot to unpack, so let me know if you would be up for this?
Can you make your model of indirect risks accessable to the public? Its asking for access. Thanks a lot.
Also, why do you assume that “most of the risk of existential catastrophe stems from AI, biorisk and currently unforeseen technological risks.”? My impression from earlier in the chapter is that you are essentially drawing the idea you can essentially ignore other potential causes from the Precipice. Is this correct?
Moreover, this assumption only seems true if you assume an X-Risk will come as a single hazard. If it is, say, a cascading risk, cascading to civilisational collapse then extinction, then the idea these are the biggest risks should be questioned. Simultanously, if you view it as a multi-pulsed thing, say civilisational collapse from one hazard or a series of hazards or cascades, and then followed by whatever may (slowly) make us extinct- once civilisation is collapsed its easier for smaller hazards to kill us all, then once again the primacy of these hazards reduces. Only if you take a reductive view that sees extinction as primarily due to direct, single or near single, hazards that kill everyone or basically everyone, can this model be valid.
Of course, you do talk a little about multipulsed, subextinction risks followed by recovery being harder, but not in much detail. In particular, you claim that extreme climate change may make civilisational recovery from collapse much harder, but then don’t seem to deal in detail with this question, which may be considered to be highly important, particularly if we think civilisational collapse is considerably more likely than extinction. Moreover, you suggest that “there is some
chance of civilisational collapse due to nuclear war or engineered pandemics,” essentially suggesting other causes of civilisational collapse that are less direct, and therefore could be made more likely due to climate change, are negligable. This assumption should be stated and evidenced, and yet you seem to include no sources on this.Moreover, you state (uncited) that “the main indirect effect is Great power Conflict.” Whats your source for this claim, and why are you so certain of this that you are confident that you can discount other indirect effects? This feels like the assumptions once again should be supported;
If this is the case, then I might say that relying on the (in my opinion) rather reductive, hazard-centric, simple risk assessment model of Ord etc. is our crux of disagreement. This is why I would say from my (still moderately limited unfortunatly) reading of the report, it appears that most of your facts are in order, however a lot of what I think that it is very bad that you fail to mention (systemic risk, cascading risk, vulnerabilities, exposures, complex risk assessments (I don’t use this to suggest my way is inherently intellectually superior than yours, as indeed it is a plausible position to hold that X-Risks may emerge out of epistemically simplier more direct more “simple” risks) etc) originates out of this hazard-centric approach. I won’t overthrow a paradigm in a single comment, and I won’t even try, but do please tell me if you agree with me that this is the crux of our disagreement. Moreover, whilst in a previous comment to me you have said you have argued for this methodology of viewing X-Risks at length in the piece, I am yet to find such an argument. If you could point me to where you think you make this argument in the piece, I will reread that section, or I may have missed it (apologies if I have). If not, it feels this approach needs considerably greater justification.
I have more comments/criticisms which I will post in other comments, but certainly on this indirect risk things, these are my questions.
Hi John,
Sorry to revisit this, and I understand if you don’t. I must apologies if my previous comments felt a bit defensive from my side, as I do feel your statements towards me were untrue, but I think I have more clarity on the perspective you’ve come from and some of the possible baggage brought to this conversation, and I’m truly sorry if I’ve be ignorant of relevant context.
I think this comment is more going to address the overall conversation between us two on here, and where I perceive it to have gone, although I may be wrong, and I am open to corrections.
Firstly, I think you have assumed this statement is essentially a product of CSER, perhaps because it has come from me, who was a visit at CSER, and has been similarly critical of your work in a way that I know some at CSER have. [I should say, for the record on this, I do think your work is of high quality, and I hope you’ve never got the impression that I don’t. Perhaps some of my criticisms last year towards the review process your report went through felt poor quality (and I can’t remember what they were and may not stand by them today), but if so, I am sorry.] Nonetheless, I think its really important to keep in mind that this statement is absolutely not a ‘CSER’ statement; I’d like to remind you of the signatories, and whilst every signatory doesn’t agree with everything, I hope you can see why I got so defensive when you claimed that the signatories weren’t being transparent and actually attempting to just make EA another left-wing movement. I tried really hard to get a plurality of voices in this document, which is why such an accusation offended me, but ultimately I shouldn’t have got defensive over this, and I must apologise.
Secondly, on that point, I think we may have been talking about different things when you said ‘heterodox CSER approaches to EA.’ Certainly, I think Ehrlich and much of what he has called for is deeply morally reprehensible, and the capacity for ideas like his to gain ground is a genuine danger of pluralistic xrisk, because it is harder to police which ideas are acceptable or not (similarly, I have recieved criticism because this letter fails to call out eugenics explicitly, another danger). Nonetheless, I think we can trust as a more pluralistic community develops it would better navigate where the bounds of acceptable or unacceptable views and behaviours are, and that this would be better than us simply suggesting this now. Maybe this is a crux we/the signatories and much of the commens section disagree on. I think we can push for more pluralism and diversity in response to our situation whilst trusting that the more pluralistic ERS community will police how far this can go. You disagree and think we need to lay this out now otherwise it will either a) end up with anything goes, including views we find moral reprehensible or b) will mean EA is hijaked by the left. I think the second argument is weaker, particularly because this statement is not about EA, but about building a broader field of Existential Risk Studies, although perhaps you see this as a bit of a trojan horse. I understand I am missing some of the historical context that makes you think it is, but I hope that the signatories list may be enough to show you that I really do mean what I say when I call for pluralism.
I also must apologise if the call for retraction of certain parts of your comment seemed uncollegiate or disrespectful to you; this was certainly not my intention. I, however, felt that your painting of my views was incorrect, and thought you may, in light of this, be happy to change; although given you are not happy to retract, I assume you are either trying to make the argument that these are in fact my underlying beliefs (or that I am being dishonest, although I have no reason to suspect you would say this!).
I think there are a few more substantive points we disagree on, but to me this seems like the crux of the more heated discussion, and I must apologise it got so heated
One thing I think this comment ignores is just how many of the suggestions are cultural, and thus do need broad communal buy in, which I assume is why they sent this publicly. Whilst they are busy, I’d be pretty disappointed if the core EAs didn’t read this and take the ideas seriously (ive tried tagging dome on twitter), and if you’re correct that presenting such a detailed set of ideas on the forum is not enough to get core EAs to take the ideas seriously I’d be concerned about where there was places for people to get their ideas taken seriously. I’m lucky, I can walk into Trajan house and knock on peoples doors, but others presumably aren’t so lucky, and you would hope that a forum post that generated a lot of discussion would be taken seriously. Moreover, if you are concerned with the ideas presented here not getting fair hearing, maybe you could try raising salient ideas to core EAs in your social circles?