Sarah Weiler

Karma: 474

Sarah Weiler 15 Mar 2024 6:33 UTC
14 points
1 ∶ 1
in reply to: Jeff Kaufman’s comment on: Critique of the notion that impact follows a power-law distribution
This is a good question. I think, if we assume everything else equal (neither got the money by causing harm, both were influenced by roughly the same number of actors to be able and willing to donate their money), then I think I agree that the altruistic impact of the first is 100x that of the second.

I am not entirely sure what that implies for my own thinking on the topic. On the face of it, it clearly contradicts the conclusion in my Empirical problem section. But it does so without, as far as I can tell, addressing the subpoints I mention in that section. Does that mean the subpoints are not relevant to the empirical claim I make? They seem relevant to me, and that seems clear in examples other than the one you presented. I’m confused, and I imagine I’ll need at least a few more days to figure out how the example you gave changes my thinking.

Update: I am currently working on a Dialogue post with JWS to discuss their responses to the essay above and my reflections since publishing it. I imagine/hope that this will help streamline my thinking on some of the issues raised in comments (as well as some of the uncertainties I had while writing the essay). For that reason, I’ll hold off on comment responses here and on updates to the original essay until work on the Dialogue post has progressed a bit further, hoping to come back to this in a few days (max 1-1.5 weeks?) with a clearer take on whether & how comments such as this one by Jeff shift my thinking. Thanks again to all critical (and supportive) commenters for kicking off these further reflections!
What links here?
- Sarah Weiler's comment on Critique of the notion that impact follows a power-law distribution by Sarah Weiler (25 Mar 2024 19:23 UTC; 4 points)

Sarah Weiler 14 Mar 2024 18:08 UTC
10 points
3 ∶ 2
in reply to: Brad West’s comment on: Critique of the notion that impact follows a power-law distribution
I think counterfactual analysis as a guide to making decisions is sometimes (!) a useful approach (especially if it is done with appropriate epistemic humility in light of the empirical difficulties).
But, tentatively, I don’t think that it is a valid method for calculating the impact an individual has had (or can be expected to have, if you calculate ex ante). I struggle a bit to put my thinking on this into words, but here’s an attempt: If I say “Alec [random individual] has saved 1,000 lives”, I think what I mean is “1,000 people now live because of Alec alone”. But if Alec was only able to save those lives with the help of twenty other people, and the 1,000 people would now be dead were it not for those twenty helpers, then it seems wrong to me to claim that the 1,000 survivors are alive only because of Alec—even if Alec played a vital role in the endeavour and if it would have been impossible to replace Alec by some other random individual. And just because any one of the twenty people were easily replaceable, I don’t think that they all suddenly count for nothing/very little in the impact evaluation; the fact seems to remain that Alec would not have been able to have any impact if he did not have twenty other people to help him… So it seems like an individual impact evaluation would need to include some sharing between Alec and the twenty other helpers; wouldn’t it??
Correct me if you (anyone reading this) think I’m misguided, but I believe the crux here is that I’m using a different definition of “impact” than the one that underlies counterfactual analysis. I agree that the impact definition underlying counterfactual analysis can sometimes be useful for making individual decisions, but I would argue that the definition I use can be helpful when talking about efforts to do good as a community and when trying to build a strategy for how to live one’s life over the long term (because it looks at what is needed for positive change in the aggregate).

Sarah Weiler 19 Nov 2022 10:11 UTC
10 points
0 ∶ 0
in reply to: AronM’s comment on: A case against focusing on tail-end nuclear war risks
Thanks for taking the time to read through the whole thing and leaving this well-considered comment! :)
In response to your points:
1) Opportunity costs
- “I do not know how a specialization would look like that is only relevant at the 100 to 1000 nukes step. I know me not being able to imagine such a specialization is only a weak argument but I am also not aware of anyone only looking at such a niche problem.”—If this is true and if people who express concern mainly/only for the worst kinds of nuclear war are actually keen on interventions that are equally relevant for preventing any deployment of nuclear weapons, then I agree that the opportunity cost argument is largely moot. I hope your impressions of the (EA) field in this regard are more accurate than mine!
- My main concern with preparedness interventions is that they may give us a false sense of ameliorating the danger of nuclear escalation (i.e., “we’ve done all these things to prepare for nuclear winter, so now the prospect of nuclear escalation is not quite as scary and unthinkable anymore”). So I guess I’m less concerned about these interventions the more they are framed as attempts to increase general global resilience, because that seems to de-emphasize the idea that they are effective means to substantially reduce the harms incurred by nuclear escalation. Overall, this is a point that I keep debating in my own mind and where I haven’t come to a very strong conclusion yet: There is a tension in my mind between the value of system slack (which is large, imo) and the possible moral hazard of preparing for an event that we should simply never allow to occur in the first place (i.e.: preparation might reduce the urgency and fervor with which we try to prevent the bad outcome in the first place).
- I mostly disagree on the point about skillsets: I think both intervention targets (focus on tail risks vs. preventing any nuclear deployment) are big enough to require input from people with very diverse skillsets, so I think it will be relatively rare for a person to be able to only meaningfully contribute to either of the two. In particular, I believe that both problems are in need of policy scholars, activists, and policymakers and a focus on the preparation side might lead people in those fields to focus less on the preventing any kind of nuclear deployment goal.
2) Neglectedness:
- I think you’re empirically right about the relative neglectedness of tail-ends & preparedness within the nuclear risk field.
- (I’d argue that this becomes less pronounced as you look at neglectedness not just as “number of people-hours” or “amount of money” dedicated to a problem, but also factor in how capable those people are and how effectively the money is spent (I believe that epistemically rigorous work on nuclear issues is severely neglected and I have the hope that EA engagement in the field could help ameliorate that).)
- That said, I must admit that the matter of neglectedness is a very small factor in convincing me of my stance on the prioritization question here. As explained in the post, I think that a focus on the tail risks and/or on preparedness is plausibly net negative because of the intractability of working on them and because of the plausible adverse consequences. In that sense, I am glad that those two are neglected and my post is a plea for keeping things that way.
3) High uncertainty around interventions: Similar thoughts to those expressed above. I have an unresolved tension in my mind when it comes to the value of preparedness interventions. I’m sympathetic to the case you’re making (heck, I even advocated (as a co-author) for general resilience interventions in a different post a few months ago); but, at the moment, I’m not exactly sure I know how to square that sympathy with the concerns I simultaneously have about preparedness rhetoric and action (at least in the nuclear risk field, where the danger of such rhetoric being misused seems particularly acute, given vested interests in maintaining the system and status-quo).
4) Civilizational Collapse:
- My claim about civilization collapse in the absence of the deployment of multiple nukes is based on the belief that civilizations can collapse for reasons other than weapons-induced physical destruction.
- Some half-baked, very fuzzy ideas of how this could happen are: destruction of communities’ social fabric and breakdown of governance regimes; economic damage, breakdown of trade and financial systems, and attendant social and political consequences; cyber warfare, and attendant social, economic, and political consequences.
- I have not spent much time trying to map out the pathways to civilizational collapse and it could be that such a scenario is much less conceivable than I currently imagine. I think I’m currently working on the heuristic that societies and societal functioning is hyper-complex and that I have little ability to actually imagine how big disruptions (like a nuclear conflict) would affect them, which is why I shouldn’t rule out the chance that such disruptions cascade into collapse (through chains of events that I cannot anticipate now).
- (While writing this response, I just found myself staring at the screen for a solid 5 minutes and wondering whether using this heuristic is bad reasoning or a sound approach on my part; I lean towards the latter, but might come back to edit this comment if, upon reflection, I decide it’s actually more the former)

Sarah Weiler 14 Mar 2024 17:44 UTC
9 points
2 ∶ 0
in reply to: Chris Leong’s comment on: Critique of the notion that impact follows a power-law distribution
Wasn’t quite sure where best to respond in this thread, hope here makes decent sense.
I did actually seek to convey the claim that individuals do not differ massively in impact ex post (as well as ex ante, which I agree is the weaker and more easily defensible version of my claim). I was hoping to make that clear in this bullet point in the summary: “I claim that there are no massive differences in impact between individual interventions, individual organisations, and individual people, because impact is dispersed across [many actions]”. So, I do want to claim that: if we tried to apportion the impact of these consequences across contributing actions ex post, then no one individual action is massively higher in impact than the average action (with the caveat that net-negative actions and neutral actions are excluded; we only look at actions that have some substantial positive impact).
That said, I can see how my chosen title may be flawed because a) it leaves out large parts of what the post is about (adverse effects, conceptual debate); and b) it is stronger than my actual claim (the more truthful title would then need to be something like “There are probably no massive differences in impact between individuals (excluding individuals who have a net-negative or no significant impact on the world)”).
I am not sure if I agree that the current title is actively misleading and click-baity, but I take seriously the concern that it could be. I’ll mull this over some more and might change the title if I conclude that it is indeed inappropriate.
[EDIT: Concluded that changing the title seems sensible and appropriate. I hope that the new title is better able to communicate fully what my post is about.]
I’m obviously not super happy about the downvote, but I appreciate that you left the comment to explain and push me to reconsider, so thank you for that.
What links here?
- Critique of the notion that impact follows a power-law distribution by Sarah Weiler (14 Mar 2024 10:28 UTC; 88 points)

Sarah Weiler 14 Mar 2024 17:32 UTC
9 points
1 ∶ 1
in reply to: Owen Cotton-Barratt’s comment on: Critique of the notion that impact follows a power-law distribution
Thanks for that thoughtful comment!
- Agree that the adverse effects that I dedicate a large part of the post to do not speak to the question of whether impact actually follows a power-law distribution. They are just arguments against thinking about impact in that way. I think I acknowledge that repeatedly in the post, but can see now that the title makes it sound like I focus mainly on the “Empirical problem”.
- “I think that after you sort through this kind of consideration you would be able to recover some version of the power law claim basically intact”—I wonder if our disagreement on that is traceable and resolvable, or whether it stems from some pretty fundamental intuitions which it’s hard to argue about sensibly?
- ex ante vs. ex post: Interesting that you raise that! I’ve talked to a few people about the ideas in the essay, and I think something like your argument here was the most common response. I think I remain more persuaded by the claim that impact is not power law distributed at all, even ex post and not just because we don’t have the means to predict ex ante. But I agree that the case for a power law distribution is harder to defend ex ante (because of all the uncertainty) than ex post, and my confidence in doubting the claim is stronger for ex ante impact evaluations than it is for ex post evaluations.
- True and good point that I basically ignored the benefits of power-law thinking. I’ll consider whether I think my thoughts on these benefits can fit somewhere in the essay, and will update it accordingly if I find an appropriate fit. Thanks for pointing this out!
- Your conclusion sounds largely agreeable to me (though I imagine we would disagree when asked to specify how large the “tail-upsides” are that people should look for in a cautious manner).

Sarah Weiler 15 Mar 2024 16:21 UTC
8 points
1 ∶ 1
in reply to: Alex Semendinger’s comment on: Critique of the notion that impact follows a power-law distribution
Thanks for your comment, very happy to hear that my post struck you as clear and thorough (I’m never sure how well I do on clarity in my philosophical writing, since I usually retain a bit of confusion and uncertainty even in my own mind).
I agree that many dangers of internalizing the “heavy-tailed impact” perspective in the wrong way are due to misguided inference, not a strictly necessary implication of the perspective itself.
Not least thanks to input from several comments below, I am back to reconsidering my stance on the claims made in the essay around empirical reality and around appropriate conceptual frameworks. I have tangentially encountered Shapley values before but not yet really tried to understand the concept, so if you think they could be useful for the contents of this post, I’ll try to find the time to read the article you linked; thanks for the input!
I share the wariness that you mention re “arguments that have the form “even if X is true, believing / saying it has bad consequences, so we shouldn’t believe / say X.”″. At the same time, I don’t think that these arguments are always completely groundless (at least the arguments around refraining from saying something; much more inclined to agree that we should never believe something just for the sake of supposed better consequences from believing it). I also tend to be more sympathetic to these arguments when X is very hard to know (“we don’t really have means to tell whether X is true, and since believing in X might well have bad side-effects, we should not claim that X and we should maybe even make an effort to debunk the certainty with which others claim that X”). But yes, agree that wariness (though maybe not unconditional rejection) around arguments of this form is generally warranted, to avoid misguided dogmatism in the flawed attempt to prevent (supposed) information hazards.

Sarah Weiler 24 Nov 2023 8:13 UTC
8 points
0 ∶ 0
in reply to: Christopher Chan’s comment on: A case against focusing on tail-end nuclear war risks
Thanks a lot for taking the time to read & comment, Chris!
Main points
I want to take this opportunitiy to steelman your case: If the lower neglectedness and higher tractability of civil movement / policy in denuclearisation to less than 300 nuclear weapons (approximate number for not causing a nuclear winter) > higher neglectedness and lower tractability of physical intervention (resilience food and supply chain resilience plan), you might be correct!
I (honestly!) appreciate your willingness to steelman a case that (somewhat?) challenges your own views. However, I think I don’t endorse/buy the steelmanned argument as you put it here. It seems to me like the kind of simplified evaluation/description that I don’t think is very well-suited for assessing strategies to tackle societal problems. More specifically, I think the simple argument/relation you outline wrongfully ignores second- and third-order effects (incl. the adverse effects outlined in the post), which I believe are both extremely important and hard to simplify/formalize.
In a similar vein, I worry about the simplification in your comment on assessing the tractability of denuclearization efforts. I don’t think it’s appropriate to assess the impact of prior denuclearization efforts based simply on observed correlations, for two main reasons: first, there are numerous relevant factors aside from civil society’s denuclearization efforts, and the evidence we have access to has a fairly small sample size of observations that are not independent from each other, which means that identifying causal impact reliably is challenging if not impossible. Second, this is likely a “threshold phenomenon” (not sure what the official term would be), where observable cause-and-effect relations are not linear but occur in jumps; in other words, it seems likely here that civil society activism needs to build up to a certain level to result in clearly visible effects in terms of denuclearization (and the level of movement mobilisation required at any given time in history depends on a number of other circumstances, such as geopolitical and economic events). I don’t think that civil society activism for denuclearization is meaningless as long as it remains below that level, because I think it potentially has beneficial side-effects (on norms, culture, nuclear doctrine, decision-makers’ inhibitions against nuclear use, etc) and because we will never get above the threshold level if we consider efforts below the level to be pointless and not worth pursuing; but I do think that its visible effects as revealed by the evidence may well appear meaningless because of this non-linear nature of the causal relationship.
squeezeing the last 10% is an extremely hard up-hill battle if not impossible as countries continue to look up for their interests
I completely agree that denuclearization is an extremely hard up-hill battle, and I would argue that this is true even before the last 10% are reached. But I don’t think we have the evidence to say that it’s an impossible battle, and since I’m not convinced by the alternatives on offer (interventions “to the right of boom”, or simple nihilism/giving up/focusing on other things), I think it’s worthwhile—vital, actually—to keep fighting that uphill battle.
Some further side-notes
But note that at least half of the nuclear weapon deployed are in the hands of authoritarian countries [Russia: 3859, China: 410, North Korea: 30] which does not have good track record in listening to civil societies. While you could argue that Russia had drastically reduced their stockpile at the end of the cold war, many non-alligned countries [non-NATO, non Russia Bloc] have only increased their stockpile absolutely.
A short comment on the point about authoritarian states: At least for Russia and China, I think civil society/public opinion is far from unimportant (dictators tend to care about public approval at least to some extent) but agree that it’s a different situation from liberal democracies, which means that assessing the potential for denuclearization advocacy would require separate considerations for the different settings. On a different note, I think there is at least some case for claiming that changing attitudes/posture in the US has some influence on possibly shifting attitudes and debate in other nuclear-weapons states, especially when considered over a longer timeframe (just as examples: there are arguments that Putin’s current bellicosity is partially informed by continued US hostility and militarism esp. during the 2000s; and China justifies its arms build-up mainly by arguing that the huge gap between its arsenal and that of the US is unacceptable). All of this would require a much larger discussion (which would probably lead to no clear conclusion, because uncertainty is immense), so I wouldn’t be surprised nor blame you if the snippets of an argument presented above don’t change your mind on this particular point ^^
(And a side-note to the side-note: I think it’s worth pointing out that the biggest reduction in Russia’s stockpiles occurred before the end of the cold war, when the Soviet Union still seriously considered themselves a superpower)
Your argument reminds me of a perspective in animal welfare. If we improve the current condition of the billions of animal suffering, we have more of an excuse to slaughter them, in turn, empowering the meat companies, and thus it impedes our transition towards a cruelty free world.
The analogy makes sense to me, since both some of my claims and the animal advocats’ claim you mention seem to fall into the moral hazards category. Without having looked closely at the animal case, I don’t think I strongly share their concern in that case (or at least, I probably wouldn’t make the tradeoff of giving up on interventions to reduce suffering).
Again, thanks a lot for your comment and thoughts! Looking forward to hearing if you have any further thoughts on the answers given above.

Sarah Weiler 27 Mar 2024 22:43 UTC
7 points
0 ∶ 0
in reply to: Sarah Weiler’s comment on: Critique of the notion that impact follows a power-law distribution
New Update (as of 2024-03-27): This comment, with its very clear example to get to the bottom of our disagreement, has been extremely helpful in pushing me to reconsider some of the claims I make in the post. I have somewhat updated my views over the last few days (see the section on “the empirical problem” in the Appendix I added today), and this comment has been influential in helping me do that. Gave it a Delta for that reason; thanks Jeff!
While I now more explicitly acknowledge and agree that, when measured in terms of counterfactual impact, some actions can have hundreds of times more impact than others, I retain a sense of unease when adopting this framing:
When evaluating impact differently (e.g. through Shapley-value-like attribution of “shares of impact”, or through a collective rationality mindset (see comments here and here for what I mean by collective rationality mindset)), it seems less clear that the larger donor is 100x more impactful than the smaller donor. One way for reasoning about this would be something like: Probably—necessarily? - the person donating $100,000 had more preceding actions leading up to the situation where she is able and willing to donate that much money and there will probably—necessarily? - be more subsequent actions needed to make the money count, to ensure that it has positive consequences. There will then be many more actors and actions between which the impact of the $100,000 donation will have to be apportioned; it is not clear whether the larger donor will appear vastly more impactful when considered from this different perspective/measurement strategy...
You can shake your head and claim—rightly, I believe—that this is irrelevant for deciding whether donating $100,000 or donating $1,000 is better. Yes, for my decision as an individual, calculating the possible impact of my actions by assessing the likely counterfactual consequences resulting directly from the action will sometimes be the most sensible thing to do, and I’m glad I’ve come to realise that explicitly in response to your comment.
But I believe recognising and taking seriously the fact that, considered differently, my choice to donate $100,000 does not mean that I individually am responsible for 100x more impact than the donor of $1,000 can be relevant for decisions in two ways:
- 1) It prevents me from discounting and devaluing all the other actors that contribute vital inputs (even if they are “easily replaceable” as individuals)
- 2) It encourages me to take actions that may facilitate, enable, or support large counterfactual impact by other people. This perspective also encourages me to consider actions that may have a large counterfactual impact themselves, but in more indirect and harder-to-observe ways (even if I appear easily replaceable in theory, it’s unclear whether I will be replaced in practice, so the counterfactual impact seems extremely hard to determine; what is very clear is that by performing a relevant supportive action, I will be contributing something vital to the eventual impact).
If you find the time to come back to this so many days after the initial post, I’d be curious to hear what you think about these (still somewhat confused?) considerations :)

Sarah Weiler 15 Mar 2024 6:51 UTC
6 points
1 ∶ 2
in reply to: Owen Cotton-Barratt’s comment on: Critique of the notion that impact follows a power-law distribution
Appreciate the attempt to make headway on the disagreement!

I feel pretty lost when trying to quantify impact at these percentiles. Taking concerns about naive attribution of impact into consideration, I don’t even really know where to start to try to come up with numbers here. I just notice that I have a strong intuition, backed up by something that seems to me like a plausible claim: given that myriad actors always contribute to any outcome, it is hard to imagine that there is one (or a very few) individual(s) that does all of the heavy lifting...

“And how much spread do we need to get here in order to justify a lot of attention going into looking for tail-upsides?” —Also a good question. I think my answer would be: it depends on the situation and how much up- or downsides come along with looking for tail-upsides. If we’re cautious about the possible adverse effects of impact maximizing mindsets, I agree that it’s often sensible to look for tail-upsides even if they would “only” allow us to double impact. Then there are some situations/problems where I believe the collective rationality mindset, which looks for “how should I and my fellows behave in order to succeed as a community” rather than “how should I act now to maximize the impact I can have as a relatively direct/traceable outcome from my own action?”
What links here?
- Sam_Coggins's comment on Critique of the notion that impact follows a power-law distribution by Sarah Weiler (20 Mar 2024 23:42 UTC; 11 points)

Sarah Weiler 28 Dec 2023 8:20 UTC
6 points
2 ∶ 0
on: Reflections on the Biological Weapons Convention
Thanks for sharing your reflections here! I was rather taken aback when reading the headline and remain so after reading your “What I think” section:

You write that you’re “glad the BWC exists,” but also that you’re “weakly against the BWC.” Did I understand correctly that you mean that you’re glad that the legal agreement exists but are weakly against supporting or engaging in work to uphold the convention (at least the kind of work that is currently undertaken with that aim)? If so, could you explain whether you think that the durability and strength of the agreement is largely unaffected by such work; or that such work has value but less value than alternative efforts; or that such work has value but should be left to “others” (maybe: “non-EA-minded people” who are less focused on reducing unconventional risks)?

Whatever the answer to the above, I’m somewhat concerned by the normative effects of the phrasing you chose, and would be curious to hear your thoughts on those concerns. What I worry about is that your title and parts of the write-up suggest that the BWC and its surrounding structure is bad, is a waste of resources, and should thus be dismantled, or at least neglected even more than it already is. I share your sentiment that it’s good to have the BWC as some backstop against state development of biological weapons programmes; and I believe that this backstop works mostly through normative means (rather than some material enforcement mechanism). I further believe that such normative measures consist of how the BWC is spoken about (whether it is taken seriously, appreciated as important, etc) and of dedicated events to bring together relevant stakeholders (i.e. those people who might contribute to upholding and further developing the convention). The concern is that articles like yours contribute to undermining both.

Sarah Weiler 14 Mar 2024 12:57 UTC
5 points
1 ∶ 0
in reply to: Chris Leong’s comment on: Critique of the notion that impact follows a power-law distribution
I appreciate the sentiment and agree that preventing clickbaity titles from becoming more common on the EA forum is a valid goal!

I’d sincerely regret if my title does indeed fall into the “does not convey what the post is about” category. But as Jeff Kaufman already wrote, I’m not sure I understand in which sense the top-level claim is untrue to the main argument in the post. Is it because only part of the post is primarily about the empirical claim that impact does not differ massively between individuals?

Sarah Weiler 25 Mar 2023 5:45 UTC
5 points
2 ∶ 0
on: Predicting what future people value: A terse introduction to Axiological Futurism

Our descendants are unlikely to have values that are both different from ours in a very significant way and predictable. Either they have values similar to ours or they have values we can’t predict. Therefore, trying to predict their values is a waste of time and resources.

I’m strongly drawn to that response. I remain so after reading this initial post, but am glad that you, by writing this sequence, are offering the opportunity for someone like me to engage with the arguments/ideas a bit more! Looking forward to upcoming installments!

~~Wrote this on my phone and wasn’t offered the option to format the paragraph as a quote (and I don’t know what the command is); might come back to edit and fix it later~~

Sarah Weiler 11 Feb 2021 11:52 UTC
5 points
0 ∶ 0
in reply to: MichaelA’s comment on: Book review: The Doomsday Machine
I’m not an expert on the topic and don’t have sources on hand that would make the argument in greater detail, but I did take a course on ‘The global nuclear regime’ (broadly about institutional developments surrounding nuclear material and weapons control since 1945) and based on my knowledge from that, I’d suggest that there is a way to reconcile the two sets of claims.

First, I think it’s important to distinguish between ‘surprise attack’ and ‘first strike’. The former is obviously a subset of the latter, but there are also other conceivable kinds of first strike attacks. Surprise attack, to me, sounds like an attack that is launched without an immediate trigger, with the purpose of hitting (and eliminating or severly weakening) an adversary unexpectedly. A nuclear first strike might, instead, be considered in a situation where a conflict is escalading to a point that a nuclear strike by the other party seems to be growing more likely. It might be considered as an instrument to prevent the other party from launching their missiles by hitting them first (e.g. because the costs of waiting for them to launch before counter-striking are considered unacceptable). This comes down to definitions, ultimately, but I think I wouldn’t describe such a first strike as a surprise attack.

Second, there is not necessarily a contradiction between there being plans for first- rather than second-strike attacks and US officials expressing doubts about the USSR’s belief in US willingness to actually conduct a first strike. The US figures you mention might have thought that in that moment, the likelihood of a US first-strike was really low and that hence it would’ve been surprising for the USSR to start the detection project at that moment. These US figures might also have been disingenious or biased when assessing the honesty of the USSR leadership (I would argue that the tendency to attach hidden, often propagandistics, motives to ‘enemy leaders’ - without strong evidence base or even a coherent plausibility argument as support—is fairly common among US ‘hawks’). Debending on who the key US figures mentioned in your summary are (unfortunately, I haven’t read The Dead Hand), it might also be that they just weren’t aware of the first strike plans of the US. Lastly (and I don’t consider that one super likely), it might be that the US figures just thought that the Soviet leadership wouldn’t expect a US first strike in spite of the plans for it (either because the Soviets didn’t know about the plans, or because they didn’t think the US was likely to act on them).

Sarah Weiler 25 Mar 2024 19:23 UTC
4 points
0 ∶ 0
in reply to: Denis ’s comment on: Critique of the notion that impact follows a power-law distribution
Thanks a lot for that comment, Dennis. You might not believe it (judging by your comment towards the end), but I did read the full thing and am glad you wrote it all up!
I come away with the following conclusions:
1. It is true that we often credit individuals with impacts that were in fact the results of contributions from many people, often over long times.
2. However, there are still cases where individuals can have outsize impact compared to the counterfactual case where they do not exist.
3. It is not easy to say in advance which choices or which individuals will have these outsize influences …
4. … but there are some choices which seem to greatly increase the chance of being impactful.
Put in this way, I have very little to object. Thanks for providing that summary of your takeaways, I think that will be quite helpful to me as I continue to puzzle out my updated beliefs in response to all the comments the essay has gotten so far (see statements of confusion here and here).
For example, anyone who thinks that being a great teacher cannot be a super-impactful role is just wrong. But if you do a very simplistic analysis, you could conclude that. It’s only when you follow through all the complex chain of influences that the teacher has on the pupils, and that the pupils have on others, and so on, that you see the potential impact.
That’s interesting. I think I hadn’t really considered the possibility of putting really good teachers (and similar people-serving professions) into the super-high-impact category, and then my reaction was something like “If obviously essential and super important roles like teachers and nurses are not amongst the roles a given theory considers relevant and worth pursuing, then that’s suspicious and gives me reason to doubt the theory.” I now think that maybe I was premature in assuming that these roles would necessarily lie outside the super-high-impact category?
The real question, even of not always posed very precisely, is: for individuals who, for whatever reason, finds themselves in a particular situation, are there choices or actions that might make them 100x more impactful? [...] And yet, it feels like there are choices we make which can greatly increase or decrease the odds that we can make a positive and even an outsize contribution. And I’m not convinced by (what I understand to be) your position that just doing good without thinking too much about potential impact is the best strategy.
I think the sentiment behind those words is one that I wrongfully neglected in my post. For practical purposes, I think I agree that it can be useful and warranted to take seriously the possibility that some actions will have much higher counterfactual impact than others. I continue to believe that there are downsides or perils to the counterfactual perspective, and that it misses some relevant features of the world; but I can now also see more clearly that there are significant upsides to that same perspective and that it can often be a powerful tool for making the world better (if used in a nuanced way). Again, I haven’t settled on a neat stance to bring my competing thoughts together here, but I feel like some of your comments above will get me closer to that goal of conceptual clarification—thanks for that!

Sarah Weiler 16 Mar 2024 8:57 UTC
4 points
1 ∶ 0
in reply to: aogara’s comment on: The US expands restrictions on AI exports to China. What are the x-risk effects?
Late response, but may still be of interest: some colleagues and I spent some time surveying the existing literature on China x AI issues and the resource list we produced includes a section on Key actors and their views on AI risks. In general, I’d recommend the Concordia AI Safety newsletter for regular news of Chinese actors commenting on AI safety (and, more or less directly, on related x-risks).

Sarah Weiler 21 Mar 2024 15:45 UTC
3 points
0 ∶ 0
in reply to: Vasco Grilo’s comment on: Critique of the notion that impact follows a power-law distribution
Thanks for the comment! Just to make sure I understand correctly: the tails would partially cancel out in expected impact estimates because many actions with potentially high positive impact could also have potentially high negative impact if any of our assumptions are wrong? Or were you gesturing at something else? (Please feel free to simply point me to the post you shared if the answer is continued therein; I haven’t had the chance to read it carefully yet)

Sarah Weiler 21 Mar 2024 12:34 UTC
3 points
0 ∶ 0
in reply to: Jason’s comment on: Critique of the notion that impact follows a power-law distribution
[Sarah] But when it comes to “the world’s most pressing problems,” I don’t have the sense that we have those 95% of people to rely on to deal with the collective action problems.
[Jason] I had global health in mind—the vast majority of the funding and people on board are not EAs or conducting EA-type analyses (although many are at least considering cost-effectiveness).
Quick point on this: I didn’t mean to suggest that EAs constitute vastly more than 5% of people working on pressing problems. Completely agree that “the vast majority of the funding and people on board [in global health] are not EAs or conducting EA-type analyses”, but I still think that relatively few of those people (EA or not) approach problems with a collective rationality mindset, which would mean asking themselves: “how do I need to act if I want to be part of the collectively most rational solution?” rather than: “how do I need to act if I want to maximise the (counterfactual) impact from my next action?” or, as maybe done by many non-EA people in global health: “how should I act given my intuitive motivations and the (funding) opportunities available to myself?”. I think—based on anecdotal evidence and observation—that the first of these questions is not asked enough, inside EA and outside of it.
I can see some circumstances in which EA’s actions could be important for collective-action purposes. [...] It’s at least possible to estimate the expected impact of switching 100 votes in a swing state in a given election given what other actors are expected to do. It’s not easy, and the error bars are considerable, but it can be done.
I think it’s correct that some collective action problems can be addressed by individuals or small groups deciding to take action based on their counterfactual impact (and I thank you for the paper and proverb references, found it helpful to read these related ideas expressed in different terms!). In practice, I think (and you seem to acknowledge) that estimating that counterfactual impact for interventions aimed at disrupting collective action problems (by convincing lots of other people to behave collectively rational) is extremely hard and I thus doubt whether counterfactual impact calculations are the best (most practicable) tool for deciding whether and when to take such actions (I think the rather unintuitive analysis by 80,000 Hours on voting demonstrates the impracticability of these considerations for everyday decisions relatively well). But I can see how this might sometimes be a tenable and useful way to go. I find your reflections on how to do this interesting (checking for a plausible theory of change; checking for the closeness of reaching a required tipping point); my quick response (because this comment is already awfully long) would be that they seem useful but limited heuristics (what exactly makes a theory of change in deeply uncertain and empirically-poor domains “plausible”?; and for the tipping point, you mentioned my counterpoint already: if everybody always waiting for a fairly reachable tipping point, many large social changes would never have happened).
But the approach that I gesture at when I talk about “we should often act guided by principles of collective rationality” is different from guesstimating the counterfactual impact of an action that tries to break the collective action dilemma. I think what the collective rationality approach (in my mind) comes down to is an acceptance that sometimes we should take an action that has a low counterfactual impact, because our community (local, national, all of humanity) depends on many people taking such actions to add up to a huge impact. The very point of the collective action problem is that, counterfactually considered, my impact from taking that action will be low, because one individual taking or not taking the action is usually either completely or largely pointless. An example of that would be “making an effort to engage in dinner-table conversations on societally-important issues.” If (and I acknowledge this may be a controversial if) we believe that a vibrant and functioning democracy would be one where most citizens have such conversations every once in a while, then it would be collectively rational for me to engage in these conversations. But this will only really become an impactful and useful action (for our country’s democracy, ignoring benefits to myself) if many other citizens in my country do the same thing. And if many other citizens in my country do do the same thing, then paradoxically it doesn’t really matter that much anymore whether I do it; because it’s the mass doing it that counts, and any one individual that is added or subtracted from that mass has little effect. I think such dynamics can be captured by counterfactual impact reasoning only relatively unintuitively and in ways that are often empirically intractable in practice.
I don’t see anything about this hypothetical intervention that renders it incapable of empirical analysis. If one can determine how effective the organization is at dismantling stereotypes and self-images per dollar spent, then a donor can adjudicate the tradeoff between donating to it and donating to AMF based on how super harmful they think internalized racism is vs. how bad they think toddlers dying of malaria is.
Weakly agree that there can be some empirical analysis to estimate part of the effectiveness of the hypothetical stereotypes-intervention (though I do want to note that such estimates run a large risk of missing important, longer-running effects that only surface after long-time engagement and/or are not super easy to observe at all). I think the main point I was trying to make here is that the empirical question of “how bad internalized racism is”, i.e. how much it decreases development and flourishing, is one that seems hard if not impossible to address via quantitative empirical analysis. I could imagine your response being that we can run some correlational studies on communities or individuals with less vs. more internalized racism and then go from there; I don’t think this will give us meaningful causal knowledge given the many hidden variables that will differ between the groups we analyze and given the long-running effects we seek to find.
What links here?
- Sarah Weiler's comment on Critique of the notion that impact follows a power-law distribution by Sarah Weiler (27 Mar 2024 22:43 UTC; 7 points)

Sarah Weiler 18 Mar 2024 22:29 UTC
3 points
0 ∶ 0
in reply to: Jason’s comment on: Critique of the notion that impact follows a power-law distribution
I think I agree that the perspective I describe is far less relevant/valuable when 95% of actors hold and act in accordance with that perspective already. In those cases, it is relatively harmless to ignore the collective actions that would be required for the commom good because one can safely assume that the others (the 95%) will take care of those collective efforts by themselves. But when it comes to “the world’s most pressing problems,” I don’t have the sense that we have those 95% of people to rely on to deal with the collective action problems. And I think that, even if the situation is such that 95% of other people take care of collective efforts thus leaving room for 5% to choose actions unconstrained by responsibilities for those collective action needs, it remains useful and important to keep the collective rationality perspective in mind, to remember how much one relies on that large mass of people doing relatively mundane, but societally essential tasks.

I strongly sympathise with the concern of EA (or anyone) being pulled away from a drive to take action informed by robust data! I think especially for fields like Global Health (where we do have robust data for several, though not all, important questions), my response would be to insist that data-driven attempts to find particularly good actions as measured by their relatively direct, individual, counterfactual impact can, to some extent, coexist with and be complemented by a collective rationality perspective.

The way I imagine decision-making when based on both perspectives is something like: an action can be worth taking either because it has an exceptionally large expected counterfactual impact (e.g., donations to AMF); or it can be worth taking because a certain collective problem will not be solved unless many people take that kind of action (e.g., donations to an org that somehow works to dismantle colonial-area stereotypes and negative self-images in a localised setting within a formerly colonised country [please take the worldview-dependent beliefs underlying this, such as that internalised racism is super harmful to development and flourishing, as a given for the sake of the example]; or, more easy to do complementarily: working for a global health org and being very transparent and honest about the result of one’s interventions, and refraining from manipulative donation ads, even if that approach is expected to decrease donations at least in the short run [again, I’d suggest putting aside the question of whether the approach would in fact decrease donation volumes overall]), where any one person taking the action has an impact that is negligible or impossible to measure at all.

I don’t have a good answer for how to decide between these two buckets of action, especially when faced with a decision between two actions that need to be traded off against one another (donations to two different orgs) (my own current approach is to diversify somewhat arbitrarily, without a very clear distribution rule). But I would still argue that considering actions from both buckets as potentially worthwhile is the right way to go here. Curious to hear if that sparks any thoughts in response (and if you think it makes basic sense in the first place)!

Sarah Weiler 15 Mar 2024 16:35 UTC
3 points
0 ∶ 0
in reply to: Jason’s comment on: Critique of the notion that impact follows a power-law distribution
Thanks for reiterating the distinction, it seems quite helpful to the topic (on first consideration; I’ll have to mull this over a bit more over the next few days to really understand how the distinction fits into and may shift my thinking)!
I partially (largely?) agree with your comments. It seems right that in specific decision-situations, it will often not be relevant to consider how prior influences account for (and take away from the individual impact of) my own actions or the actions of a person I’m trying to influence. But I do think that it’s useful to remain aware of the fact that our actions are so heavily influenced by others, and especially that our actions will in turn contribute to influencing the behaviour (and thoughts, and attitudes) of many other people. Remaining aware of that fact seems to push away from evaluating actions only on the basis of how much counterfactual impact one can expect from that one isolated action: the fact that all actions are always the result of many preceding actions, where (often) every single preceding action only contributes a small part to shaping the resulting action, makes conceivable the idea that actions with low direct counterfactual impact can still be quite important and justifiable when considered from a perspective of behavioural heuristics or collective rationality (both of which recognise that some hugely important outcomes can only ever be attained if many people decide to take actions that have low expected impact on their own).
What links here?
- Sarah Weiler's comment on Critique of the notion that impact follows a power-law distribution by Sarah Weiler (27 Mar 2024 22:43 UTC; 7 points)

Sarah Weiler 15 Mar 2024 7:33 UTC
3 points
0 ∶ 0
in reply to: Ben Stevenson’s comment on: Critique of the notion that impact follows a power-law distribution
Yikes, I linked to the wrong Todd article there, apologies! Meat consumption is mentioned in Todd 2021(2023):
We’ve basically found this pattern wherever data is available.
- Other global health datasets, such as WHO-CHOICE and the DCP3
- Public health in rich countries
- Education interventions in developing countries
- Ways to alter meat consumption to reduce animal suffering
- Policies to reduce CO2 emissions
I’ll add the source to that part of the essay, thanks for the alert!

Sarah Weiler

[EDIT: Concluded that changing the title seems sensible and appropriate. I hope that the new title is better able to communicate fully what my post is about.]

Main points

Some further side-notes