Researcher (on bio) at FHI
Although I understand the nationalism example isn’t meant to be analogous, but my impression is this structural objection only really applies when our situation is analogous. If historically EA paid a lot of attention to nationalism (or trans-humanism, the scepticism community, or whatever else) but had by-and-large collectively ‘moved on’ from these, contemporary introductions to the field shouldn’t feel obliged to cover them extensively, nor treat it the relative merits of what they focus on now versus then as an open question.Yet, however you slice it, EA as it stands now hasn’t by-and-large ‘moved on’ to be ‘basically longtermism’, where its interest in (e.g) global health is clearly atavistic. I’d be willing to go to bat for substantial slants to longtermism, as (I aver) its over-representation amongst the more highly engaged and the disproportionate migration of folks to longtermism from other areas warrants claims that epistocratic weighting of consensus would favour longtermism over anything else. But even this has limits which ‘greatly favouring longtermism over everything else’ exceeds. How you choose to frame an introduction is up for grabs, and I don’t think ‘the big three’ is the only appropriate game in town. Yet if your alternative way of framing an introduction to X ends up strongly favouring one aspect (further, the one you are sympathetic to) disproportionate to any reasonable account of its prominence within X, something has gone wrong.
Per others: This selection isn’t really ‘leans towards a focus on longtermism’, but rather ‘almost exclusively focuses on longtermism’: roughly any ‘object level’ cause which isn’t longtermism gets a passing mention, whilst longtermism is the subject of 3⁄10 of the selection. Even some not-explicitly-longtermist inclusions (e.g. Tetlock, MacAskill, Greaves) ‘lean towards’ longtermism either in terms of subject matter or affinity.Despite being a longtermist myself, I think this is dubious for a purported ‘introduction to EA as a whole’: EA isn’t all-but-exclusively longtermist in either corporate thought or deed.Were I a more suspicious sort, I’d also find the ‘impartial’ rationales offered for why non-longtermist things keep getting the short (if not pointy) end of the stick scarcely credible:
i) we decided to focus on our overall worldview and way of thinking rather than specific cause areas (we also didn’t include a dedicated episode on biosecurity, one of our ‘top problems’), and ii) both are covered in the first episode with Holden Karnofsky, and we prominently refer people to the Bollard and Glennerster interviews in our ‘episode 0’, as well as the outro to Holden’s episode.
The first episode with Karnofsky also covers longtermism and AI—at least as much as global health and animals. Yet this didn’t stop episodes on the specific cause areas of longtermism (Ord) and AI (Christiano) being included. Ditto the instance of “entrepreneurship, independent thinking, and general creativity” one wanted to highlight just-so-happens to be a longtermist intervention (versus, e.g. this).
I also thought along similar lines, although (lacking subtlety) I thought you could shove in a light cone from the dot, which can serve double duty as the expanding future. Another thing you could do is play with a gradient so this curve/the future gets brighter as well as bigger, but perhaps someone who can at least successfully colour in have a comparative advantage here.
A less important motivation/mechanism is probabilities/ratios (instead of odds) are bounded above by one. For rare events ‘doubling the probability’ versus ‘doubling the odds’ get basically the same answer, but not so for more common events. Loosely, flipping a coin three times ‘trebles’ my risk of observing it landing tails, but the probability isn’t 1.5. (cf).
Sibling abuse rates are something like 20% (or 80% depending on your definition). And is the most frequent form of household abuse. This means by adopting a child you are adding something like an additional 60% chance of your other child going through at least some level of abuse (and I would estimate something like a 15% chance of serious abuse). [my emphasis]
If you used the 80% definition instead of 20%, then the ‘4x’ risk factor implied by 60% additional chance (with 20% base rate) would give instead an additional 240% chance.[(Of interest, 20% to 38% absolute likelihood would correspond to an odds ratio of ~2.5, in the ballpark of 3-4x risk factors discussed before. So maybe extrapolating extreme event ratios to less-extreme event ratios can do okay if you keep them in odds form. The underlying story might have something to do with logistic distributions closely resemble normal distributions (save at the tails), so thinking about shifting a normal distribution across the x axis so (non-linearly) more or less of it lies over a threshold loosely resembles adding increments to log-odds (equivalent to multiplying odds by a constant multiple) giving (non-linear) changes when traversing a logistic CDF.But it still breaks down when extrapolating very large ORs from very rare events. Perhaps the underlying story here may have something to do with higher kurtosis : ‘>2SD events’ are only (I think) ~5X more likely than >3SD events for logistic distributions, versus ~20X in normal distribution land. So large shifts in likelihood of rare(r) events would imply large logistic-land shifts (which dramatically change the whole distribution, e.g. an OR of 10 makes evens --> >90%) much more modest in normal-land (e.g. moving up an SD gives OR>10 for previously 3SD events, but ~2 for previously ‘above average’ ones)]
Most views in population ethics can entail weird/intuitively toxic conclusions (cf. the large number of’X conclusion’s out there). Trying to weigh these up comparatively are fraught.In your comparison, it seems there’s a straightforward dominance argument if the ‘OC’ and ‘RC’ are the things we should be paying attention to. Your archetypal classical utilitarian is also committed to the OC as ‘large increase in suffering for one individual’ can be outweighed by a large enough number of smaller decreases in suffering for others—aggregation still applies to negative numbers for classical utilitarians. So the negative view fares better as the classical one has to bite one extra bullet.There’s also the worry in a pairwise comparison one might inadvertently pick a counterexample for one ‘side’ that turns the screws less than the counterexample for the other one. Most people find the ‘very repugnant conclusion’ (where not only Z > A, but ‘large enough Z and some arbitrary number having awful lives > A’) even more costly than the ‘standard’ RC. So using the more or less costly variant on one side of the scales may alter intuitive responses.By my lights, it seems better to have some procedure for picking and comparing cases which isolates the principle being evaluated. Ideally, the putative counterexamples share counterintuitive features both theories endorse, but differ in one is trying to explore the worst case that can be constructed which the principle would avoid, whilst the other the worst case that can be constructed with its inclusion.It seems the main engine of RC-like examples is the aggregation—it feels like one is being nickel-and-dimed taking a lot of very small things to outweigh one very large thing, even though the aggregate is much higher. The typical worry a negative view avoids is trading major suffering for sufficient amounts of minor happiness—most typically think this is priced too cheaply, particularly at extremes. The typical worry of the (absolute) negative view itself is it fails to price happiness at all—yet often we’re inclined to say enduring some suffering (or accepting some risk of suffering) is a good deal at least at some extreme of ‘upside’.So with this procedure the putative counter-example to the classical view would be the vRC. Although negative views may not give crisp recommendations against the RC (e.g. if we stipulate no one ever suffers in any of the worlds, but are more or less happy), its addition clearly recommends against the vRC: the great suffering isn’t outweighed by the large amounts of relatively trivial happiness (but it would be on the classical view).Yet with this procedure, we can construct a much worse counterexample to the negative view than the OC—by my lights, far more intuitively toxic than the already costly vRC. (Owed to Carl Shulman). Suppose A is a vast but trivially-imperfect utopia—Trillions (or googleplexes, or TREE(TREE(3))) lives lives of all-but-perfect bliss, but for each enduring an episode of trivial discomfort or suffering (e.g. a pin-prick, waiting a queue for an hour). Suppose Z is a world with a (relatively) much smaller number of people (e.g. a billion) living like the child in Omelas. The negative view ranks Z > A: the negative view only considers the pinpricks in this utopia, and sufficiently huge magnitudes of these can worse than awful lives (the classical view, which wouldn’t discount all the upside in A, would not). In general, this negative view can countenance any amount of awful suffering if this is the price to pay to abolish a near-utopia of sufficient size.(This axiology is also anti-egalitarian (consider replacing half the people in A with half the people in Z) and—depending how you litigate—susceptible to a sadistic conclusion. If the axiology claims welfare is capped above by 0, then there’s never an option of adding positive welfare lives so nothing can be sadistic. If instead it discounts positive welfare, then it prefers (given half of A) adding half of Z (very negative welfare lives) to adding the other half of A (very positive lives)).I take this to make absolute negative utilitarianism (similar to average utilitarianism) a non-starter. In the same way folks look for a better articulation of egalitarian-esque commitments that make one (at least initially) sympathetic to average utilitarianism, so folks with negative-esque sympathies may look for better articulations of this commitment. One candidate could be what one is really interested in cases of severe rather than trivial suffering, so this rather than suffering in general should be the object of sole/lexically prior concern. (Obviously there are many other lines, and corresponding objections to each).But note this is an anti-aggregation move. Analogous ones are available for classical utilitarians to avoid the (v/)RC (e.g. a critical-level view which discounts positive welfare below some threshold). So if one is trying to evaluate a particular principle out of a set, it would be wise to aim for ‘like-for-like’: e.g. perhaps a ‘negative plus a lexical threshold’ view is more palatable than classical util, yet CLU would fare even better than either.
[Mea culpa re. messing up the formatting again]1) I don’t closely follow the current state of play in terms of ‘shorttermist’ evaluation. The reply I hope (e.g.) a Givewell Analyst would make to (e.g.) “Why aren’t you factoring in impacts on climate change for these interventions?” would be some mix of:a) “We have looked at this, and we’re confident we can bound the magnitude of this effect to pretty negligible values, so we neglect them in our write-ups etc.”b) “We tried looking into this, but our uncertainty is highly resilient (and our best guess doesn’t vary appreciably between interventions) so we get higher yield investigating other things.”c) “We are explicit our analysis is predicated on moral (e.g. “human lives are so much more important than animals lives any impact on the latter is ~moot”) or epistemic (e.g. some ‘common sense anti-cluelessness’ position) claims which either we corporately endorse and/or our audience typically endorses.” Perhaps such hopes would be generally disappointed.2) Similar to above, I don’t object to (re. animals) positions like “Our view is this consideration isn’t a concern as X” or “Given this consideration, we target Y rather than Z”, or “Although we aim for A, B is a very good proxy indicator for A which we use in comparative evaluation.”But I at least used to see folks appeal to motivations which obviate (inverse/) logic of the larder issues, particularly re. diet change (“Sure, it’s actually really unclear becoming vegan reduces or increases animal suffering overall, but the reason to be vegan is to signal concern for animals and so influence broader societal attitudes, and this effect is much more important and what we’re aiming for”). Yet this overriding motivation typically only ‘came up’ in the context of this discussion, and corollary questions like:* “Is maximizing short term farmed animal welfare the best way of furthering this crucial goal of attitude change?”
* “Is encouraging carnivores to adopt a vegan diet the best way to influence attitudes?”
* “Shouldn’t we try and avoid an intervention like v*ganism which credibly harms those we are urging concern for, as this might look bad/be bad by the lights of many/most non-consequentialist views?” seemed seldom asked. Naturally I hope this is a relic of my perhaps jaundiced memory.
FWIW, I don’t think ‘risks’ is quite the right word: sure, if we discover a risk which was so powerful and so tractable that we end up overwhelming the good done by our original intervention, that obviously matters. But the really important thing there, for me at least, is the fact that we apparently have a new and very powerful lever for impacting the world. As a result, I would care just as much about a benefit which in the medium term would end up being worth >>1x the original target good (e.g. “Give Directly reduces extinction risk by reducing poverty, a known cause of conflict”); the surprisingly-high magnitude of an incidental impact is what is really catching my attention, because it suggests there are much better ways to do good.
(Apologies in advance I’m rehashing unhelpfully)The usual cluelessness scenarios are more about that there may be powerful lever for impacting the future, and your intended intervention may be pulling it in the wrong direction (rather than a ‘confirmed discovery’). Say your expectation for the EV of GiveDirectly on conflict has a distribution with a mean of zero but an SD of 10x the magnitude of the benefits you had previously estimated. If it were (e.g.) +10, there’s a natural response of ‘shouldn’t we try something which targets this on purpose?‘; if it were 0, we wouldn’t attend to it further; if it meant you were −10, you wouldn’t give to (now net EV = “-9”) GiveDirectly. The right response where all three scenarios are credible (plus all the intermediates) but you’re unsure which one you’re in isn’t intuitively obvious (at least to me). Even if (like me) you’re sympathetic to pretty doctrinaire standard EV accounts (i.e. you quantify this uncertainty + all the others and just ‘run the numbers’ and take the best EV) this approach seems to ignore this wide variance, which seems to be worthy of further attention.The OP tries to reconcile this with the standard approach by saying this indeed often should be attended to, but under the guise of value of information rather than something ‘extra’ to orthodoxy. Even though we should still go with our best guess if we to decide (so expectation neutral but high variance terms ‘cancel out’), we might have the option to postpone our decision and improve our guesswork. Whether to take that option should be governed by how resilient our uncertainty is. If your central estimate of GiveDirectly and conflict would move on average by 2 units if you spent an hour thinking about it, that seems an hour well spent; if you thought you could spend a decade on it and remain where you are, going with the current best guess looks better. This can be put in plain(er) English (although familiar-to-EA jargon like ‘EV’ may remain). Yet there are reasons to be hesitant about the orthodox approach (even though I think the case in favour is ultimately stronger): besides the usual bullets, we would be kidding ourselves if we ever really had in our head an uncertainty distribution to arbitrary precision, and maybe our uncertainty isn’t even remotely approximate to objects we manipulate in standard models of the same. Or (owed to Andreas) even if so, similar to how rule-consequentialism may be better than act-consequentialism, some other epistemic policy would get better results than applying the orthodox approach in these cases of deep uncertainty. Insofar as folks are more sympathetic to this, they would not want to be deflationary and perhaps urge investment in new techniques/vocab to grapple with the problem. They may also think we don’t have a good ‘answer’ yet of what to do in these situations, so may hesitate to give ‘accept there’s uncertainty but don’t be paralysed by it’ advice that you and I would. Maybe these issues are an open problem we should try and figure out better before pressing on.
Belatedly:I read the stakes here differently to you. I don’t think folks thinking about cluelessness see it as substantially an exercise in developing a defeater to ‘everything which isn’t longtermism’. At least, that isn’t my interest, and I think the literature has focused on AMF etc. more as salient example to explore the concepts, rather than an important subject to apply them to. The AMF discussions around cluelessness in the OP are intended as toy example—if you like, deliberating purely on “is it good or bad to give to AMF versus this particular alternative?” instead of “Out of all options, should it be AMF?” Parallel to you, although I do think (per OP) AMF donations are net good, I also think (per the contours of your reply) it should be excluded as a promising candidate for the best thing to donate to: if what really matters is how the deep future goes, and the axes of these accessible at present are things like x-risk, interventions which are only tangentially related to these are so unlikely to be best they can be ruled-out ~immediately.
So if that isn’t a main motivation, what is? Perhaps something like this:1) How to manage deep uncertainty over the long-run ramifications of ones decisions is a challenge across EA-land—particularly acute for longtermists, but also elsewhere: most would care about risks about how in the medium term a charitable intervention could prove counter-productive. In most cases, these mechanisms for something to ‘backfire’ are fairly trivial, but how seriously credible ones should be investigated is up for grabs.Although “just be indifferent if it is hard to figure out” is a bad technique which finds little favour, I see a variety of mistakes in and around here. E.g.:a) People not tracking when the ground of appeal for an intervention has changed. Although I don’t see this with AMF, I do see this in and around animal advocacy. One crucial consideration around here is WAS, particularly an ‘inverse logic of the larder’ (see), such as “per area, a factory farm has a lower intensity of animal suffering than the environment it replaced”. Even if so, it wouldn’t follow the best thing to do would to be as carnivorous as possible. There are also various lines of response. However, one is to say that the key objective of animal advocacy is to encourage greater concern about animal welfare, so that this can ramify through to benefits in the medium term. However, if this is the rationale, metrics of ‘animal suffering averted per $’ remain prominent despite having minimal relevance. If the aim of the game is attitude change, things like shelters and companion animals over changes in factory farmed welfare start looking a lot more credible again in virtue of their greater salience.b) Early (or motivated) stopping across crucial considerations. There are a host of ramifications to population growth which point in both directions (e.g. climate change, economic output, increased meat consumption, larger aggregate welfare, etc.) Although very few folks rely on these when considering interventions like AMF (but cf.) they are often being relied upon by those suggesting interventions specifically targeted to fertility: enabling contraceptive access (e.g. more contraceptive access --> fewer births --> less of a poor meat eater problem), or reducing rates of abortion (e.g. less abortion --> more people with worthwhile lives --> greater total utility).Discussions here are typically marred by proponents either completely ignoring considerations on the ‘other side’ of the population growth question, or giving very unequal time to them/sheltering behind uncertainty (e.g. “Considerations X, Y, and Z all tentatively support more population growth, admittedly there’s A, B, C, but we do not cover those in the interests of time—yet, if we had, they probably would tentatively oppose more population growth”). 2) Given my fairly deflationary OP, I don’t think these problems are best described as cluelessness (versus attending to resilient uncertainty and VoI in fairly orthodox evaluation procedures). But although I think I’m right, I don’t think I’m obviously right: if orthodox approaches struggle here, less orthodox ones with representors, incomparability or other features may be what should be used in decision-making (including when we should make decisions versus investigate further). If so then this reasoning looks like a fairly distinct species which could warrant it’s own label.
I may be missing the thread, but the ‘ignoring’ I’d have in mind for resilient cluelessness would be straight-ticket precision, which shouldn’t be intransitive (or have issues with principle of indifference).E.g. Say I’m sure I can make no progress on (e.g.) the moral weight of chickens versus humans in moral calculation—maybe I’m confident there’s no fact of the matter, or interpretation of the empirical basis is beyond our capabilities forevermore, or whatever else.Yet (I urge) I should still make a precise assignment (which is not obliged to be indifferent/symmetrical), and I can still be in reflective equilibrium between these assignments even if I’m resiliently uncertain.
Mea culpa. I’ve belatedly ‘fixed’ it by putting it into text.
The issue is more the being stuck than the range: say it is (0.4, 0.6) rather than (0, 1), you’d still be inert. Vallinder (2018) discusses this extensively, including issues around infectiousness and generality.
For my part, I’m more partial to ‘blaming the reader’, but (evidently) better people mete out better measure than I in turn.Insofar as it goes, I think the challenge (at least for me) is qualitative terms can cover multitudes (or orders of magnitudes) of precision. I’d take ~0.3% to be ‘significant’ credence for some values of significant. ‘Strong’ ‘compelling’ or ‘good’ arguments could be an LR of 2 (after all, RCT confirmation can be ~3) or 200. I also think quantitative articulation would help the reader (or at least this reader) better benchmark the considerations here. Taking the rough posterior of 0.1% and prior of 1 in 100 million, this implies a likelihood ratio of ~~100 000 - loosely, ultra-decisive evidence. If we partition out the risk-based considerations (which it discussion seems to set as ‘less than decisive’ so <100), the other considerations (perhaps mostly those in S5) give you a LR of > ~1000 - loosely, very decisive evidence. Yet the discussion of the considerations in S5 doesn’t give the impression we should conclude they give us ‘massive updates’. You note there are important caveats to these considerations, you say in summing up these arguments are ‘far from watertight’, and I also inferred the sort of criticisms given in S3 around our limited reasoning ability and scepticism of informal arguments would also apply here too. Hence my presumption these other considerations, although more persuasive than object level arguments around risks, would still end up below the LR ~ 100 for ‘decisive’ evidence, rather than much higher. Another way this would help would be illustrating the uncertainty. Given some indicative priors you note vary by ten orders of magnitude, the prior is not just astronomical but extremely uncertain. By my lights, the update doesn’t greatly reduce our uncertainty (and could compound it, given challenges in calibrating around very high LRs). If the posterior odds could be ‘out by 100 000x either way’ the central estimate being at ~0.3% could still give you (given some naive log-uniform) 20%+ mass distributed at better than even odds of HH. The moaning about hiding the ball arises from the sense this numerical articulation reveals (I think) some powerful objections the more qualitative treatment obscures. E.g.
Typical HH proponents are including considerations around earliness/single planet/ etc. in their background knowledge/prior when discussing object level risks. Noting the prior becomes astronomically adverse when we subtract these out of background knowledge, and so the object level case for (e.g.) AI risk can’t possibly be enough to carry the day alone seems a bait-and-switch: you agree the prior becomes massively less astronomical when we include single planet etc. in background knowledge, and in fact things like ‘we live on only one planet’ are in our background knowledge (and were being assumed at least tacitly by HH proponents).
The attempt to ‘bound’ object level arguments by their LR (e.g. “Well, these are informal, and it looks fishy, etc. so it is hard to see how you can get LR >100 from these”) doesn’t seem persuasive when your view is that the set of germane considerations (all of which seem informal, have caveats attached, etc.) in concert are giving you an LR of ~100 000 or more. If this set of informal considerations can get you more than half way from the astronomical prior to significant credence, why be so sure additional ones (e.g.) articulating a given danger can’t carry you the rest of the way?
I do a lot of forecasting, and I struggle to get a sense of what priors of 1/ 100 M or decisive evidence to the tune of LR 1000 would look like in ‘real life’ scenarios. Numbers this huge (where you end up virtually ‘off the end of the tail’ of your stipulated prior) raise worries about consilience (cf. “I guess the sub-prime morgage crisis was a 10 sigma event”), but moreover pragmatic defeat: there seems a lot to distrust in an epistemic procedure along the lines of “With anthropics given stipulated subtracted background knowledge we end up with an astronomically minute prior (where we could be off by many orders of magnitude), but when we update on adding back in elements of our actual background knowledge this shoots up by many orders of magnitude (but we are likely still off by many orders of magnitude)”. Taking it face value would mean a minute update to our ‘pre theoretic prior’ on the topic before embarking on this exercise (providing these overlapped and was not as radically uncertain, varying no more than a couple rather than many orders of magnitude). If we suspect (which I think we should) this procedure of partitioning out background knowledge into update steps which approach log log variance and where we have minimal calibration is less reliable than using our intuitive gestalt over our background knowledge as whole, we should discount its deliverances still further.
But what is your posterior? Like Buck, I’m unclear whether your view is the central estimate should be (e.g.) 0.1% or 1 / 1 million. I want to push on this because if your own credences are inconsistent with your argument, the reasons why seem both important to explore and to make clear to readers, who may be mislead into taking this at ‘face value’. From this on page 13 I guess a generous estimate (/upper bound) is something like 1/ 1 million for the ‘among most important million people’:
[W]e can assess the quality of the arguments given in favour of the Time of Perils or Value Lock-in views, to see whether, despite the a priori implausibility and fishiness of HH, the evidence is strong enough to give us a high posterior in HH. It would take us too far afield to discuss in sufficient depth the arguments made in Superintelligence, or Pale Blue Dot, or The Precipice. But it seems hard to see how these arguments could be strong enough to move us from a very low prior all the way to significant credence in HH. As a comparison, a randomised controlled trial with a p-value of 0.05, under certain reasonable assumptions, gives a Bayes factor of around 3 in favour of the hypothesis; a Bayes factor of 100 is regarded as ‘decisive’ evidence. In order to move from a prior of 1 in 100 million to a posterior of 1 in 10, one would need a Bayes factor of 10 million — extraordinarily strong evidence.
I.e. a prior of ~ 1/ 100 million (which is less averse than others you moot earlier), and a Bayes factor < 100 (i.e. we should not think the balance of reason, all considered, is ‘decisive’ evidence), so you end up at best at ~1/ 1 million. If this argument is right, you can be ‘super confident’ giving a credence of 0.1% is wrong (out by an ratio of >~ 1000, the difference between ~ 1% and 91%), and vice-versa. Yet I don’t think your credence on ‘this is the most important century’ is 1/ 1 million. Among other things it seems to imply we can essentially dismiss things like short TAI timelines, Bostrom-Yudkowsky AI accounts etc, as these are essentially upper-bounded by the 1/ 1M credence above.*So (presuming I’m right and you don’t place negligible credence on these things) I’m not sure how these things can be in reflective equilibrium.1: ‘Among the most important million people’ and ‘this is the most important century’ are not the same thing, and so perhaps one has a (much) higher prior on the latter than the former. But if the action really was here, then the precisification of ‘hinge of history’ as the former claim seems misguided: “Oh, this being the most important century could have significant credence, but this other sort-of related proposition nonetheless has an astronomically adverse prior” confuses rather than clarifies.2: Another possibility is there are sources of evidence which give us huge updates, even if the object level arguments in (e.g.) Superintelligence, The Precipice etc. are not among them. Per the linked conversation, maybe earliness gives a huge shift up from the astronomically adverse prior, so this plus the weak object level evidence gets you to lowish but not negligible credence. Whether cashed out via prior or update, it seems important to make such considerations explicit, as the true case in favour of HH would include these considerations too. Yet the discussion of ‘how far you should update’ on p11-13ish doesn’t mention these massive adjustments, instead noting reasons to be generally sceptical (e.g. fishiness) and the informal/heuristic arguments for object level risks should not be getting you Bayes factors ~100 or more. This seems to be hiding the ball if in fact your posterior is ultimately 1000x or more your astronomically adverse prior, but not for reasons which are discussed (and so a reader may neglect to include when forming their own judgement). *: I think there’s also a presumptuous philosopher-type objection lurking here too. Folks (e.g.) could have used a similar argument to essentially rule out any x-risk from nuclear winter before any scientific analysis, as this implies significant credence in HH, which the argument above essentially rules out. Similar to ‘using anthropics to hunt’, something seems to be going wrong where the mental exercise of estimating potentially-vast future populations can also allow us to infer the overwhelming probable answers for disparate matters in climate modelling, AI development, the control problem, civilisation recovery and so on.
“It’s not clear why you’d think that the evidence for x-risk is strong enough to think we’re one-in-a-million, but not stronger than that.” This seems pretty strange as an argument to me. Being one-in-a-thousand is a thousand times less likely than being one-in-a-million, so of course if you think the evidence pushes you to thinking that you’re one-in-a-million, it needn’t push you all the way to thinking that you’re one-in-a-thousand. This seems important to me. Yes, you can give me arguments for thinking that we’re (in expectation at least) at an enormously influential time—as I say in the blog post and the comments, I endorse those arguments! I think we should update massively away from our prior, in particular on the basis of the current rate of economic growth. (My emphasis)
Asserting an astronomically adverse prior, then a massive update, yet being confident you’re in the right ballpark re. orders of magnitude does look pretty fishy though. For a few reasons:First, (in the webpage version you quoted) you don’t seem sure of a given prior probability, merely that it is ‘astronomical’: yet astronomical numbers (including variations you note about whether to multiply by how many accessible galaxies there are or not, etc.) vary by substantially more than three orders of magnitude—you note two possible prior probabilities (of being among the million most influential people) of 1 in a million trillion (10^-18) and 1 in a hundred million (10^-8) - a span of 10 orders of magnitude. It seems hard to see how a Bayesian update from this (seemingly) extremely wide prior would give a central estimate at a (not astronomically minute) value, yet confidently rule against values ‘only’ 3 orders of magnitude higher (a distance a ten millionth the width of this implicit span in prior probability). [It also suggests the highest VoI is to winnow this huge prior range, rather than spending effort evaluating considerations around the likelihood ratio]Second, whatever (very) small value we use for our prior probability, getting to non-astronomical posteriors implies likelihood ratios/Bayes factors which are huge. From (say) 10^-8 to 10^-4 is a factor of 10 000. As you say in your piece, this is much much stronger than the benchmark for decisive evidence of ~100. It seems hard to say (e.g.) evidence from the rate of economic growth is ‘decisive’ in this sense, and so it is hard to see how in concert with other heuristic considerations you get 10-100x more confirmation (indeed, your subsequent discussion seems to supply many defeaters exactly this). Further, similar to worries about calibration out on the tail, it seems unlikely many of us can accurately assess LRs > 100 which are not direct observations within orders of magnitude. Third, priors should be consilient, and can be essentially refuted by posteriors. A prior that get surprised to the tune of a 1-in-millions should get hugely penalized versus any alternative (including naive intuitive gestalts) which do not. It seems particularly costly as non-negligible credences in (e.g.) nuclear winter, the industrial revolution being crucial etc. are facially represent this prior being surprised by ‘1 in large X’ events at a rate much greater than 1/X.To end up with not-vastly lower posteriors than your interlocutors (presuming Buck’s suggestion of 0.1% is fair, and not something like 1/million), it seems one asserts both a much lower prior which is mostly (but not completely) cancelled out by a much stronger update step. This prior seems to be ranging over many orders of magnitude, yet the posterior does not—yet it is hard to see where the orders of magnitude of better resolution are arising from (if we knew for sure the prior is 10^-12 versus knowing for sure it is 10^-8, shouldn’t the posterior shift a lot between the two cases?)It seems more reasonable to say ‘our’ prior is rather some mixed gestalt on considering the issue as a whole, and the concern about base-rates etc. should be seen as an argument for updating this downwards, rather than a bid to set the terms of the discussion.
I agree with this in the abstract, but for the specifics of this particular case, do you in fact think that online mobs / cancel culture / groups who show up to protest your event without warning should be engaged with on a good faith assumption? I struggle to imagine any of these groups accepting anything other than full concession to their demands, such that you’re stuck with the BATNA regardless.
I think so. In the abstract, ‘negotiating via ultimatum’ (e.g. “you must cancel the talk, or I will do this”) does not mean one is acting in bad faith. Alice may foresee there is no bargaining frontier, but is informing you what your BATNA looks like and gives you the opportunity to consider whether ‘giving in’ is nonetheless better for you (this may not be very ‘nice’, but it isn’t ‘blackmail’). A lot turns on whether her ‘or else’ is plausibly recommended by the lights of her interests (e.g. she would do these things if we had already held the event/she believed our pre-commitment to do so) or she is threatening spiteful actions where their primary value is her hope they alter our behaviour (e.g. she would at least privately wish she didn’t have to ‘follow through’ if we defied her). The reason these are important to distinguish is ‘folk game theory’ gives a pro tanto reason to not give in the latter case, even if doing so is better than suffering the consequences (as you deter future attempts to coerce you). But not in the former one, as Alice’s motivation to retaliate does not rely on the chance you may acquiesce to her threats, and so she will not ‘go away’ after you’ve credibly demonstrated to her you will never do this. On the particular case I think some of it was plausibly bad faith (i.e. if a major driver was ‘fleet in being’ threat that people would antisocially disrupt the event) but a lot of it probably wasn’t: “People badmouthing/thinking less of us for doing this” or (as Habryka put it) the ‘very explicit threat’ of an organisation removing their affiliation from EA Munich are all credibly/probably good faith warnings even if the only way to avoid them would have been complete concession. (There are lots of potential reasons I would threaten to stop associating with someone or something where the only way for me to relent is their complete surrender)(I would be cautious about labelling things as mobs or cancel culture.)
[G]iven that she’s taking actions that destroy value for Bob without generating value for Alice (except via their impact on Bob’s actions), I think it is fine to think of this as a threat. (I am less attached to the bully metaphor—I meant that as an example of a threat.)
Let me take a more in-group example readers will find sympathetic.When the NYT suggested it will run an article using Scott’s legal name, may of his supporters responded by complaining to the editor, organising petitions, cancelling their subscriptions (and encouraging others to do likewise), trying to coordinate sources/public figures to refuse access to NYT journalists, and so on. These are straightforwardly actions which ‘destroy value’ for the NYT, are substantially motivated to try and influence its behaviour, and was an ultimatum to boot (i.e. the only way the NYT can placate this ‘online mob’ is to fully concede on not using Scott’s legal name). Yet presumably this strategy was not predicated on ‘only we are allowed to (or smart enough to) use game theory, so we can expect the NYT to irrationally give in to our threats when they should be ostentatiously doing exactly what we don’t want them to do to demonstrate they won’t be bullied’. For although these actions are ‘threats’, they are warnings/ good faith/ non-spiteful, as these responses are not just out of hope to coerce: these people would be minded to retaliate similarly if they only found out NYT’s intention after the article had been published. Naturally the hope is that one can resolve conflict by a meeting of the minds: we might hope we can convince Alice to see things our way; and the NYT probably hopes the same. But if the disagreement prompting conflict remains, we should be cautious about how we use the word threat, especially in equivocating between commonsense use of the term (e.g. “I threaten to castigate Charlie publicly if she holds a conference on holocaust denial”) and the subspecies where folk game theory—and our own self-righteousness—strongly urges us to refute (e.g. “Life would be easier for us at the NYT if we acquiesced to those threatening to harm our reputation and livelihoods if we report things they don’t want us to. But we will never surrender the integrity of our journalism to bullies and blackmailers.”)
Another case where ‘precommitment to refute all threats’ is an unwise strategy (and a case more relevant to the discussion, as I don’t think all opponents to hosting a speaker like Hanson either see themselves or should be seen as bullies attempting coercion) is where your opponent is trying to warn you rather than trying to blackmail you. (cf. 1, 2)Suppose Alice sincerely believes some of Bob’s writing is unapologetically misogynistic. She believes it is important one does not give misogynists a platform and implicit approbation. Thus she finds hosting Bob abhorrent, and is dismayed that a group at her university is planning to do just this. She approaches this group, making clear her objections and stating her intention to, if this goes ahead, to (e.g.) protest this event, stridently criticise the group in the student paper for hosting him, petition the university to withdraw affiliation, and so on. This could be an attempt to bully (where usual game theory provides a good reason to refuse to concede anything on principle). But it also could not be: Alice may be explaining what responses she would make to protect her interests which the groups planned action would harm, and hoping to find a better negotiated agreement for her and the EA group besides “They do X and I do Y”. It can be hard to tell the difference, but some elements in this example speak against Alice being a bully wanting to blackmail the group to get her way: First is the plausibility of her interests recommending these actions to her even if they had no deterrent effect whatsoever (i.e. she’d do the same if the event had already happened). Second the actions she intends falls roughly falls in ‘fair game’ of how one can retaliate against those doing something they’re allowed to do which you deem to be wrong. Alice is still not a bully even if her motivating beliefs re. Bob are both completely mistaken and unreasonable. She’s also still not a bully even if Alice’s implied second-order norms are wrong (e.g. maybe the public square would be better off if people didn’t stridently object to hosting speakers based on their supposed views on topics they are not speaking upon, etc.) Conflict is typically easy to navigate when you can dictate to your opponent what their interests should be and what they can license themselves to do. Alas such cases are rare.It is extremely important not to respond to Alice as if she was a bully if in fact she is not, for two reasons. First, if she is acting in good faith, pre-committing to refuse any compromise for ‘do not give in to bullying’ reasons means one always ends up at ones respective BATNAs even if there was mutually beneficial compromises to be struck. Maybe there is no good compromise with Alice this time, but there may be the next time one finds oneself at cross-purposes.Second, wrongly presuming bad faith for Alice seems apt to induce her to make a symmetrical mistake presuming bad faith for you. To Alice, malice explains well why you were unwilling to even contemplate compromise, why you considered yourself obliged out of principle to persist with actions that harm her interests, and why you call her desire to combat misogyny bullying and blackmail. If Alice also thinks about these things through the lens of game theory (although perhaps not in the most sophisticated way), she may reason she is rationally obliged to retaliate against you (even spitefully) to deter you from doing harm again. The stage is set for continued escalation. Presumptive bad faith is pernicious, and can easily lead to martyring oneself needlessly on the wrong hill. I also note that ‘leaning into righteous anger’ or ‘take oneself as justified in thinking the worst of those opposed to you’ are not widely recognised as promising approaches in conflict resolution, bargaining, or negotiation.
This isn’t much more than a rotation (or maybe just a rephrasing), but:When I offer a 10 second or less description of Effective Altruism, it is hard avoid making it sound platitudinous. Things like “using evidence and reason to do the most good”, or “trying to find the best things to do, then doing them” are things I can imagine the typical person nodding along with, but then wondering what the fuss is about (“Sure, I’m also a fan of doing more good rather than less good—aren’t we all?”) I feel I need to elaborate with a distinctive example (e.g. “I left clinical practice because I did some amateur health econ on how much good a doctor does, and thought I could make a greater contribution elsewhere”) for someone to get a good sense of what I am driving at.I think a related problem is the ‘thin’ version of EA can seem slippery when engaging with those who object to it. “If indeed intervention Y was the best thing to do, we would of course support intervention Y” may (hopefully!) be true, but is seldom the heart of the issue. I take most common objections are not against the principle but the application (I also suspect this may inadvertently annoy an objector, given this reply can paint them as—bizarrely - ‘preferring less good to more good’). My best try at what makes EA distinctive is a summary of what you spell out with spread, identifiability, etc: that there are very large returns to reason for beneficence (maybe ‘deliberation’ instead of ‘reason’, or whatever). I think the typical person does “use reason and evidence to do the most good”, and can be said to be doing some sort of search for the best actions. I think the core of EA (at least the ‘E’ bit) is the appeal that people should do a lot more of this than they would otherwise—as, if they do, their beneficence would tend to accomplish much more.Per OP, motivating this is easier said than done. The best case is for global health, as there is a lot more (common sense) evidence one can point to about some things being a lot better than others, and these object level matters a hypothetical interlocutor is fairly likely to accept also offers support for the ‘returns to reason’ story. For most other cause areas, the motivating reasons are typically controversial, and the (common sense) evidence is scant-to-absent. Perhaps the best moves are here would be pointing to these as salient considerations which plausibly could dramatically change ones priorities, and so exploring to uncover these is better than exploiting after more limited deliberation (but cf. cluelessness).
I’m afraid I’m also not following. Take an extreme case (which is not that extreme given I think ’average number of forecasts per forecaster per question on GJO is 1.something). Alice predicts a year out P(X) = 0.2 and never touches her forecast again, whilst Bob predicts P(X) = 0.3, but decrements proportionately as time elapses. Say X doesn’t happen (and say the right ex ante probability a year out was indeed 0.2). Although Alice > Bob on the initial forecast (and so if we just scored that day she would be better), if we carry forward Bob overtakes her overall [I haven’t checked the maths for this example, but we can tweak initial forecasts so he does].
As time elapses, Alice’s forecast steadily diverges from the ‘true’ ex ante likelihood, whilst Bob’s converges to it. A similar story applies if new evidence emerges which dramatically changes the probability, if Bob updates on it and Alice doesn’t. This seems roughly consonant with things like the stock-market—trading off month (or more) old prices rather than current prices seems unlikely to go well.