NunoSempere

• NunoSempere 2 Aug 2021 11:38 UTC

  2 points

  in reply to: Jsevillamol’s comment on: Fore­cast­ing Newslet­ter: July 2021

  Thanks

Fore­cast­ing Newslet­ter: July 2021

• NunoSempere 27 Jul 2021 18:42 UTC

  2 points

  on: Deep­Mind: Gen­er­ally ca­pa­ble agents emerge from open-ended play

  My hot take: This seems like a somewhat big deal to me. It’s what I would have predicted, but that’s scary, given my timelines

  Might be confirmation bias. But is it.

• NunoSempere 23 Jul 2021 10:55 UTC

  2 points

  in reply to: Buck’s comment on: Buck’s Shortform

  But if you already have this coalition value function, you’ve already solved the coordination problem and there’s no reason to actually calculate the Shapley value! If you know how much total value would be produced if everyone worked together, in realistic situations you must also know an optimal allocation of everyone’s effort. And so everyone can just do what that optimal allocation recommended.

  This seems correct

  ---

  A related claim is that the Shapley value is no better than any other solution to the bargaining problem. For example, instead of allocating credit according to the Shapley value, we could allocate credit according to the rule “we give everyone just barely enough credit that it’s worth it for them to participate in the globally optimal plan instead of doing something worse, and then all the leftover credit gets allocated to Buck”, and this would always produce the same real-life decisions as the Shapley value.

  This misses some considerations around cost-efficiency/​prioritization. If you look at your distorted “Buck values”, you come away that Buck is super cost-effective; responsible for a large fraction of the optimal plan using just one salary. If we didn’t have a mechanistic understanding of why that was, trying to get more Buck would become an EA cause area.

  In contrast, if credit was allocated according to Shapley values, we could look at the groups whose Shapley value is the highest, and try to see if they can be scaled.

  ---

  The section about “purely local” Shapley values might be pointing to something, but I don’t quite know what it is, because the example is just Shapley values but missing a term? I don’t know. You also say “by symmetry...”, and then break that symmetry by saying that one of the parts would have been able to create $6,000 in value and the other $0. Needs a crisper example.

  ---

  Re: coordination between people who have different values using SVs, I have some stuff here, but looking back the writting seems too corny.

  ---

  Lastly, to some extent, Shapley values are a reaction to people calculating their impact as their counterfactual impact. This leads to double/​triple counting impact for some organizations/​opportunities, but not others, which makes comparison between them more tricky. Shapley values solve that by allocating impact such that it sums to the total impact & other nice properties. Then someone like OpenPhilanthropy or some EA fund can come and see which groups have the highest Shapley value (perhaps highest Shapley value per unit of money/​ressources) and then try to replicate them/​scale them. People might also make better decisions if they compare Shapley instead of counterfactual values (because Shapley values mostly give a more accurate impression of the impact of a position.)

  So I see the benefits of Shapley values as fixing some common mistakes arising from using counterfactual values. This would make impact accounting slightly better, and coordination slightly better to the extent it relies on impact accounting for prioritization (which tbh might not be much.)

  I’m not sure to what extent I agree with the claim that people are overhyping/​misunderstanding Shapley values. It seems a plausible.

• NunoSempere 22 Jul 2021 23:14 UTC

  17 points

  on: A Se­quence Against Strong Longter­mism

  I think that some of your anti-expected-value beef can be addressed by considering stochastic dominance as a backup decision theory in cases where expected value fails.

  For instance, maybe I think that a donation to ALLFED in expectation leads to more lives saved than a donation to a GiveWell charity. But you could point out that the expected value is undefined, because maybe the future contains infinite amount of both flourishing and suffering. Then donating to ALLFED can still be the superior option if I think that it’s stochastically dominant.

  There are probably also tweaks to make to stochastic dominance, e.g., if you have two “games”,

  • Game 1: Get X expected value in the next K years, then play game 3

  • Game 2: Get Y expected value in the next K years, then play game 3

  • Game 3: Some Pasadena-like game with undefined value

  then one could also have a principle where Game 1 is preferable to Game 2 if X > Y, and this also sidesteps some more expected value problems.

• NunoSempere 22 Jul 2021 22:57 UTC

  2 points

  on: NunoSem­pere’s Shortform

  Notes on: A Sequence Against Strong Longtermism

  Summary for myself. Note: Pretty stream-of-thought.

  Proving too much

  • The set of all possible futures is infinite which somehow breaks some important assumptions longtermists are apparently making.

    • Somehow this fails to actually bother me

  • ...the methodological error of equating made up numbers with real data

    • This seems like a cheap/​unjustified shot. In the world where we can calculate the expected values, it would seems fine to compare (wide, uncertain) speculative interventions with harcore GiveWell data (note that the next step would probably be to get more information, not to stop donating to GiveWell charities)

  • Sometimes, expected utility is undefined (Pasadena game)

    • The Pasadena game also fails to bother me, because the series hasn’t (yet) showed that longtermism bets are “Pasadena-like”

    • (Also, note that you can use stochastic dominance to solve many expected value paradoxes, e.g, to decide between two universes with infinite expected value, or with undefined expected value.)

  • ...mention of E.T. Jaynes

    • Yeah, I’m also a fan of E.T. Jaynes, and I think that this is a cheap shot, not an argument.

  • Subject, Object, Instrument

    • This section seems confused/​bad. In particular, there is a switch from “credences are subjective” to “we should somehow change our credences if this is useful”. No, if one’s best guess is that “the future is vast in size”, then considering that one can change one’s opinions to better attain goals doesn’t make it stop being one’s best guess

  Overall: The core of this section seems to be that expected values are sometimes undefined. I agree, but this doesn’t deter me from trying to do the most good by seeking more speculative/​longtermist interventions. I can use stochastic dominance when expected utility fails me.

  The post also takes issue with the following paragraph from The Case For Strong Longtermism:

  Then, using our figure of one quadrillion lives, the expected good done by Shivani contributing $10,000 to [preventing world domination by a repressive global political regime] would, by the lights of utilitarian axiology, be 100 lives. In contrast, funding for the Against Malaria Foundation, often regarded as the most cost-effective intervention in the area of short-term global health improvements, on average saves one life per $3500. (Nuño: italics and bold from the OP, not from original article)

  I agree that the paragraph just intuitively looks pretty bad, so I looked at the context:

  Now, the argument we are making is ultimately a quantitative one: that the expected impact
  one can have on the long-run future is greater than the expected impact one can have on the
  short run. It’s not true, in general, that options that involve low probabilities of high stakes
  systematically lead to greater expected values than options that involve high probabilities of
  modest payoffs: everything depends on the numbers. (For instance, not all insurance contracts
  are worth buying.) So merely pointing out that one ​might be able to influence the long run, or
  that one can do so to a nonzero extent (in expectation), isn’t enough for our argument. But,
  we will claim, any reasonable set of credences would allow that for at least one of these
  pathways, the expected impact is greater for the long-run.
  
  Suppose, for instance, Shivani thinks there’s a 1% probability of a transition to a world
  government in the next century, and that $1 billion of well-targeted grants — aimed (say) at
  decreasing the chance of great power war, and improving the state of knowledge on optimal
  institutional design — would increase the well-being in an average future life, under the
  world government, by 0.1%, with a 0.1% chance of that effect lasting until the end of
  civilisation, and that the impact of grants in this area is approximately linear with respect to
  the amount of spending. Then, using our figure of one quadrillion lives to come, the expected
  good done by Shivani contributing $10,000 to this goal would, by the lights of a utilitarian
  axiology, be 100 lives. In contrast, funding for Against Malaria Foundation, often regarded
  as the most cost-effective intervention in the area of short-term global health improvements,
  on average saves one life per $3500

  Yeah, this is in the context of a thought experiment. I’d still do this with distributions rather than with point estimates, but ok.

  The Credence Assumption

  • Ok, so the OP wants to argue that expected value theory breaks ⇒ the tool is not useful ⇒ we should abandon credences ⇒ longtermism somehow fails.

    • But I think that “My best guess is that I can do more good with more speculative interventions” is fairly robust to that line of criticism; it doesn’t stop being my best guess just because credences are subjective.

      • E.g., if my best guess is that ALLFED does “more good” (e.g., more lives saved in expectation) than GiveWell charities, pointing out that actually the expected value is undefined (maybe the future contains both infinite amounts of flourishing and suffering) doesn’t necessarily change my conclusion if I still think that donating to ALLFED is stochastically dominant.

  • Cox Theorem requires that probabilities be real numbers

    • The OP doesn’t buy that. Sure, a piano is not going to drop on his head, but he might e.g., make worse decisions on account of being overconfident because he has not been keeping track of his (numerical) predictions and thus suffers from more hindsight bias than someone who kept track.

  • But what alternative do we have?

    • One can use e.g., upper and lower bounds on probabilities instead of real valued numbers: Sure, I do that. Longtermism still doesn’t break.

  • Some thought experiment which looks like The Whispering Earring.

  • Instead of relying on explicit expected value calculations, we should rely on evolutionary approaches

  The Poverty of Longtermism

  • “In 1957, Karl Popper proved it is impossible to predict the future of humanity, but scholars at the Future of Humanity Institute insist on trying anyway”

    • Come on

  • Yeah, this is just fairly bad

  • Lesson of the 20th Century

    • This is going to be an ad-hitlerium, isn’t it

      • No, an ad-failures of communism

        • At this point, I stopped reading.

• NunoSempere 9 Jul 2021 11:07 UTC

  4 points

  on: Shal­low eval­u­a­tions of longter­mist organizations

  (Edited to add Centre for the Study of Existential Risk Four Month Report June—September 2020 to the CSER sources)

• NunoSempere 6 Jul 2021 10:35 UTC

  4 points

  in reply to: kokotajlod’s comment on: Shal­low eval­u­a­tions of longter­mist organizations

  Sure, but it was particularly salient to me in this case because the evaluation was so negative

• NunoSempere 5 Jul 2021 11:03 UTC

  2 points

  in reply to: Question Mark’s comment on: Shal­low eval­u­a­tions of longter­mist organizations

  In what capacity are you asking? I’d be more likely to do so if you were asking as a team member, because the organization right now looks fairly small and I would almost be evaluating individuals.

• NunoSempere 5 Jul 2021 11:01 UTC

  12 points

  in reply to: kokotajlod’s comment on: Shal­low eval­u­a­tions of longter­mist organizations

  So what I specifically meant was: It’s interesting that the current leadership probably thinks that CSER is valuable (e.g., valuable enough to keep working at it, rather than directing their efforts somewhere else, and presumably valuable enough to absorb EA funding and talent). This presents a tricky updating problem, where I should probably average my own impressions from my shallow review with their (probably more informed) perspective. But in the review, I didn’t do that, hence the “unmitigated inside view” label.

• NunoSempere 4 Jul 2021 8:27 UTC

  2 points

  in reply to: Stefan_Schubert’s comment on: What should we call the other prob­lem of clue­less­ness?

  I like “opaqueness” for the reason that it is gradable.

Fore­cast­ing Newslet­ter: June 2021

• NunoSempere 28 Jun 2021 15:17 UTC

  6 points

  in reply to: BrianTan’s comment on: Shal­low eval­u­a­tions of longter­mist organizations

  This is a good question, and in hindsight, something I should have recorded. For the project as a whole, maybe two weeks to a month, but not of full-time work. I don’t remember the times for each organization.

• NunoSempere 26 Jun 2021 22:36 UTC

  12 points

  in reply to: MichaelA’s comment on: Shal­low eval­u­a­tions of longter­mist organizations

  Thanks Michael. Going through your options one by one.

  1. Inform decisions about donations that are each in something like the $10-$5000 dollar range. Not an aim I had, but sure, why not.

  2. Inform decisions about donations/​grants that are each in something like the >$50,000 dollar range. So rather than inform those directly, inform the kind of research that you can either do or buy with money to inform that donation. $50,000 feels a little bit low for commissioning research to make a decision, though (could a $5k to $10k investment in a better version of this post make a $50k donation more than 10-20% better? Plausibly.

     • That said, I’d be curious if any largish donations are changed as a result of this post, and why, and in particular why they didn’t defer to the LTF fund.

  3. Inform decisions about which of these orgs (if any) to work for. Not really for myself, but I’d be happy for people to read this post as part of their decisions. Also, 80,000 hours exists.

  4. Provide feedback to these orgs that causes them to improve. Sure, but not a primary aim.

  5. Provide an accountability mechanism for these orgs that causes them to work harder or smarter so that they look better on such evaluations in future. No, not really.

  6. Just see if this sort of evaluation can be done, learn more about how to do that, and share that meta-level info with the EA public. Yep.

  7. [Something else]. Show the kind of thing that an organization like QURI can do! In particular, you can’t do this kind of thing using software other than foretold (Metaculus is great, but the questions are too ambiguous; getting them approved takes time & in the case of a tournament, money, and for this post I only needed my own predictions (not that you can’t run a tournament on foretold.))

  8. [Something else]. Learn more about the longtermist ecosystem myself

  9. [Something else]. So this was sort of on the edges of this project, but for making large amounts of predictions, one does need a pipeline, and improving that pipeline has been on my mind (and on Ozzie Gooen’s). For instance, creating the 27 predictions one by one would be kind of a pain, so instead I use a Google doc script which feeds them to foretold.

  I also think that 4. and 5. are too strongly worded. To the extent I’m providing feedback, I imagine it’s more of a) of the sanity check variety or b) about how a relatively sane person perceives these organizations. For instance, if I don’t get pushback about it in the comments, I’ll think that its a good idea for the APPGFG to expand, but I doubt it’s something that they themselves haven’t thought about.

• NunoSempere 26 Jun 2021 21:20 UTC

  3 points

  in reply to: Denkenberger’s comment on: Shal­low eval­u­a­tions of longter­mist organizations

  Thanks for considering ALLFED. We try to respond to inquiries quickly. We have looked back, and have not be able to locate any such inquiries. We will be finalizing our 2020 report with financial details soon.

  This is most likely my fault; I think I got confused between allfed.org and allfed.info

  To clarify, the cost of preparation does not include the scale up in a catastrophe

  For clarity:
  1. Your guesstimate model: 3% to 50% mitigation of the impact of war with a 30M to 200M, a war which has a probability 0.02% to 5% per year. You also say that so far, you’ve already mitigated the impact of such a war by 1% to 20%.
  2. My model: a 0% to 15% mitigation (previously 0 to 5%, see below) of the impact of such a war with a 50M to 50B investment, where this is maybe not being fully prepared, but does include some serious paranoid preparation, some factories running, supply chains established, etc.
  3. Objection: You’re planning to go with the 30M to 200M path; my estimates should be for that path.
  4. Answer: I’d have to think about it. Maybe 2x to 10x lower. Essentially I’d expect any preparation to at least fail partially, fail to get implemented, be ignored, not survive in institutional memory, etc.

  In this paper, we found that if there were no resilient foods, expenditure on stored foods in a catastrophe would be approximately $90 trillion and about 10% of people would survive. However, if resilient foods could be produced at $2.5 per dry kilogram retail, 97% of people would survive but the total expenditure would only be ~$20 trillion. So one could argue that resilient foods would actually save money in a catastrophe

  I’ll read the paper.

  Just to make sure we are on the same page, if there were a 10% probability of full-scale nuclear war in the next 30 years and there were a 10% reduction in the long-term future potential of humanity given nuclear war, and if planning and R&D for resilient foods mitigated the far future impact of nuclear war by 50%, then that would improve the long-term potential of humanity by 0.5 percentage points (the product of the three percentages).

  I see, thanks, I think I was getting this wrong (I’ve changed this in the guesstimate, but not in the post). With that in mind, your estimates now seem less high (but still very high). It changes my estimates slightly.

  Separately, your numbers still seem fairly high. Suppose that in 1980 you had $100M and knew that there was going to be a pandemic (or another global financial crisis) in the next 100 years, but didn’t knew the details; it seems unlikely that you could have made the covid pandemic or the 2008 financial crisis more than 10% better.

• NunoSempere 26 Jun 2021 20:01 UTC

  4 points

  in reply to: MichaelA’s comment on: Shal­low eval­u­a­tions of longter­mist organizations

  Thanks Michael, beautiful comment.

Shal­low eval­u­a­tions of longter­mist organizations

• NunoSempere 22 Jun 2021 18:02 UTC

  0 points

  on: Event-driven mis­sion hedg­ing and the 2020 US election

  I think you’d have to think about the market equilibrium here. So for instance, if the price of capturing a tCO2e falls to $0.1/​tonne, then more people will want to buy them, and the impact of a marginal tonne captured [1] might be lower. More generally, more people would be doing climate related projects, because the administration would be more welcoming of them.

  In contrast, under $1/​tonne, less people might want to buy them, and thus the marginal impact of a tonne captured might be higher. Similarly, perhaps fewer people would choose to carry out climate related projects, so the ones that exist might be more valuable.

  Would it be 10x higher? No, probably not, but then again a presidency probably wouldn’t make carbon capture 10x more cost-effective (maybe 1.2x? I’m just shooting from the hip here).

  [1]: or, closer to my own heart, the Shapley value of each tonne captured.

• NunoSempere 19 Jun 2021 7:51 UTC

  4 points

  in reply to: jacobjacob’s comment on: 2018-2019 Long Term Fu­ture Fund Gran­tees: How did they do?

  I would also want more bins than the ones I provide, i.e., not considering the total value is probably one of the parts I like less about this post.

• NunoSempere 19 Jun 2021 7:49 UTC

  4 points

  in reply to: Max_Daniel’s comment on: 2018-2019 Long Term Fu­ture Fund Gran­tees: How did they do?

  Makes sense. In particular, noticing that grants are all particularly legible might lead you update in the direction of a truncated distribution like you consider. So far, the LTFF seems like it has maybe moved a bit in the direction of more legibility, but not that much.