Zach Stein-Perlman

• Zach Stein-Perlman 26 Jun 2022 4:01 UTC

  4 points

  on: What are the best re­sources on com­par­ing x-risk pre­ven­tion to im­prov­ing the value of the fu­ture in other ways?

  I would replace “avoiding x-risk” with “avoiding stuff like extinction” in this question. SBF’s usage is nonstandard—an existential catastrophe is typically defined as something that causes us to be able to achieve at most a small fraction of our potential. Events which cause us to achieve only 10^-30 of our potential are an existential catastrophe.

  Regardless, I’m not aware of much thought on how to improve the future conditional on avoiding stuff like extinction (or similar questions, like how to improve the future conditional on achieving aligned superintelligence).

• Zach Stein-Perlman 25 Jun 2022 23:13 UTC

  5 points

  in reply to: Question Mark’s comment on: More fund­ing is re­ally good

  I don’t know of particular estimates. I do know that different (smart, reasonable, well-informed) people would give very different answers—some would even say that the marginal AI safety researcher has negative expected value.

  Personally, I’m optimistic that even if you’re skeptical of AI safety research in general, you can get positive expected value by (as a lower bound) doing something like giving particular researchers whose judgment you trust money to support researchers they think are promising.

  My guess is that the typical AI-concerned community leader would say at least a one in 10 billion chance for $1,000.

• Zach Stein-Perlman 25 Jun 2022 23:13 UTC

  2 points

  in reply to: Larks’s comment on: More fund­ing is re­ally good

  Yeah, I don’t know much about this; if someone has a good justification for the marginal cost-effectiveness of global health & development interventions, I’d love to see it.

More fund­ing is re­ally good

• Zach Stein-Perlman 23 Jun 2022 19:20 UTC

  29 points

  on: The Fo­rum should con­sider anonymiz­ing names

  I find names helpful to decide what to read and to associate ideas with people. If you want to hide names on your end, you can view the forum through https://​​ea.greaterwrong.com and click the anti-kibitzer button on the right side of the page.

• Zach Stein-Perlman 21 Jun 2022 21:41 UTC

  4 points

  in reply to: Michael_Wiebe’s comment on: Longter­mist slo­gans that need to be retired

  Not actively. I buy that doing a few projects with sharper focus and tighter feedback loops can be good for community health & epistemics. I would disagree if it took a significant fraction of funding away from interventions with a more clear path to doing an astronomical amount of good. (I almost added that it doesn’t really feel like lead elimination is competing with more longtermist interventions for FTX funding, but there probably is a tradeoff in reality.)

• Zach Stein-Perlman 21 Jun 2022 20:52 UTC

  6 points

  in reply to: Michael_Wiebe’s comment on: Longter­mist slo­gans that need to be retired

  I mostly agree that obviously great stuff gets funding, but I think the “marginal stuff” is still orders of magnitude better in expectation than almost any neartermist interventions.

• Zach Stein-Perlman 20 Jun 2022 17:00 UTC

  2 points

  in reply to: Zach Stein-Perlman’s comment on: On Defer­ence and Yud­kowsky’s AI Risk Estimates

  I’m still not totally comfortable. I think my confusion arose because I was considering the related question of whether I could use my better knowledge than Eli to win money from bets (in expectation) -- I couldn’t, because Eli has no reason to bet on the bomb going off. More generally, Eliezer never had reason to bet (in the sense that he gets epistemic credit if he’s right) on nanotech-doom-by-2010, because in the worlds where he’s right we’re dead. It feels weird to update against Eliezer on the basis of beliefs that he wouldn’t have bet on; updating against him doesn’t seem to be incentive-compatible… but maybe that’s just the sacrifice immanent to the epistemic virtue of publicly sharing your belief in doom.

• Zach Stein-Perlman 19 Jun 2022 17:35 UTC

  22 points

  in reply to: Derek Shiller’s comment on: On Defer­ence and Yud­kowsky’s AI Risk Estimates

  No, my survival for 12 hours is evidence against Eli being correct about the bomb.

  So: oops, I think.

• Zach Stein-Perlman 19 Jun 2022 16:29 UTC

  2 points

  on: On Defer­ence and Yud­kowsky’s AI Risk Estimates

  Almost all of this seems reasonable. But:

  Yudkowsky has previously held short AI timeline views that turned out to be wrong

  I don’t think we should update based on this, or eg on the fact that we didn’t go extinct due to nanotechnology, because anthropics /​ observer selection. (We should only update based on whether we think the reasons for those beliefs were bad.)

• Zach Stein-Perlman 3 Jun 2022 18:11 UTC

  1 point

  in reply to: Arjun Panickssery’s comment on: Just Say No to Utili­tar­i­anism

  Sometimes intuitions conflict, like if someone intuits

  • I should pull the lever to save five in a trolley problem

  • I shouldn’t kill someone and use their organs to save five others

  • The two scenarios have no morally relevant differences

  So we need principles, and I find the principle “consult your intuition about the specific case” unappealing because I feel more strongly about more abstract intuitions like the third one above (or maybe like universalizability or linearity) than intuitions about specific cases.

• Zach Stein-Perlman 3 Jun 2022 17:57 UTC

  1 point

  in reply to: Arjun Panickssery’s comment on: Just Say No to Utili­tar­i­anism

  Clearly we can’t just throw intuitions out, since then we have nothing left! But I—and Peter Singer in the linked piece—think intuitions should be a guide to help us generate broader principles (and I happen to find utilitarianism appealing as such a principle), and I think intuitions shouldn’t be seen of as data that give us direct access to moral truth and that ethical theories must explain, since considering the source of our intuitions, they’re not directly generated by moral truth.

• Zach Stein-Perlman 3 Jun 2022 17:19 UTC

  3 points

  on: Just Say No to Utili­tar­i­anism

  There’s no totally satisfying answer, I think, to the question of how ethical intuitions should affect what we think is good/​right/​best/​etc. Among other reasons, those of us who prefer principles over myopic intuition-consultation point to the fact that human intuitions seem to turn on irrelevant facts (e.g., different framings of the trolley problem provoke different responses from many humans, even when the differences seem like they shouldn’t be morally relevant). I recommend Peter Singer’s Ethics and Intuitions. (And if it seems too long, I would prioritize section 3.)

• Zach Stein-Perlman 24 May 2022 0:58 UTC

  1 point

  in reply to: theme_arrow’s comment on: St. Peters­burg De­mon – a thought ex­per­i­ment that makes me doubt Longter­mism

  I agree with you that the limit of the EV of “bet until you win n times” is infinite as n→∞. But I agree with Guy Raveh that we probably can’t just take this limit and call it the EV of “always bet.” Maybe it depends on what precise question we’re asking...

  (parent comment edited)

• Zach Stein-Perlman 24 May 2022 0:47 UTC

  1 point

  in reply to: Gabin’s comment on: St. Peters­burg De­mon – a thought ex­per­i­ment that makes me doubt Longter­mism

  [Parent comment deleted, so I integrated this comment into its grandparent]

• Zach Stein-Perlman 23 May 2022 21:55 UTC

  1 point

  in reply to: Alex_shleizer’s comment on: St. Peters­burg De­mon – a thought ex­per­i­ment that makes me doubt Longter­mism

  Well, what value do we assign a scenario in which you’re still talking with the demon? If those get value 0, then sure, the EV is 0. But calling those scenarios 0 seems problematic, I think.

• Zach Stein-Perlman 23 May 2022 21:55 UTC

  1 point

  in reply to: Guy Raveh’s comment on: St. Peters­burg De­mon – a thought ex­per­i­ment that makes me doubt Longter­mism

  The expected value is, in fact, defined, and it is zero.

  Is the random variable you’re thinking of, whose expectation is zero, just the random variable that’s uniformly zero? That doesn’t seem to me to be the right way to describe the “bet” strategy; I would prefer to say the random variable is undefined. (But calling it zero certainly doesn’t seem to be a crazy convention.)

• Zach Stein-Perlman 23 May 2022 18:34 UTC

  4 points

  in reply to: AABoyles’s comment on: AABoyles’s Shortform

  If there’s at least a 1% chance that we don’t experience catastrophe soon, and we can have reasonable expected influence over no-catastrophe-soon futures, and there’s a reasonable chance that such futures have astronomical importance, then patient philanthropy is quite good in expectation. Given my empirical beliefs, it’s much better then GiveDirectly. And that’s just a lower bound; e.g., investing in movement-building might well be even better.

• Zach Stein-Perlman 23 May 2022 13:51 UTC

  5 points

  in reply to: Jeff Kaufman’s comment on: St. Peters­burg De­mon – a thought ex­per­i­ment that makes me doubt Longter­mism

  Short answer: yes.

  Medium answer: observe that the value after winning n bets is at least 10×2^n, and the limit of (215/​216)^n × 10×2^n as n→∞ is undefined (infinite). Or let ω be an infinite number and observe that the infinitesimal probability (215/​216)^ω times the infinite value 10×2^ω is 10×(2×215/​216)^ω which is infinite. (Note also that the EVs of the strategy “bet with probability p” goes to ∞ as p→1.)

  Edit: hmm, in response to comments, I’d rephrase as follows.

  Yes, the “always bet” strategy has value 0 with probability 1. But if a random variable is 0 with probability measure 1 and is undefined with probability measure 0, we can’t just say it’s identical to the zero random variable or that it has expected value zero (I think, happy to be corrected with a link to a math source). And while ‘infinite bets’ doesn’t really make sense, if we have to think of ‘the state after infinite bets’ I think we could describe its value with the aforementioned random variable.

  While you will never actually create a world with this strategy, I don’t think the expected value is defined because ‘after infinite bets’ you could still be talking to the demon (with probability 0, but still possibly, and talking-with-the-demon-after-winning-infinitely-many-bets seems significant even with probability 0).

• Zach Stein-Perlman 23 May 2022 13:23 UTC

  5 points

  on: St. Peters­burg De­mon – a thought ex­per­i­ment that makes me doubt Longter­mism

  • Some academic papers including A paradox for tiny probabilities and enormous values would be of interest to you.

  You could say, that the strategy “always take the demons bet” has an expected value of 0 QUALYs

  • This is not true. The expected value of this strategy is undefined. [Edit: commenters reasonably disagree.]

  • So maybe we want to keep our normal expected-utility-maximizing behavior for some nicely-behaved prospects, where “nicely-behaved” includes conditions like “expected values are defined,” and accept that we might have to throw up our hands otherwise.

  • That said, I agree that thought experiments like this give us at least some reason to disfavor simple expected-utility-maximizing, but I caution against the jump to expected-utility-maximizing is wrong since (in my opinion) other (non-nihilistic) theories are even worse. It’s easy to criticize a particular theory; if you offered a particular alternative, there would be strong reasons against that theory.

  • Regardless, note that you can have a bounded utility function and be a longtermist, depending on the specifics of the theory. And it’s prima facie the case that (say) x-risk reduction is very good even if you only care about (say) the next billion years.

  • (QUALY should be QALY)