NunoSempere

• NunoSempere 4 Apr 2021 18:51 UTC

  2 points

  in reply to: Ben Esche’s comment on: Mun­dane trou­ble with EV /​ utility

  So here is something which sometimes breaks people: You’re saying that you prefer A = 10% chance of saving 10 people over B = 1 in a million chance of saving a billion lives. Do you still prefer a 10% chance of A over a 10% chance of B?

  If you are, note how you can be Dutch-booked.

• NunoSempere 4 Apr 2021 13:12 UTC

  2 points

  in reply to: AppliedDivinityStudies’s comment on: Mun­dane trou­ble with EV /​ utility

  On Pascal’s mugging specifically, Robert Miles has an interesting youtube video arguing that AI Safety is not a Pascal mugging, which the OP might be interested in:

• NunoSempere 3 Apr 2021 8:53 UTC

  10 points

  on: Mun­dane trou­ble with EV /​ utility

  1 & 2 might be normally be answered by the Von Neumann–Morgenstern utility theorem*

  In the case you mentioned, you can try to calculate the impact of an education throughout the beneficiaries’ lives. In this case, I’d expect it to mostly be an increase in future wages, but also some other positive externalities. Then you look at the willingness to trade time for money, or the willingness to trade years of life for money, or the goodness and badness of life at different earning levels, and you come up with a (very uncertain) comparison.

  If you want to look at an example of this, you might want to look at GiveWell’s evaluations in general, or at their evaluation of deworming charities in particular.

  I hope that’s enough to point you to some directions which might answer your questions.

  * But e.g., for negative utilitarians, axiom’s 3 and 3′ wouldn’t apply in general (because they prefer to avoid suffering infinitely more than promoting happiness, i.e. consider L=some suffering, M=non-existence, N=some happiness) but they would still apply for the particular case where they’re trading-off between different quantities of suffering. In any case, even if negative utilitarians would represent the world with two points (total suffering, total happiness), they still have a way of comparing between possible worlds (choose the one with the least suffering, then the one with the most happiness if suffering is equal).

Fore­cast­ing Newslet­ter: March 2021

• NunoSempere 1 Apr 2021 15:38 UTC

  5 points

  on: An­nounc­ing “Nam­ing What We Can”!

  Unsong: The Origins.

• NunoSempere 1 Apr 2021 8:24 UTC

  97 points

  on: New Top EA Causes for 2021?

  This isn’t exactly a proposal for a new cause area, but I’ve felt that the current names of EA organizations are confusingly named. So I’m proposing some name-swaps:

  • Probably Good should now be called “80,000 hours”. Since 80,000 hours explicitly moved towards a more longtermist direction, it has abandoned some of its initial relationship to its name, and Probably Good seems to be picking some of that slack.

  • “80,000 hours should be renamed to “Center for Effective Altruism” (CEA). Although technically a subsidiary, 80,000 hours reaches more people than CEA, and produces more research. This change in name would reflect its de-facto leadership position in the EA community.

  • The Center for Effective Altruism should rebrand to “EA Infrastructure Fund”, per CEA’s strategical focus on events, local groups and the EA forum, and on providing infrastructure for community building more generally.

  • However, this leaves the “EA Infrastructure Fund” without a name. I think the main desiderata for a name is basically prestige, and so I suggest “Future of Humanity Institute”, which sounds suitably ominous. Further, the association with Oxford might lead more applicants to apply, and require a lower salary (since status and monetary compensation are fungible), making the fund more cost-effective.

  • Fortunately, the Global Priorities Institute (GPI) recently determined that helping factory farmed animals is the most pressing priority, and that we never cared that much about humans in the first place. This leaves a bunch of researchers at the Future of Humanity Institute and at the Global Priorities Institute, which recently disbanded, unemployed, but Animal Charity Evaluators is offering them paid junior researcher positions. To reflect its status as the indisputable global priority, Animal Charity Evaluators should consider changing their name to “Doing Good Better”.

  • To enable this last change and to avoid confusion, Doing Good Better would have to be put out of print.

  I estimate that having better names only has a small or medium impact, but that tractability is sky-high. No comment on neglectedness.

  What do you blokes think?

• NunoSempere 31 Mar 2021 20:18 UTC

  2 points

  on: Re­port on Semi-in­for­ma­tive Pri­ors for AI timelines (Open Philan­thropy)

  Random thought on anthropics:

  • If AGI had been developed early and been highly dangerous, one can’t update on not seeing it

  • Anthropic reasoning might also apply to calculating the base rate of AGI; in the worlds where it existed and was beneficial, one might not be trying to calculate its a priori outside view.

• NunoSempere 29 Mar 2021 17:07 UTC

  11 points

  on: Re­port on Semi-in­for­ma­tive Pri­ors for AI timelines (Open Philan­thropy)

  Some notes on the Laplace prior:

  • On footnote 16, you “For example, the application of Laplace’s law described below implies that there was a 50% chance of AGI being developed in the first year of effort”. But historically, participants in the Dartmouth conference were gloriously optimistic

  “We propose that a 2-month, 10-man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.”

  • When you write “I also find that pr(AGI by 2036) from Laplace’s law is too high,” what outside-view consideration are you basing that on? Also, is it really too high?

    • If you rule out AGI until 2028 (as you do in your report), the Laplace prior gives you 1 - (1-[1/​(2028-1956)+1])^(2036-2028) ≈ 10.4% ≈ 10%, which is well withing your range of 1% to 18%, and really near to your estimate of 8%.

    • The point that Laplace’s prior depends on the unit of time chosen is really interesting, but it ends up not mattering once a bit of time has passed. For example, if we choose to use days instead of years, with (days since June 18 1956=23660, days until Mar 29 2028=2557, days until Jan 1 2036=5391), then Laplace’s rule would give for the probability of AGI until 2036: 1 - (1-[1/​(23660+2557+1)])^(5391-2557) = 10.2% ≈ 10%, pretty much the same as above.

      • It’s fun to see that (1-(1/​x))^x converges to 1/​e pretty quickly, and that changing from years to days is equivalent to changing from ~(1+(1/​x))^(x*r) to ~(1+(1/​(365*x)))^(365*x*r) , where x is the time passed in years and x*r is the time remaining in years. But both converge pretty quickly to (1/​e)^r.

  • It is not clear to me that by adjusting the Laplace prior down when you categorize AGI as a “highly ambitious but feasible technology” you are not updating twice: Once on the actual passage of time and another time given that AGI seems “highly ambitious”. But one knows that AGI is “highly ambitious” because it has hasn’t been solved in the first 65 years.

  Given that, I’d still be tempted to go with the Laplace prior for this question, though I haven’t really digested the report yet.

• NunoSempere 23 Mar 2021 16:19 UTC

  8 points

  in reply to: RomeoStevens’s comment on: Don’t Be Bycatch

  Nemo day, perhaps

• NunoSempere 23 Mar 2021 16:18 UTC

  6 points

  in reply to: HaukeHillebrandt’s comment on: Want to alle­vi­ate de­vel­op­ing world poverty? Alle­vi­ate price risk.​ (2018)

  See also: https://​​en.wikipedia.org/​​wiki/​​Onion_Futures_Act

• NunoSempere 22 Mar 2021 11:54 UTC

  2 points

  in reply to: Hamish Huggard’s comment on: BitBets: A Sim­ple Scor­ing Sys­tem for Fore­caster Training

  The auctioning scheme might not end up being proper, though

• NunoSempere 22 Mar 2021 11:48 UTC

  2 points

  in reply to: Misha_Yagudin’s comment on: Rel­a­tive Im­pact of the First 10 EA Fo­rum Prize Winners

  1. Yes, we agree

  2. No, we don’t agree. I think that Adam did better than other potential donor lottery winners, and so his counterfactual value is higher, and thus his Shapley value is also higher. If all the other donors had been clones of Adam, I agree that you’d just divide by n. Thus, the “In every example here, this will be equivalent to calculating counterfactual value, and dividing by the number of necessary stakeholders” is in fact wrong, and I was implicitly doing both of the following in one step: a. Calculating Shapley values with “evaluators” as one agent and b. thinking of Adam’s impact as a high proportion of the SV of the evaluator round,

  3. The rest of our disagreements hinge on 2., and I agree that judging the evaluator step alone would make more sense.

• NunoSempere 18 Mar 2021 13:09 UTC

  9 points

  on: BitBets: A Sim­ple Scor­ing Sys­tem for Fore­caster Training

  This has beautiful elements.

  I’m also interested in using scoring rules for actual tournaments, so some thoughts on that:

  • This scoring rule incentivizes people to predict on questions for which their credence is closer to the extremes, rather than on questions where their credence is closer to even.

  • The rule is is some ways analogous to an automatic market maker which resets for each participant, which is an interesting idea. You could use a set-up such as this to elicit probabilities from forecasters, and give them points/​money in the process.

  • You could start your bits somewhere other than at ⁵⁰⁄₅₀ (this would be equivalent to starting your automatic market maker somewhere else).

• NunoSempere 18 Mar 2021 12:41 UTC

  2 points

  in reply to: jimrandomh’s comment on: Rel­a­tive Im­pact of the First 10 EA Fo­rum Prize Winners

  Yes, the scale is under construction, and you’re not the first person to mention that the specific research agenda mentioned is overvalued.

• NunoSempere 18 Mar 2021 12:34 UTC

  2 points

  in reply to: MichaelA’s comment on: Rel­a­tive Im­pact of the First 10 EA Fo­rum Prize Winners

  I’m also not sure how to interpret your upper bound itself having a range?

  Upper bound being a range is a mistake, fixed now.

• NunoSempere 18 Mar 2021 12:33 UTC

  4 points

  in reply to: MichaelA’s comment on: Rel­a­tive Im­pact of the First 10 EA Fo­rum Prize Winners

  I think that’s a substantial part of the impact, but that there may be other substantial parts too, such as...

  Yes, those seem like at least somewhat important pathways to impact that I’ve neglected, particularly the first two points. I imagine that could easily lead to a 2x to 3x error (but probably not to a 10x error)

• NunoSempere 18 Mar 2021 12:30 UTC

  2 points

  in reply to: MichaelA’s comment on: Rel­a­tive Im­pact of the First 10 EA Fo­rum Prize Winners

  Yes, I expect the intuitions and for estimation to generalize/​help a great deal with the forecasting step, and though I agree that this might not be intuitively obvious. I understand that estimation and forecasting seem like different categories, but I don’t expect that to be a significant hurdle in practice.

• NunoSempere 18 Mar 2021 12:23 UTC

  4 points

  in reply to: MichaelA’s comment on: Rel­a­tive Im­pact of the First 10 EA Fo­rum Prize Winners

  Oof, no it didn’t, good point.

• NunoSempere 18 Mar 2021 12:22 UTC

  3 points

  in reply to: MichaelA’s comment on: Rel­a­tive Im­pact of the First 10 EA Fo­rum Prize Winners

  I have nothing to disagree about here :)

• NunoSempere 18 Mar 2021 12:20 UTC

  3 points

  in reply to: Misha_Yagudin’s comment on: Rel­a­tive Im­pact of the First 10 EA Fo­rum Prize Winners

  You are understanding correctly that the Shapley value multiplier is responsible for preventing double-counting, but you’re making a mistake when you say that it “implies that shallowly evaluated giving is as impactful as “0 to 0.4″ of in-depth evaluated giving”; the latter doesn’t follow.

  In the two player game, you have Value({}), Value({1}), Value({2}), Value({1,2}), and the Shapley value for player 1 (the funders) is ([Value({1})- Value({})] + [Value({1,2})- Value({2})] )/​2, and the value of player 2 (the donor lottery winner) is ([Value({2})- Value({})] + [Value({1,2})- Value({1})] )/​2

  In this case, I’m taking ([Value({2})- Value({})] to be ~0 for simplicity, so the value of player 2 is [Value({1,2})- Value({1})] )/​2. Note that this is just the counterfactual value divided by a fraction.

  If there were more players, it would be a little bit more complicated, but you’d end up with something similar to [Value({1,2,3})- Value({1,3})] )/​3. Note again this is just the counterfactual value divided by a fraction.

  But now, I don’t know how many players there are, so I just consider [Value({The World})- Value({The world without player 2})] )/​(some estimates of how many players there are).

  And the Shapley value multiplier would be 1/​(some estimates of how many players there are).

  At no point am I assuming that “shallowly evaluated giving is as impactful as 0 to 0.4 of in-depth evaluated giving”; the thing that I’m doing is just allocating value so that the sum of the value of each player is equal to the total value.