Jobst Heitzig (vodle.it)

My lab’s small AI safety agenda

An­nounc­ing vo­dle, a web app for con­sen­sus-aiming col­lec­tive decisions

[Question] De­ci­sion mak­ing un­der model am­bi­guity, moral un­cer­tainty, and other agents with free will?

• Jobst Heitzig (vodle.it) 3 Nov 2022 13:31 UTC

  13 points

  2 ∶ 0

  in reply to: Jobst Heitzig (vodle.it)’s comment on: A pro­posed hi­er­ar­chy of longter­mist concepts

  Related to that:

  Your figure says

  Longtermist regress IS EITHER Contraction OR Average wellbeing decrease,

  but consider a certain baseline trajectory A on which

  • longterm population = 3 gazillion person life years for sure

  • average wellbeing = 3 utils per person per life year for sure,

  so that their expected product equals 9 gazillion utils, and an uncertain alternative trajectory B on which

  • if nature’s coin lands heads, longterm population = 7 gazillion person life years but average wellbeing = 1 util per person per life year

  • if nature’s coin lands tails, longterm population = 1 gazillion person life years but average wellbeing = 7 utils per person per life year,

  so that their expected product equals (7 x 1 + 1 x 7) /​ 2 = 7 gazillion utils.

  Then an event that changes the trajectory from A to B is a longtermist regress since it reduces the expected utility.

  But it is NEITHER a contraction NOR an average wellbeing decrease. In fact, it is BOTH an Expansion, since the expected longterm population increases from 3 to 4 gazillion person life years, AND an average wellbeing increase, since that increases from 3 to 4 utils per person per life year.

Thought ex­per­i­ment: Trad­ing off risk, in­tra­gen­er­a­tional and in­ter­gen­er­a­tional in­equal­ity, and fairness

• Jobst Heitzig (vodle.it) 31 Oct 2022 23:49 UTC

  8 points

  0 ∶ 0

  in reply to: Emrik’s comment on: An­nounc­ing vo­dle, a web app for con­sen­sus-aiming col­lec­tive decisions

  Yes:

  • In a vodle poll, each participant gives each option a “wap” (willingness to approve) between 0 and 100.

  • Based on all waps, it is calculated who “approves” which options: if i gave x a wap of r, then i approves x iff less than r% of all participants do not approve x (this coupled system of inequalities has a unique largest solution, which vodle uses).

  • Based on who approves what, the “share” of each option is calculated: x’s share is the fraction of participants who approve x but approve no other option y that is approved by more participants than x. In other words: each participant “owns” an equal share, and a participant’s share goes to the most-approved option among those approved by that participant.

  • If the poll is about distributing a budget, each option gets their share of the budget.

  • If the poll is instead about picking a single winner, the winner is determined by a lottery in which each option’s winning probability equals its share.

  Rationale: By giving suitable waps, participants can make sure their share only goes to a good compromise option if others’ shares also go to that option. One can show that this possibility of “conditional commitment to approve” makes sure that if there is a good compromise, it will win with certainty in a strategic equilibrium between rational participants. Majoritarian methods like Plurality Voting, Approval Voting, STAR Voting, Instant Runoff Voting, Score Voting etc. cannot guarantee this whenever one faction has a majority (if only a slight one) and has thus no incentive whatsoever to seek a compromise with the other participants.

  See the theory paper for details.

In­ter­gen­er­a­tional equity and in­finite-pop­u­la­tion ethics: a sur­vey by Mar­cus Pi­vato and Marc Fleurbaey

• Jobst Heitzig (vodle.it) 3 Nov 2022 13:08 UTC

  7 points

  1 ∶ 1

  on: A pro­posed hi­er­ar­chy of longter­mist concepts

  You write

  expected utility, and its main factors, expected number of people, and expected value per person

  but that is only true if number of people and value per person are stochastically independent, which they probably aren’t, right?

• Jobst Heitzig (vodle.it) 26 Oct 2022 16:45 UTC

  7 points

  0 ∶ 0

  on: You Should Write a Fo­rum Bio

  Hi all, I’m new here and added a profile just now :-)

Fair Col­lec­tive Effec­tive Altruism

• Jobst Heitzig (vodle.it) 25 May 2023 11:31 UTC

  5 points

  1 ∶ 0

  on: On nav­i­gat­ing se­ri­ous philo­soph­i­cal con­fu­sion (Joe Car­l­smith on The 80,000 Hours Pod­cast)

  Joe thinks, in contrast with the dominant theory of correct decision-making, that it’s clear you should send a million dollars to your twin.

  I’m deeply confused about this. According to the premise, you are a deterministic AI system. That means what you will do is fully determined by your code and your input, both of which are already given. So at this point, there is no longer any freedom to many a choice – you will just do what your given code and input determine. So what does it mean to ask what you should do? Does that actually mean: (i) what code should your programmer have written? Or does it mean: (ii) what would the right choice be in the counterfactual situation in which you are not deterministic after all and do have a choice (while your twin doesn’t? or does as well?). In order to answer version (i), we need to know the preferences of the programmer (rather than your own preferences). If the programmer is interested in the joint payoff of both twins, she should have written code that makes you cooperate. In order to answer version (ii), we would need to know what the consequences of making either choice in the counterfactual world where you do have a choice are on the possibility of the other twin to make a choice. If your choice does not influence the possibility of the other twin to make a choice, the dominant strategy is defection, as in the simple PD. Otherwise, who knows...

• Jobst Heitzig (vodle.it) 11 Dec 2022 22:08 UTC

  5 points

  3 ∶ 1

  in reply to: DirectedEvolution’s comment on: De­moc­ra­tiz­ing the work­place as a cause area

  Yes, he is, so what? Does that mean he has to do all the work alone? Surely not. Bob pointed out a neglected area that should be investigated. By who? Probably by those best placed to do this within EA. Are you suggesting Bob is this person?

• Jobst Heitzig (vodle.it) 29 Dec 2023 16:57 UTC

  4 points

  2 ∶ 0

  in reply to: Larks’s comment on: The priv­ilege of na­tive English speak­ers in reach­ing high-sta­tus, in­fluen­tial po­si­tions in EA

  Still, it might add more effort for the non-native speaker because a native speaker can identify something as jargon more easily. This is only a hypothesis of course, so to make progress in this discussion it might he helpful to review the literature on this.

• Jobst Heitzig (vodle.it) 2 Nov 2023 9:30 UTC

  4 points

  1 ∶ 0

  on: Effi­cacy of AI Ac­tivism: Have We Ever Said No?

  Small remark regarding your the metric “* 100% minus the probability that the given technological restraint would have occurred without protests” (let’s call the latter probability x): this seems to suggest that given the protests the probability became 100% while before it had been x and that hence the protests raised the probability from x to 100%. But the fact that the event eventually did occur does not mean at all that after the protests it had a probability of 100% of occurring. It could even have had the very same probability of occurring as before the protests, namely x, or even a smaller probability than that, if only x>0.

  What you would actually want to compare here is the probability of occurring given no protests (x) and the probability of occurring given protests (which would have to be estimated separately).

  In short: your numbers overestimate the influence of protests by an unknown amount.

• Jobst Heitzig (vodle.it) 18 Jun 2023 15:11 UTC

  4 points

  1 ∶ 0

  in reply to: Yonatan Cale’s comment on: My lab’s small AI safety agenda

  Hey Yonatan,

  first, excuse my spelling your name incorrectly originally, I fixed it now.

  Thank you for your encouragement with funding. As it happens, we did apply for funding from several sources and are waiting for their response.

  Regarding Rob Miles’ videos on satisficing:

  One potential misunderstanding relates to the question of with what probability the agent is required to reach a certain goal. If I understand him correctly, he assumes satisficing needs to imply maximizing the probability that some constraint is met, which would still constitute a form of optimization (namely of the probability). This is why our approach is different: In a Markov Decision Process, the client would for example specify a feasibility interval for the expected value of the return (= long-term discounted sum of rewards according to some reward function that we explicitly do not assume to be a proper measure of utility), and the learning algorithm would seek a policy that makes the expected return fall anywhere into this interval.

  The question of whether an agent somehow necessarily must optimize something is a little philosophical in my view. Of course, given an agent’s behavior, one can always find some function that is maximal for the given behavior. This is a mathematical triviality. But this is not the problem we need to address here. The problem we need to address is that the behavior of the agent might get chosen by the agent or its learning algorithm by maximizing some objective function.

  It is all about a paradigm shift: In my view, AI systems should be made to achieve reasonable goals that are well-specified w.r.t. one or more proxy metrics, not to maximize whatever metric. What would be the reasonable goal for your modified paperclip maximizer?

  Regarding “weakness”:

  Non-maximizing does not imply weak, let alone “very weak”. I’m not suggesting to build a very weak system at all. In fact, maximizing an imperfect proxy metric will tend to give low score on the real utility. Or, to turn this around: The maximum of the actual utility function is most achieved by a policy that does not maximize the proxy metric. We will study this in example environments and report results later this year.

[Question] What should my re­search lab fo­cus on in the first week of 2023?

[Question] EA com­mu­nity val­ues and at­ti­tudes sur­vey /​ polls?

• Jobst Heitzig (vodle.it) 10 Apr 2023 20:55 UTC

  3 points

  0 ∶ 0

  in reply to: BrownHairedEevee’s comment on: No­body’s on the ball on AGI alignment

  Not exactly. A typical SLA only contains a lower bound, that would still allow for maximization. The program for a satisficer in the sense I meant it would states that the AL system really aims to do no better than requested. So, for example, quantilizers would not qualify since they might still (by chance) choose that action which maximizes return.

• Jobst Heitzig (vodle.it) 3 Nov 2022 14:09 UTC

  3 points

  0 ∶ 0

  on: Model­ling civil­i­sa­tion af­ter a catastrophe

  I like this type of models very much! As it happens, a few years ago I had a paper in Sustainability that used an even simpler model in a similar spirit, which I used to discuss and compare the implications of various normative systems.

  Abstract: We introduce and analyze a simple formal thought experiment designed to reflect a qualitative decision dilemma humanity might currently face in view of anthropogenic climate change. In this exercise, each generation can choose between two options, either setting humanity on a pathway to certain high wellbeing after one generation of suffering, or leaving the next generation in the same state as the current one with the same options, but facing a continuous risk of permanent collapse. We analyze this abstract setup regarding the question of what the right choice would be both in a rationality-based framework including optimal control, welfare economics, and game theory, and by means of other approaches based on the notions of responsibility, safe operating spaces, and sustainability paradigms. Across these different approaches, we confirm the intuition that a focus on the long-term future makes the first option more attractive while a focus on equality across generations favors the second. Despite this, we generally find a large diversity and disagreement of assessments both between and within these different approaches, suggesting a strong dependence on the choice of the normative framework used. This implies that policy measures selected to achieve targets such as the United Nations Sustainable Development Goals can depend strongly on the normative framework applied and specific care needs to be taken with regard to the choice of such frameworks.

  The model is this:

  

  Here L is the current “lake” state, from which action A (“taking action”) certainly leads to the absorbing “shelter” state S via the passing “valley of tears” state P, while action B (“business as usual”) likely keeps us in L but might also lead to the absorbing “trench” state.

  The names “lake”, “shelter”, “trench” I had introduced earlier in another paper on sustainable management that contained this figure to explain that terminology:

  

  (Humanity sits in a boat and can row against slow currents but not up waterfalls. The dark half is ethically undesirable, the light half desirable. We are likely in a “lake” or “glade” state now, roughly comparable to your “age of perils”. Your “interstellar” state would be a “shelter”.)

• Jobst Heitzig (vodle.it) 3 Nov 2022 13:04 UTC

  3 points

  0 ∶ 0

  on: Longter­mist ter­minol­ogy has bi­as­ing assumptions

  Thank you so much, this is especially helpful for a newbie like me who was a little confused about these things and now thinks that is mainly to do with terminology.