Jobst Heitzig (vodle.it)

In­ter­gen­er­a­tional equity and in­finite-pop­u­la­tion ethics: a sur­vey by Mar­cus Pi­vato and Marc Fleurbaey

• Jobst Heitzig (vodle.it) 29 Dec 2023 16:57 UTC

  4 points

  2 ∶ 0

  in reply to: Larks’s comment on: The priv­ilege of na­tive English speak­ers in reach­ing high-sta­tus, in­fluen­tial po­si­tions in EA

  Still, it might add more effort for the non-native speaker because a native speaker can identify something as jargon more easily. This is only a hypothesis of course, so to make progress in this discussion it might he helpful to review the literature on this.

• Jobst Heitzig (vodle.it) 30 Nov 2023 17:47 UTC

  1 point

  0 ∶ 0

  in reply to: Gavin’s comment on: Shal­low re­view of live agen­das in al­ign­ment & safety

  What is OAA? And, more importantly: where now would you put it in your taxonomy?

• Jobst Heitzig (vodle.it) 29 Nov 2023 21:46 UTC

  1 point

  0 ∶ 0

  in reply to: Gavin’s comment on: Shal­low re­view of live agen­das in al­ign­ment & safety

  “targeting NNs” sounds like work that takes a certain architecture (NNs) as a given rather than work that aims at actively designing a system.

  To be more specific: under the proposed taxonomy, where would a project be sorted that designs agents composed of a Bayesian network as a world model and an aspiration-based probabilistic programming algorithm for planning?

• Jobst Heitzig (vodle.it) 29 Nov 2023 16:01 UTC

  1 point

  0 ∶ 0

  on: Shal­low re­view of live agen­das in al­ign­ment & safety

  Where in your taxonomy does the design of AI systems go – what high-level architecture to use (non-modular? modular with a perception model, world-model, evaluation model, planning model etc.?), what type of function approximators to use for the modules (ANNs? Bayesian networks? something else?), what decision theory to base it on, what algorithms to use to learn the different models occurring in these modules (RL? something else?), how to curate training data, etc.?

• Jobst Heitzig (vodle.it) 2 Nov 2023 9:30 UTC

  4 points

  1 ∶ 0

  on: Effi­cacy of AI Ac­tivism: Have We Ever Said No?

  Small remark regarding your the metric “* 100% minus the probability that the given technological restraint would have occurred without protests” (let’s call the latter probability x): this seems to suggest that given the protests the probability became 100% while before it had been x and that hence the protests raised the probability from x to 100%. But the fact that the event eventually did occur does not mean at all that after the protests it had a probability of 100% of occurring. It could even have had the very same probability of occurring as before the protests, namely x, or even a smaller probability than that, if only x>0.

  What you would actually want to compare here is the probability of occurring given no protests (x) and the probability of occurring given protests (which would have to be estimated separately).

  In short: your numbers overestimate the influence of protests by an unknown amount.

• Jobst Heitzig (vodle.it) 24 Oct 2023 10:09 UTC

  1 point

  0 ∶ 0

  in reply to: Seth Herd’s comment on: My lab’s small AI safety agenda

  So we’re converging...

  One final comment on your argument about odds: In our algorithms, specifying an allowable aspiration includes specifying a desired probability of success that is sufficiently below 100%. This is exactly to avoid the problem of fulfilling the aspiration becoming an optimization problem through the backdoor.

• Jobst Heitzig (vodle.it) 20 Oct 2023 21:55 UTC

  1 point

  0 ∶ 0

  in reply to: Seth Herd’s comment on: My lab’s small AI safety agenda

  Dear Seth, thank you again for your opinion. I agree that many instrumental goals such as power would be helpful also for final goals that are not of the type “maximize this or that”. But I have yet to see a formal argument that show that they would actually emerge in a non-maximizing agent just as likely as in a maximizer.

  Regarding your other claim, I cannot agree that “mismatched goals is the problem”. First of all, why do you think there is just a single problem, “the” problem? And then, is it helpful to consider something a “problem” that is an unchangeable fact of life? As long as there is more than one human who is potentially affected by an AI system’s actions, and these humans’ goals are not matched with each other (which they usually aren’t), no AI system can have goals matched to all humans affected by it. Unless you want to claim that “having matched goals” is not a transitive relation. So I am quite convinced that the fact that AI systems will have mismatched goals is not a problem we can solve but a fact we have to deal with.

• Jobst Heitzig (vodle.it) 20 Oct 2023 13:25 UTC

  1 point

  0 ∶ 0

  in reply to: Seth Herd’s comment on: My lab’s small AI safety agenda

  Dear Seth,

  if Yonatan meant it the way you interpret it, I would still respond: Where is the evidence that such a reward function exists and guides humans’ behavior? I spoke to several high-ranking scientists from psychology and social psychology who very much doubt this. I suspect that the theory of humans aiming to maximize reward functions might be a non-testable one, and in that sense “non-scientific” – you might believe in it or not. It helps explaining some stuff, but it is also misleading in other respects. I choose not to believe it until I see evidence.

  I also don’t agree that optimization is a red herring. It is a true issue, just not the only one, and maybe not the most severe one (if one believes one can separate out the relative severity of several interlinked issues, which I don’t). I do agree that powerful agents are another big issue, whether competent or not. But powerful, competent, and optimizing agents are certainly the most scary kind :-)

• Jobst Heitzig (vodle.it) 20 Oct 2023 12:28 UTC

  1 point

  0 ∶ 0

  in reply to: Seth Herd’s comment on: My lab’s small AI safety agenda

  Hi Seth, thank you for your thoughts!

  I totally agree that it’s just a start, and I hope to have made clear that it is just a start. If it was not sufficiently clear before, I have now added more text making explicit that of course I don’t think that dropping the optimization paradigm is sufficient to make AI safe, just that it is necessary. And because if appears necessary and under-explored, I chose to study it for some time.

  I don’t agree with your 2nd point however: If an agent turns 10% of the world into paperclips, we might still have a chance to survive. If it turns everything into paperclips, we don’t.

  Regarding the last point:

  • Quantilizers are optimizing (namely a certain “regularized” reward function)

  • By “surprising amount” you probably mean “surprisingly large amount”? Why is that surprising you then if you agree that they are a “start on taking the points off of the tiger’s teeth”? Given the obvious risks of optimization, I am also surprised by the amount of support non-maximization approaches get: namely, I am surprised how small that support is. To me this just shows how strong the grip of the optimization paradigm on people’s thinking still is :-)

  • I believe any concentration of power is too risky, regardless whether in the hands of a superintelligent AI or dumb human. I have now added some text on this as well.

• Jobst Heitzig (vodle.it) 13 Sep 2023 5:45 UTC

  2 points

  0 ∶ 0

  in reply to: Aaron Bergman’s comment on: My ten­ta­tive best guess on how EAs and Ra­tion­al­ists some­times turn crazy

  The “impossible to correlate perfectly” piece is like in AI alignment, where one could also argue that perfect alignment of a reward function to the “true” utility function is impossible.

  Indeed, one might even argue that the joint cognition implemented by the EA/​rationality/​x-risk community as a whole is a form of “artificial” intelligence, let’s call it “EI” and thus we face an “EI alignment” problem. As EA becomes more powerful in the world, we get “ESI” (effective altruism superhuman intelligence) and related risks from misaligned ESI.

  The obvious solution in my opinion is the same for AI and EI: don’t maximize, since the metric you might aim to maximize is most likely imperfectly aligned with true utility. Rather satisfice: be ambitious, but not infinitely so. After reaching an ambitious goal, check if your reward function still makes sense before setting the next, more ambitious goal. And have some human users constantly verify your reward function :-)

• Jobst Heitzig (vodle.it) 6 Sep 2023 15:48 UTC

  1 point

  0 ∶ 0

  in reply to: Roman Leventov’s comment on: Thought ex­per­i­ment: Trad­ing off risk, in­tra­gen­er­a­tional and in­ter­gen­er­a­tional in­equal­ity, and fairness

  Would I be right in interpreting your choice for E as a strong form of risk aversion that is stronger than any fairness considerations? Because E is also the only option that does not give everyone the same ex ante chance of belonging to the winners.

• Jobst Heitzig (vodle.it) 4 Sep 2023 14:39 UTC

  1 point

  0 ∶ 0

  in reply to: Roman Leventov’s comment on: Thought ex­per­i­ment: Trad­ing off risk, in­tra­gen­er­a­tional and in­ter­gen­er­a­tional in­equal­ity, and fairness

  Thanks, @Roman, for your remark regarding the interpretation of the coin toss. I have added a PS to the post in reaction.

• Jobst Heitzig (vodle.it) 4 Sep 2023 14:27 UTC

  1 point

  0 ∶ 0

  in reply to: Larks’s comment on: Thought ex­per­i­ment: Trad­ing off risk, in­tra­gen­er­a­tional and in­ter­gen­er­a­tional in­equal­ity, and fairness

  Thank you for this hint, @Larks. Would you be willing to suggest a different way of presenting the table that would have made it easier? For now, I will add some verbal descriptions to the text.

Thought ex­per­i­ment: Trad­ing off risk, in­tra­gen­er­a­tional and in­ter­gen­er­a­tional in­equal­ity, and fairness

• Jobst Heitzig (vodle.it) 13 Jul 2023 21:47 UTC

  3 points

  0 ∶ 0

  in reply to: Geoffrey Miller’s comment on: What new psy­chol­ogy re­search could best pro­mote AI safety & al­ign­ment re­search?

  This becomes particularly important in human feedback/​input about “higher-level” or more “abstract” questions, as in OpenAI’s deliberative mini-public /​ citizen assembly idea (https://​​openai.com/​​blog/​​democratic-inputs-to-ai).

• Jobst Heitzig (vodle.it) 13 Jul 2023 19:04 UTC

  3 points

  1 ∶ 0

  on: What new psy­chol­ogy re­search could best pro­mote AI safety & al­ign­ment re­search?

  As long as state-of-the-art alignment attempts by industry involve eliciting human evaluations of actual or hypothetical AI behaviors (e.g. responses a chatbot might give to a prompt, as in RLHF), it seems important to understand the psychological aspects of such human-AI interactions. I plan to do some experiments on what I call collective RLHF myself, more from a social choice perspective (see http://​​amsterdam.vodle.it ), and can imagine collaborating on similar questions.

• Jobst Heitzig (vodle.it) 12 Jul 2023 9:36 UTC

  2 points

  0 ∶ 0

  in reply to: Will Petillo’s comment on: My lab’s small AI safety agenda

  Dear Will,

  thanks for these thoughtful comments. I’m not sure I understand some aspects of what you say correctly, but let me try to make sense of this in the example of Zhuang et al., http://​​arxiv.org/​​abs/​​2102.03896. If the utility function is defined only in terms of a proper subset of the attributes, it will exploit the seemingly irrelevant remaining attributes in the optimization, whether or not some of the attributes it uses represent conflicting goals. Even when conflicting goals are “present across all dimensions of the agent’s utility function”, that utility function might simple ignore relevant side-effects, e.g. because the designers and teachers have not anticipated them at all.

  Their example in Fig. 2 shows this nicely. In contrast, with a satisficing goal of achieving only, say, 6 in Fig. 2, the agent will not exploit the unrepresented features as much and actual utility will be much larger.

• Jobst Heitzig (vodle.it) 20 Jun 2023 11:05 UTC

  2 points

  1 ∶ 0

  in reply to: titotal’s comment on: My lab’s small AI safety agenda

  That depends what you mean by “continuously improving until you reach a limit which is not necessarily the global limit”.

  I guess by “continuously” you probably do not mean “in continuous time” but rather “repeatedly in discrete time steps”? So you imagine a sequence r(s1) < r(s2) < … ? Well, that could converge to anything larger than each of the r(sn). E.g., if r(sn) = 1 − 1/​n, it will converge to 1. (It will of course never “reach” 1 since it will always below 1.) This is completely independent of what the local or global maxima of r are. They can obviously be way larger. For example, if the function is r(s) = s and the sequence is sn = 1 − 1/​n, then r(sn) converges to 1 but the maximum of r is infinity. So, as I said before, unless your sequence of improvements is part of an attempt to find a maximum (that is, part of an optimization process), there is no reason to expect that it will converge to some maximum.

  Btw., this also shows that if you have two competing satisficers whose only goal is to outperform the other and who therefore repeatedly improve their reward to be larger than the other agents’ current reward, this does not imply that their rewards will converge to some maximum reward. They can easily be programmed to avoid this by just outperforming the other by an amount of 2**(-n) in the n-th step, so that their rewards converge to the initial reward plus one, rather than to whatever maximum reward might be possible.

• Jobst Heitzig (vodle.it) 20 Jun 2023 9:37 UTC

  1 point

  0 ∶ 0

  in reply to: titotal’s comment on: My lab’s small AI safety agenda

  I agree.

  Except for one detail: Humans who hold preferences that don’t comply to the axioms cannot necessarily be “dutch-booked” for real. That would require them not only to hold certain preferences but also to always act on those preferences like an automaton, see this nice summary discussion: https://​​plato.stanford.edu/​​entries/​​dutch-book/​​