Jobst Heitzig (vodle.it)

• Jobst Heitzig (vodle.it) 28 Jan 2025 22:55 UTC

  2 points

  1 ∶ 0

  in reply to: yams’s comment on: The Game Board has been Flipped: Now is a good time to re­think what you’re doing

  The author is using “we” in several places and maybe not consistently. Sometimes “we” seems to be them and the readers, or them and the EA community, and sometimes it seems to be “the US”. Now you are also using an “us” without it being clear (at least to me) who that refers to.

  Who do you mean by ‘The country with the community of people who have been thinking about this the longest’ and what is your positive evidence for the claim that other communities (e.g., certain national intelligence communities) haven’t thought about that for at least as long?

• Jobst Heitzig (vodle.it) 28 Jan 2025 21:53 UTC

  9 points

  2 ∶ 0

  on: The Game Board has been Flipped: Now is a good time to re­think what you’re doing

  I’m confused by your seeming “we vs China” viewpoint—who is this “we” that you are talking about?

Aspira­tion-based, non-max­i­miz­ing AI agent designs

• Jobst Heitzig (vodle.it) 1 May 2024 7:05 UTC

  1 point

  0 ∶ 0

  on: “Open Source AI” is a lie, but it doesn’t have to be

  What about EleutherAI?

In­ter­gen­er­a­tional equity and in­finite-pop­u­la­tion ethics: a sur­vey by Mar­cus Pi­vato and Marc Fleurbaey

• Jobst Heitzig (vodle.it) 29 Dec 2023 16:57 UTC

  4 points

  2 ∶ 0

  in reply to: Larks’s comment on: The priv­ilege of na­tive English speak­ers in reach­ing high-sta­tus, in­fluen­tial po­si­tions in EA

  Still, it might add more effort for the non-native speaker because a native speaker can identify something as jargon more easily. This is only a hypothesis of course, so to make progress in this discussion it might he helpful to review the literature on this.

• Jobst Heitzig (vodle.it) 30 Nov 2023 17:47 UTC

  1 point

  0 ∶ 0

  in reply to: technicalities’s comment on: Shal­low re­view of live agen­das in al­ign­ment & safety

  What is OAA? And, more importantly: where now would you put it in your taxonomy?

• Jobst Heitzig (vodle.it) 29 Nov 2023 21:46 UTC

  1 point

  0 ∶ 0

  in reply to: technicalities’s comment on: Shal­low re­view of live agen­das in al­ign­ment & safety

  “targeting NNs” sounds like work that takes a certain architecture (NNs) as a given rather than work that aims at actively designing a system.

  To be more specific: under the proposed taxonomy, where would a project be sorted that designs agents composed of a Bayesian network as a world model and an aspiration-based probabilistic programming algorithm for planning?

• Jobst Heitzig (vodle.it) 29 Nov 2023 16:01 UTC

  1 point

  0 ∶ 0

  on: Shal­low re­view of live agen­das in al­ign­ment & safety

  Where in your taxonomy does the design of AI systems go – what high-level architecture to use (non-modular? modular with a perception model, world-model, evaluation model, planning model etc.?), what type of function approximators to use for the modules (ANNs? Bayesian networks? something else?), what decision theory to base it on, what algorithms to use to learn the different models occurring in these modules (RL? something else?), how to curate training data, etc.?

• Jobst Heitzig (vodle.it) 2 Nov 2023 9:30 UTC

  4 points

  1 ∶ 0

  on: Effi­cacy of AI Ac­tivism: Have We Ever Said No?

  Small remark regarding your the metric “* 100% minus the probability that the given technological restraint would have occurred without protests” (let’s call the latter probability x): this seems to suggest that given the protests the probability became 100% while before it had been x and that hence the protests raised the probability from x to 100%. But the fact that the event eventually did occur does not mean at all that after the protests it had a probability of 100% of occurring. It could even have had the very same probability of occurring as before the protests, namely x, or even a smaller probability than that, if only x>0.

  What you would actually want to compare here is the probability of occurring given no protests (x) and the probability of occurring given protests (which would have to be estimated separately).

  In short: your numbers overestimate the influence of protests by an unknown amount.

• Jobst Heitzig (vodle.it) 24 Oct 2023 10:09 UTC

  2 points

  0 ∶ 0

  in reply to: Seth Herd’s comment on: My lab’s small AI safety agenda

  So we’re converging...

  One final comment on your argument about odds: In our algorithms, specifying an allowable aspiration includes specifying a desired probability of success that is sufficiently below 100%. This is exactly to avoid the problem of fulfilling the aspiration becoming an optimization problem through the backdoor.

• Jobst Heitzig (vodle.it) 20 Oct 2023 21:55 UTC

  2 points

  1 ∶ 0

  in reply to: Seth Herd’s comment on: My lab’s small AI safety agenda

  Dear Seth, thank you again for your opinion. I agree that many instrumental goals such as power would be helpful also for final goals that are not of the type “maximize this or that”. But I have yet to see a formal argument that show that they would actually emerge in a non-maximizing agent just as likely as in a maximizer.

  Regarding your other claim, I cannot agree that “mismatched goals is the problem”. First of all, why do you think there is just a single problem, “the” problem? And then, is it helpful to consider something a “problem” that is an unchangeable fact of life? As long as there is more than one human who is potentially affected by an AI system’s actions, and these humans’ goals are not matched with each other (which they usually aren’t), no AI system can have goals matched to all humans affected by it. Unless you want to claim that “having matched goals” is not a transitive relation. So I am quite convinced that the fact that AI systems will have mismatched goals is not a problem we can solve but a fact we have to deal with.

• Jobst Heitzig (vodle.it) 20 Oct 2023 13:25 UTC

  2 points

  0 ∶ 0

  in reply to: Seth Herd’s comment on: My lab’s small AI safety agenda

  Dear Seth,

  if Yonatan meant it the way you interpret it, I would still respond: Where is the evidence that such a reward function exists and guides humans’ behavior? I spoke to several high-ranking scientists from psychology and social psychology who very much doubt this. I suspect that the theory of humans aiming to maximize reward functions might be a non-testable one, and in that sense “non-scientific” – you might believe in it or not. It helps explaining some stuff, but it is also misleading in other respects. I choose not to believe it until I see evidence.

  I also don’t agree that optimization is a red herring. It is a true issue, just not the only one, and maybe not the most severe one (if one believes one can separate out the relative severity of several interlinked issues, which I don’t). I do agree that powerful agents are another big issue, whether competent or not. But powerful, competent, and optimizing agents are certainly the most scary kind :-)

• Jobst Heitzig (vodle.it) 20 Oct 2023 12:28 UTC

  2 points

  0 ∶ 1

  in reply to: Seth Herd’s comment on: My lab’s small AI safety agenda

  Hi Seth, thank you for your thoughts!

  I totally agree that it’s just a start, and I hope to have made clear that it is just a start. If it was not sufficiently clear before, I have now added more text making explicit that of course I don’t think that dropping the optimization paradigm is sufficient to make AI safe, just that it is necessary. And because if appears necessary and under-explored, I chose to study it for some time.

  I don’t agree with your 2nd point however: If an agent turns 10% of the world into paperclips, we might still have a chance to survive. If it turns everything into paperclips, we don’t.

  Regarding the last point:

  • Quantilizers are optimizing (namely a certain “regularized” reward function)

  • By “surprising amount” you probably mean “surprisingly large amount”? Why is that surprising you then if you agree that they are a “start on taking the points off of the tiger’s teeth”? Given the obvious risks of optimization, I am also surprised by the amount of support non-maximization approaches get: namely, I am surprised how small that support is. To me this just shows how strong the grip of the optimization paradigm on people’s thinking still is :-)

  • I believe any concentration of power is too risky, regardless whether in the hands of a superintelligent AI or dumb human. I have now added some text on this as well.

• Jobst Heitzig (vodle.it) 13 Sep 2023 5:45 UTC

  2 points

  0 ∶ 0

  in reply to: Aaron Bergman’s comment on: My ten­ta­tive best guess on how EAs and Ra­tion­al­ists some­times turn crazy

  The “impossible to correlate perfectly” piece is like in AI alignment, where one could also argue that perfect alignment of a reward function to the “true” utility function is impossible.

  Indeed, one might even argue that the joint cognition implemented by the EA/​rationality/​x-risk community as a whole is a form of “artificial” intelligence, let’s call it “EI” and thus we face an “EI alignment” problem. As EA becomes more powerful in the world, we get “ESI” (effective altruism superhuman intelligence) and related risks from misaligned ESI.

  The obvious solution in my opinion is the same for AI and EI: don’t maximize, since the metric you might aim to maximize is most likely imperfectly aligned with true utility. Rather satisfice: be ambitious, but not infinitely so. After reaching an ambitious goal, check if your reward function still makes sense before setting the next, more ambitious goal. And have some human users constantly verify your reward function :-)

• Jobst Heitzig (vodle.it) 6 Sep 2023 15:48 UTC

  1 point

  0 ∶ 0

  in reply to: Roman Leventov’s comment on: Thought ex­per­i­ment: Trad­ing off risk, in­tra­gen­er­a­tional and in­ter­gen­er­a­tional in­equal­ity, and fairness

  Would I be right in interpreting your choice for E as a strong form of risk aversion that is stronger than any fairness considerations? Because E is also the only option that does not give everyone the same ex ante chance of belonging to the winners.

• Jobst Heitzig (vodle.it) 4 Sep 2023 14:39 UTC

  1 point

  0 ∶ 0

  in reply to: Roman Leventov’s comment on: Thought ex­per­i­ment: Trad­ing off risk, in­tra­gen­er­a­tional and in­ter­gen­er­a­tional in­equal­ity, and fairness

  Thanks, @Roman, for your remark regarding the interpretation of the coin toss. I have added a PS to the post in reaction.

• Jobst Heitzig (vodle.it) 4 Sep 2023 14:27 UTC

  1 point

  0 ∶ 0

  in reply to: Larks’s comment on: Thought ex­per­i­ment: Trad­ing off risk, in­tra­gen­er­a­tional and in­ter­gen­er­a­tional in­equal­ity, and fairness

  Thank you for this hint, @Larks. Would you be willing to suggest a different way of presenting the table that would have made it easier? For now, I will add some verbal descriptions to the text.

Thought ex­per­i­ment: Trad­ing off risk, in­tra­gen­er­a­tional and in­ter­gen­er­a­tional in­equal­ity, and fairness

• Jobst Heitzig (vodle.it) 13 Jul 2023 21:47 UTC

  3 points

  0 ∶ 0

  in reply to: Geoffrey Miller’s comment on: What new psy­chol­ogy re­search could best pro­mote AI safety & al­ign­ment re­search?

  This becomes particularly important in human feedback/​input about “higher-level” or more “abstract” questions, as in OpenAI’s deliberative mini-public /​ citizen assembly idea (https://​​openai.com/​​blog/​​democratic-inputs-to-ai).