Jobst Heitzig (vodle.it)

Karma: 252

I’m a mathematician working mostly on technical AI safety and a bit on collective decision making, game theory, and formal ethics. I used to work on international coalition formation, and a lot of stuff related to climate change. Here’s my professional profile.

My definition of value :

I have a wide moral circle (including aliens as long as they can enjoy or suffer life)
I have a zero time discount rate, i.e., value the future as much as the present
I am (utility-) risk-averse: I prefer a sure 1 util to a coin toss between 0 and 2 utils
I am (ex post) inequality-averse: I prefer 2 people to each get 1 util for sure to one getting 0 and one getting 2 for sure
I am (ex ante) fairness-seeking: I prefer 2 people getting an expected 1 util to one getting an expected 0 and one getting an expected 2.
Despite all this, I am morally uncertain
Conditional on all of the above, I also value beauty, consistency, simplicity, complexity, and symmetry

Jobst Heitzig (vodle.it) 27 Oct 2025 11:57 UTC
1 point
0 ∶ 0
on: Human Empowerment versus the Longtermist Imperium?
I wonder how to correctly conceptualize the idea of “a net-negative influence on civilization” in view of the fact that the future is highly uncertain and that that uncertainty is a major motivating factor.

E.g., assume at some time point t1, a longtermist’s proposed plan has higher expected longterm value than an alternative plan because the alternative plan takes a major risk. The longtermist’s plan is realized and at some later time point t2 someone points out that the alternative plan would have produced more value between t1 and t2 (tacitly assuming the risk not realizing between t1 and t2 because the realized longterm plan has successfully avoided it).

Would that constitute an example of what these critics would call a “net-negative influence on civilization”? If so, it’s just a fallacy. If not, then what comparison exactly is meant?

More generally: How to plausibly construct a “counterfactual” world in view of large uncertainties? It seems the only valid comparison would not be between the one realization that actually emerged from a certain behavior and one (potentially overly optimistic) realization that might have emerged from an alternative behavior, but between whole ensembles of realizations. This goes similarly for the effects of drug regulation, workplace laws, historic technology bans etc.

Jobst Heitzig (vodle.it) 22 Oct 2025 18:48 UTC
3 points
0 ∶ 0
on: AI Safety Has a Very Particular Worldview
Maybe this is true in the EA branch of AI safety. In the wider community, e.g. as represented by those attending IASEAI in February, I believe this is not a correct assessment. Since I began working on AI safety, I have heard many cautious and uncertainty-aware statements along the line that the things you claim people believe will almost certainly happen are merely likely enough to worry deeply and work on preventing them. I also don’t see that community having an AI-centric worldview – they seem to worry about many other cause areas as well such as inequality, war, pandemics, climate.

Jobst Heitzig (vodle.it) 2 Jul 2025 14:21 UTC
1 point
0 ∶ 5
on: Debate: Depopulation Matters
Depopulation is Bad
It’s kind of obvious to a sustainability scientist that fewer people eat up less of the remaining cake. It’s a no-brainer. Only naive tech optimists can think some magical tech (maybe AI?) will allow us to decouple from resource use...

Jobst Heitzig (vodle.it) 30 Apr 2025 21:00 UTC
5 points
2 ∶ 0
on: Poll: Should people with more forum karma have more powerful votes?
We need to push back on echo bubbles.

Jobst Heitzig (vodle.it) 28 Jan 2025 22:55 UTC
2 points
1 ∶ 0
in reply to: yams’s comment on: The Game Board has been Flipped: Now is a good time to rethink what you’re doing
The author is using “we” in several places and maybe not consistently. Sometimes “we” seems to be them and the readers, or them and the EA community, and sometimes it seems to be “the US”. Now you are also using an “us” without it being clear (at least to me) who that refers to.

Who do you mean by ‘The country with the community of people who have been thinking about this the longest’ and what is your positive evidence for the claim that other communities (e.g., certain national intelligence communities) haven’t thought about that for at least as long?

Jobst Heitzig (vodle.it) 28 Jan 2025 21:53 UTC
15 points
5 ∶ 2
on: The Game Board has been Flipped: Now is a good time to rethink what you’re doing
I’m confused by your seeming “we vs China” viewpoint—who is this “we” that you are talking about?

Jobst Heitzig (vodle.it) 1 May 2024 7:05 UTC
1 point
0 ∶ 0
on: “Open Source AI” is a lie, but it doesn’t have to be
What about EleutherAI?

Jobst Heitzig (vodle.it) 29 Dec 2023 16:57 UTC
4 points
2 ∶ 0
in reply to: Larks’s comment on: The privilege of native English speakers in reaching high-status, influential positions in EA
Still, it might add more effort for the non-native speaker because a native speaker can identify something as jargon more easily. This is only a hypothesis of course, so to make progress in this discussion it might he helpful to review the literature on this.

Jobst Heitzig (vodle.it) 30 Nov 2023 17:47 UTC
1 point
0 ∶ 0
in reply to: technicalities’s comment on: Shallow review of live agendas in alignment & safety
What is OAA? And, more importantly: where now would you put it in your taxonomy?

Jobst Heitzig (vodle.it) 29 Nov 2023 21:46 UTC
1 point
0 ∶ 0
in reply to: technicalities’s comment on: Shallow review of live agendas in alignment & safety
“targeting NNs” sounds like work that takes a certain architecture (NNs) as a given rather than work that aims at actively designing a system.

To be more specific: under the proposed taxonomy, where would a project be sorted that designs agents composed of a Bayesian network as a world model and an aspiration-based probabilistic programming algorithm for planning?

Jobst Heitzig (vodle.it) 29 Nov 2023 16:01 UTC
1 point
0 ∶ 0
on: Shallow review of live agendas in alignment & safety
Where in your taxonomy does the design of AI systems go – what high-level architecture to use (non-modular? modular with a perception model, world-model, evaluation model, planning model etc.?), what type of function approximators to use for the modules (ANNs? Bayesian networks? something else?), what decision theory to base it on, what algorithms to use to learn the different models occurring in these modules (RL? something else?), how to curate training data, etc.?

Jobst Heitzig (vodle.it) 2 Nov 2023 9:30 UTC
4 points
1 ∶ 0
on: Efficacy of AI Activism: Have We Ever Said No?
Small remark regarding your the metric “* 100% minus the probability that the given technological restraint would have occurred without protests” (let’s call the latter probability x): this seems to suggest that given the protests the probability became 100% while before it had been x and that hence the protests raised the probability from x to 100%. But the fact that the event eventually did occur does not mean at all that after the protests it had a probability of 100% of occurring. It could even have had the very same probability of occurring as before the protests, namely x, or even a smaller probability than that, if only x>0.

What you would actually want to compare here is the probability of occurring given no protests (x) and the probability of occurring given protests (which would have to be estimated separately).

In short: your numbers overestimate the influence of protests by an unknown amount.

Jobst Heitzig (vodle.it) 24 Oct 2023 10:09 UTC
2 points
0 ∶ 0
in reply to: Seth Herd’s comment on: My lab’s small AI safety agenda
So we’re converging...

One final comment on your argument about odds: In our algorithms, specifying an allowable aspiration includes specifying a desired probability of success that is sufficiently below 100%. This is exactly to avoid the problem of fulfilling the aspiration becoming an optimization problem through the backdoor.

Jobst Heitzig (vodle.it) 20 Oct 2023 21:55 UTC
2 points
1 ∶ 0
in reply to: Seth Herd’s comment on: My lab’s small AI safety agenda
Dear Seth, thank you again for your opinion. I agree that many instrumental goals such as power would be helpful also for final goals that are not of the type “maximize this or that”. But I have yet to see a formal argument that show that they would actually emerge in a non-maximizing agent just as likely as in a maximizer.

Regarding your other claim, I cannot agree that “mismatched goals is the problem”. First of all, why do you think there is just a single problem, “the” problem? And then, is it helpful to consider something a “problem” that is an unchangeable fact of life? As long as there is more than one human who is potentially affected by an AI system’s actions, and these humans’ goals are not matched with each other (which they usually aren’t), no AI system can have goals matched to all humans affected by it. Unless you want to claim that “having matched goals” is not a transitive relation. So I am quite convinced that the fact that AI systems will have mismatched goals is not a problem we can solve but a fact we have to deal with.

Jobst Heitzig (vodle.it) 20 Oct 2023 13:25 UTC
2 points
0 ∶ 0
in reply to: Seth Herd’s comment on: My lab’s small AI safety agenda
Dear Seth,

if Yonatan meant it the way you interpret it, I would still respond: Where is the evidence that such a reward function exists and guides humans’ behavior? I spoke to several high-ranking scientists from psychology and social psychology who very much doubt this. I suspect that the theory of humans aiming to maximize reward functions might be a non-testable one, and in that sense “non-scientific” – you might believe in it or not. It helps explaining some stuff, but it is also misleading in other respects. I choose not to believe it until I see evidence.

I also don’t agree that optimization is a red herring. It is a true issue, just not the only one, and maybe not the most severe one (if one believes one can separate out the relative severity of several interlinked issues, which I don’t). I do agree that powerful agents are another big issue, whether competent or not. But powerful, competent, and optimizing agents are certainly the most scary kind :-)

Jobst Heitzig (vodle.it) 20 Oct 2023 12:28 UTC
2 points
0 ∶ 1
in reply to: Seth Herd’s comment on: My lab’s small AI safety agenda
Hi Seth, thank you for your thoughts!

I totally agree that it’s just a start, and I hope to have made clear that it is just a start. If it was not sufficiently clear before, I have now added more text making explicit that of course I don’t think that dropping the optimization paradigm is sufficient to make AI safe, just that it is necessary. And because if appears necessary and under-explored, I chose to study it for some time.

I don’t agree with your 2nd point however: If an agent turns 10% of the world into paperclips, we might still have a chance to survive. If it turns everything into paperclips, we don’t.

Regarding the last point:
- Quantilizers are optimizing (namely a certain “regularized” reward function)
- By “surprising amount” you probably mean “surprisingly large amount”? Why is that surprising you then if you agree that they are a “start on taking the points off of the tiger’s teeth”? Given the obvious risks of optimization, I am also surprised by the amount of support non-maximization approaches get: namely, I am surprised how small that support is. To me this just shows how strong the grip of the optimization paradigm on people’s thinking still is :-)
- I believe any concentration of power is too risky, regardless whether in the hands of a superintelligent AI or dumb human. I have now added some text on this as well.

Jobst Heitzig (vodle.it) 13 Sep 2023 5:45 UTC
5 points
0 ∶ 0
in reply to: Aaron Bergman’s comment on: My tentative best guess on how EAs and Rationalists sometimes turn crazy
The “impossible to correlate perfectly” piece is like in AI alignment, where one could also argue that perfect alignment of a reward function to the “true” utility function is impossible.

Indeed, one might even argue that the joint cognition implemented by the EA/rationality/x-risk community as a whole is a form of “artificial” intelligence, let’s call it “EI” and thus we face an “EI alignment” problem. As EA becomes more powerful in the world, we get “ESI” (effective altruism superhuman intelligence) and related risks from misaligned ESI.

The obvious solution in my opinion is the same for AI and EI: don’t maximize, since the metric you might aim to maximize is most likely imperfectly aligned with true utility. Rather satisfice: be ambitious, but not infinitely so. After reaching an ambitious goal, check if your reward function still makes sense before setting the next, more ambitious goal. And have some human users constantly verify your reward function :-)

Jobst Heitzig (vodle.it) 6 Sep 2023 15:48 UTC
1 point
0 ∶ 0
in reply to: Roman Leventov’s comment on: Thought experiment: Trading off risk, intragenerational and intergenerational inequality, and fairness
Would I be right in interpreting your choice for E as a strong form of risk aversion that is stronger than any fairness considerations? Because E is also the only option that does not give everyone the same ex ante chance of belonging to the winners.

Jobst Heitzig (vodle.it) 4 Sep 2023 14:39 UTC
1 point
0 ∶ 0
in reply to: Roman Leventov’s comment on: Thought experiment: Trading off risk, intragenerational and intergenerational inequality, and fairness
Thanks, @Roman, for your remark regarding the interpretation of the coin toss. I have added a PS to the post in reaction.

Jobst Heitzig (vodle.it) 4 Sep 2023 14:27 UTC
1 point
0 ∶ 0
in reply to: Larks’s comment on: Thought experiment: Trading off risk, intragenerational and intergenerational inequality, and fairness
Thank you for this hint, @Larks. Would you be willing to suggest a different way of presenting the table that would have made it easier? For now, I will add some verbal descriptions to the text.