what makes you think our expected value calculations will ever become good enough, and how will you know if they do?

Agree with this: it seems unclear to me that they’ll become good enough in many cases since our reasoning capabilities are fairly limited and the world is really complicated. I think this point is what Eliezer is trying to communicate with the tweet pictured in the post:

In my field (operations research), which is literally all about using optimization to make ‘optimal’ decisions, one way in which we account for issues like these is with robust optimization (RO).

In RO, you account for uncertainty by assuming unknown parameters (e.g. the weights between different possible objectives) lie within some predefined uncertainty set. You then maximize your objective over the worst case values of the uncertainty (ie, maximin optimization). In this way, you protect yourself against being wrong in a bounded worst-case way. Of course, this punts the problem to choosing a good uncertainty set, but I still think the heuristic “prefer to take actions which are good even under some mistaken assumptions” should be used more often.

The difference is, though, that in robust optimization you don’t really predict the future. You create an algorithm that you control, which uses some randomness or noisy estimates, and you prove that after enough time you can expect it to wind up somewhere. This is different than straight up giving an estimate of what a single choice will lead to, in a complex non-convex system where you have no control over anything else.

I wouldn’t quite say that. In any sort of multi-period optimization you must model the consequences of actions and are (at least implicitly) predicting something about the future.

Regardless I was mostly gesturing at the intuition, I agree this doesn’t solve the problem.

Agree with this: it seems unclear to me that they’ll become good enough in many cases since our reasoning capabilities are fairly limited and the world is really complicated. I think this point is what Eliezer is trying to communicate with the tweet pictured in the post:

In my field (operations research), which is literally all about using optimization to make ‘optimal’ decisions, one way in which we account for issues like these is with robust optimization (RO).

In RO, you account for uncertainty by assuming unknown parameters (e.g. the weights between different possible objectives) lie within some predefined uncertainty set. You then maximize your objective over the worst case values of the uncertainty (ie, maximin optimization). In this way, you protect yourself against being wrong in a bounded worst-case way. Of course, this punts the problem to choosing a good uncertainty set, but I still think the heuristic “prefer to take actions which are good even under some mistaken assumptions” should be used more often.

The difference is, though, that in robust optimization you don’t really predict the future. You create an algorithm that you control, which uses some randomness or noisy estimates, and you prove that after enough time you can expect it to wind up somewhere. This is different than straight up giving an estimate of what a single choice will lead to, in a complex non-convex system where you have no control over anything else.

I wouldn’t quite say that. In any sort of multi-period optimization you must model the consequences of actions and are (at least implicitly) predicting something about the future.

Regardless I was mostly gesturing at the intuition, I agree this doesn’t solve the problem.