“writing down stylized models of the world and solving for the optimal thing for EAs to do in them”
I think this is one of the most important things we can be doing. Maybe even the most important since it covers such a wide area and so much government policy is so far from optimal.
you just solve for the policy … that maximizes your objective function, whatever that may be.
Societies need many distinct systems: a transport system, a school system, etc. These systems cannot be justified if they are amoral, so they must serve morality. Each system cannot, however, achieve the best moral outcome on its own: If your transport system doesn’t cure cancer, it probably isn’t doing everything you want; if it does cure cancer, it isn’t just a “transport” system...
Unless by policy, you mean “the entirety of what government does”, then yes. But given that you’re going to consider one area at a time, and you’re “only including all the levers between which you’re considering”, you could reach a local optimum rather than a truly ideal end state. The way I like to think about it is “How would a system for prisons (for example) be in the best possible future?” This is not necessarily going to be the system that does the greatest good at the margin when constrained to the domain you’re considering (though they often are). Rather than think about a system maximizing your objective function, it’s better to think of systems as satisfying goals that are aligned with your objective function.
I think this is one of the most important things we can be doing. Maybe even the most important since it covers such a wide area and so much government policy is so far from optimal.
I don’t think that’s right. I’ve written about what it means for a system to do “the optimal thing” and the answer cannot be that a single policy maximizes your objective function:
Unless by policy, you mean “the entirety of what government does”, then yes. But given that you’re going to consider one area at a time, and you’re “only including all the levers between which you’re considering”, you could reach a local optimum rather than a truly ideal end state. The way I like to think about it is “How would a system for prisons (for example) be in the best possible future?” This is not necessarily going to be the system that does the greatest good at the margin when constrained to the domain you’re considering (though they often are). Rather than think about a system maximizing your objective function, it’s better to think of systems as satisfying goals that are aligned with your objective function.
I wonder if we could create an open source library of IAMs for researchers and EAs to use and audit.