Moloch and the Pareto optimal frontier

Moloch is a po­etic way of de­scribing failures of co­or­di­na­tion and co­her­ence in­side an agent or be­tween agents and the gen­er­a­tion of harm­ful sub­com­po­nents or harm­ful agents. Per­haps this could be de­com­posed fur­ther, or at least par­tially cov­ered, by ran­domly gen­er­ated ac­ci­dents, Good­hart’s law failures, and con­flicts of op­ti­miza­tion. Let’s zoom in on one as­pect, con­flicts of op­ti­miza­tion.

What are con­flicts of op­ti­miza­tion? They are situ­a­tions where more than one crite­rion is be­ing op­ti­mized for and in prac­tice im­prov­ing one crite­rion causes at least one other crite­rion to be­come less op­ti­mal.

When does this oc­cur? It oc­curs when you can­not find a way to im­prove all op­ti­miza­tion crite­ria at the same time. For in­stance, if you can­not pro­duce both more swords and more shields be­cause you only have a limited amount of iron then you have a con­flict of op­ti­miza­tion.

This can be de­scribed by the con­cept of Pareto op­ti­mal­ity. If you’re at a Pareto op­ti­mal point then there is no way to im­prove all crite­ria and the set of all such points is called the Pareto fron­tier.

What this looks like is that if you’re near or on the Pareto fron­tier there are few to zero ways to im­prove all crite­ria and as you get fur­ther away there are more op­tions for im­prov­ing all crite­ria. Iter­a­tively then you can imag­ine that around ev­ery point there are some known ways to move that may be vi­su­al­ized as vec­tors from that point. A se­quence of such changes is then a se­quence of move­ments along vec­tors. Gen­er­ally what you’d ex­pect (with caveats) is that the tra­jec­tory moves up and to the right un­til it hits the pareto op­ti­mal fron­tier and then skates along the fron­tier till one or the other op­ti­miza­tion pro­cess wins or they are at equil­ibrium (re­lat­edly).

An ex­am­ple of this dy­namic is job ne­go­ti­a­tions done well. At first both par­ties are work­ing to­wards find­ing changes that benefit them both but as time goes on such op­por­tu­ni­ties run out and the last parts of the ne­go­ti­a­tion pro­ceed in a zero sum way (like per­haps salary).

In prac­tice the Pareto fron­tier isn’t nec­es­sar­ily static be­cause back­ground vari­ables may be chang­ing in time. As long as the pro­cess of mov­ing to­wards the fron­tier is much faster than the speed at which the fron­tier changes though we’d con­tinue to ex­pect again the mo­tion of go­ing to­wards the fron­tier and then skat­ing along it.

Between the crite­ria this then trans­lates es­sen­tially to a bunch of pos­i­tive sum trans­for­ma­tions far away from the Pareto fron­tier and then as you get closer to it trans­for­ma­tions be­come less and less pos­i­tive sum, un­til fi­nally be­com­ing zero sum (one can only win at the ex­pense of the other los­ing). This has nat­u­ral im­pli­ca­tions about how game the­ory ac­tors re­late as progress oc­curs when go­ing to­wards a Pareto fron­tier.

Let’s now re­late this to Moloch. Let the x axis be op­ti­miz­ing for hu­man­ity’s ul­ti­mate val­ues, the y axis be op­ti­miz­ing for com­pet­i­tive­ness (things like win­ning in poli­tics, wars, per­sua­sion, and profit mak­ing), the points rep­re­sent the world’s state in terms of x and y, and the tra­jec­tory through the points be how the world de­vel­ops. Given the above we’d ex­pect that at first com­pet­i­tive­ness and the ac­com­plish­ment of hu­man­ity’s ul­ti­mate val­ues are both im­proved but even­tu­ally they come apart and the tra­jec­tory skates along the Pareto fron­tier (that roughly speak­ing hap­pens when we are at max­i­mum tech­nol­ogy or tech­nolog­i­cal change be­comes suffi­ciently slow) un­til it max­i­mizes com­pet­i­tive­ness.

This is one of Moloch’s tools: The move­ment to­wards com­pet­i­tive ad­van­tage over achiev­ing hu­man­ity’s ul­ti­mate val­ues be­cause the set of trans­for­ma­tions is con­strained near the Pareto fron­tier.