When we, as humans, consider whether or not it makes sense to break the rules for our own benefit, we are running on corrupted hardware: we are very good at justifying to ourselves that seizing money and power for own benefit is really for the good of everyone. If I found myself in a situation where it seemed to me like seizing power for myself was net good, I would worry that in fact I was fooling myself—and even if I was pretty sure I wasn’t fooling myself, I would still worry that I was falling prey to the unilateralist’s curse if it wasn’t very clearly a good idea to others as well.
Also, I would encourage people to read Elephant in the Brain, which backs up this paragraph.
Also, Goodhart’s law would appear as soon as you actually try to optimize for seeming good, when it’s not actually a good thing.
Also, I would encourage people to read Elephant in the Brain, which backs up this paragraph.
Also, Goodhart’s law would appear as soon as you actually try to optimize for seeming good, when it’s not actually a good thing.