But even if we conservatively say that it’s a 50% chance that he was using act utilitarianism as his decision procedure, that’s enough to consider it compromised, because it could lead to bad consequences.
I don’t understand this argument at all. I assume nobody thought it was literally impossible for the implementation of a moral theory (any moral theory!) to lead to bad consequences before. Maybe I’d understand your point more if you stated it quantitatively. Like:
“Previously, I thought it was x% likely that a random act utilitarian would be led by their philosophy to do worse stuff than if they’d endorsed most other moral theories. After seeing the case of SBF, I now think the probability is y% instead, because our sample size is small enough that a single data point can be a large update.”
Looks like Eliezer was similarly confused by your phrasing; your new argument (“almost no multibillion dollar frauds have ever happened, so we should do a very large update about the badness of everything that might have contributed to SBF defrauding people”) sounds very different, and makes more sense to me, though I suspect it won’t end up working.
I don’t understand this argument at all. I assume nobody thought it was literally impossible for the implementation of a moral theory (any moral theory!) to lead to bad consequences before. Maybe I’d understand your point more if you stated it quantitatively. Like:
“Previously, I thought it was x% likely that a random act utilitarian would be led by their philosophy to do worse stuff than if they’d endorsed most other moral theories. After seeing the case of SBF, I now think the probability is y% instead, because our sample size is small enough that a single data point can be a large update.”
Looks like Eliezer was similarly confused by your phrasing; your new argument (“almost no multibillion dollar frauds have ever happened, so we should do a very large update about the badness of everything that might have contributed to SBF defrauding people”) sounds very different, and makes more sense to me, though I suspect it won’t end up working.
I think you’re right—I could have avoided some confusion if I said it could lead to “multi-billion-dollar-level bad consequences”. Edited to clarify.