From the linked post
A common assumption about AGI is the orthogonality thesis, which argues that goals/utility functions and the core intelligence of an AGI system are orthogonal or can be cleanly factored apart.
This may or may not be an “orthogonality thesis” (I haven’t seen this usage before, but I also haven’t looked for it), but the orthogonality thesis I’m familiar with has the quantifiers the other way around: for any goal, there exists a possible AGI that will attempt carry it out.[1] Even if mixing-and-matching a dumb industrial control system with a smart friendly AI is safe, that doesn’t mean that a smart industrial control system arrived at through some other route won’t paperclip you.
- ^
Though in practice what people making likely-doom arguments really mean is that a generic human-created AI will not have recognizably human values, which is a somewhat stronger claim.
I don’t think this is correct. The illusionist position is that the only actual explananda are our beliefs about phenomenal properties (or rather, the brain-states responsible for utterances like “pain is bad”, since beliefs are another folk-psychological concept supposedly ripe for elimination): once those are accounted for, there will be nothing left to explain and no reason to continue believing in qualia.
For an attack on the reliability of our common-sense beliefs about qualia to constitute a defense of illusionism, we would have to think that our belief in qualia would be unjustified were it not for those intuitions. But this is begging the question—if phenomenal properties do exist and we do have first-hand access to them, then that, not common-sense, is our principal justification for believing in them.