This post is great, and I think it frames the idea very well.
My only disagreement is with the following part of the scenario you give:
Every time you try to think things through, the machine will cause you to make mistakes of reasoning that you won’t notice: indeed, you’ve already been making lots of these. You’re hopelessly confused on a basic level, and you’ll stay that way for the rest of your life.
The inclusion of this seems unhelpful to me, because it makes me wonder about the extent to which a version of me whose internal thought processes are systematically manipulated is really the same person (in the sense that I care about). Insofar as the ways I think and reason are part of my personality and identity, then I might have additional reasons to not want them to be changed (in addition to wanting my beliefs to be accurate).
As you identify, it may still be necessary to interfere with my beliefs for the purposes of maintaining social fictions, but this could plausibly require only minor distortions. Whereas losing control of my mind in the way you describe above seems quite different from just having false beliefs.
Thanks! Re: mental manipulation, do you have similar worries even granted that you’ve already been being manipulated in these ways? We can stipulate that there won’t be any increase in the manipulation in question, if you stay. One analogy might be: extreme cognitive biases that you’ve had all along. They just happen to be machine-imposed.
That said, I don’t think this part is strictly necessary for the thought experiment, so I’m fine with folks leaving it out if it trips them up.
Yes, I think I still have these concerns; if I had extreme cognitive biases all along, then I would want them removed even if it didn’t improve my understanding of the world. It feels similar to if you told me that I’d lived my whole life in a (pleasant) dreamlike fog, and I had the opportunity to wake up. Perhaps this is the same instinct that motivates meditation? I’m not sure.
This post is great, and I think it frames the idea very well.
My only disagreement is with the following part of the scenario you give:
The inclusion of this seems unhelpful to me, because it makes me wonder about the extent to which a version of me whose internal thought processes are systematically manipulated is really the same person (in the sense that I care about). Insofar as the ways I think and reason are part of my personality and identity, then I might have additional reasons to not want them to be changed (in addition to wanting my beliefs to be accurate).
As you identify, it may still be necessary to interfere with my beliefs for the purposes of maintaining social fictions, but this could plausibly require only minor distortions. Whereas losing control of my mind in the way you describe above seems quite different from just having false beliefs.
Thanks! Re: mental manipulation, do you have similar worries even granted that you’ve already been being manipulated in these ways? We can stipulate that there won’t be any increase in the manipulation in question, if you stay. One analogy might be: extreme cognitive biases that you’ve had all along. They just happen to be machine-imposed.
That said, I don’t think this part is strictly necessary for the thought experiment, so I’m fine with folks leaving it out if it trips them up.
Yes, I think I still have these concerns; if I had extreme cognitive biases all along, then I would want them removed even if it didn’t improve my understanding of the world. It feels similar to if you told me that I’d lived my whole life in a (pleasant) dreamlike fog, and I had the opportunity to wake up. Perhaps this is the same instinct that motivates meditation? I’m not sure.