The chain of thought is still generated via feed-forward next token prediction, right?
Yes, it is.. But it still feels different to me.
If itâs possible to create consciousness on a computer at all, then at some level it will have to consist of mechanical operations which canât by themselves be conscious. This is because you could ultimately understand what it is doing as a set of simple instructions being carried out on a processor. So although I canât see how a single forward pass through a neural network could involve consciousness, I donât think a larger system being built out of these operations should rule out that larger system being conscious.
In a non-reasoning model, each token in the output is generated spontaneously, which means I canât see how there could be any conscious deliberation behind it. For example, it canât decide to spend longer thinking about a hard problem than an easier one, in the way a human might. I find it hard to get my head around a conscioussness that canât do that.
In a reasoning model, none of this applies.
(Although itâs true that the distinction probably isnât quite as clear cut as Iâm making out. A non-reasoning model could still decide to use its output to write out âchain of thoughtâ style reasoning, for example.)
Yes, it could well be that an LLM isnât conscious on a single pass, but it becomes conscious across multiple passes.
This is analogous to the Chinese room argument, but I donât take the Chinese room argument as a reductio ad absurdumâunless youâre a substance dualist or a panpsychist, I think you have to believe that a conscious being is made up of parts that are not themselves conscious.
(And even under panpsychism I think you still have to believe that the composed being is conscious in a way that the individual parts arenât? Not sure.)
Yes, it is.. But it still feels different to me.
If itâs possible to create consciousness on a computer at all, then at some level it will have to consist of mechanical operations which canât by themselves be conscious. This is because you could ultimately understand what it is doing as a set of simple instructions being carried out on a processor. So although I canât see how a single forward pass through a neural network could involve consciousness, I donât think a larger system being built out of these operations should rule out that larger system being conscious.
In a non-reasoning model, each token in the output is generated spontaneously, which means I canât see how there could be any conscious deliberation behind it. For example, it canât decide to spend longer thinking about a hard problem than an easier one, in the way a human might. I find it hard to get my head around a conscioussness that canât do that.
In a reasoning model, none of this applies.
(Although itâs true that the distinction probably isnât quite as clear cut as Iâm making out. A non-reasoning model could still decide to use its output to write out âchain of thoughtâ style reasoning, for example.)
Yes, it could well be that an LLM isnât conscious on a single pass, but it becomes conscious across multiple passes.
This is analogous to the Chinese room argument, but I donât take the Chinese room argument as a reductio ad absurdumâunless youâre a substance dualist or a panpsychist, I think you have to believe that a conscious being is made up of parts that are not themselves conscious.
(And even under panpsychism I think you still have to believe that the composed being is conscious in a way that the individual parts arenât? Not sure.)