Hmm, I have a different take. I think if I tried to predict as many tokens as possible in response to a particular question, I would say all the words that I could guess someone who knew the answer would say, and then just blank out the actual answer because I couldn’t predict it.
Ah, you want to know about the Riemann hypothesis? Yes, I can explain to you what this hypothesis is, because I know it well. Wise of you ask me in particular, because you certainly wouldn’t ask anyone you knew didn’t have a clue. I will state its precise definition as follows:
~Kittens on the rooftop they sang nya nya nya.~
And that, you see, is what the hypothesis that Riemann hypothesised.
I’m not very good at even pretending to pretend to know what it is, so even if you blanked out the middle, you could still guess I was making it up. But if you blank out the substantive parts of GPT’s answer when it’s confabulating, you’ll have a hard time telling whether it knows the answer or not. It’s just good at what it does.
Hmm, I have a different take. I think if I tried to predict as many tokens as possible in response to a particular question, I would say all the words that I could guess someone who knew the answer would say, and then just blank out the actual answer because I couldn’t predict it.
I’m not very good at even pretending to pretend to know what it is, so even if you blanked out the middle, you could still guess I was making it up. But if you blank out the substantive parts of GPT’s answer when it’s confabulating, you’ll have a hard time telling whether it knows the answer or not. It’s just good at what it does.