I continue to have a lot of uncertainty about how likely it is that AI development will look like “there’s this separate project of trying to figure out what goals to give these AI systems” vs a development process where capability and goals are necessarily connected. (I didn’t find your arguments in favor of the latter very persuasive.) For example it seems GPT-3 can be seen as more like the former than the latter. (See this thread for background on this.)
I don’t think I caught the point about GPT-3, although this might just be a matter of using concepts differently.
In my mind: To whatever extent GPT-3 can be said to have a “goal,” its goal is to produce text that it would be unsurprising to find on the internet. The training process both imbued it with this goal and made the system good at achieving it.
There are other things we might want spin-offs of GPT-3 to do: For example, compose better-than-human novels. Doing this would involve shifting both what GPT-3 is “capable” of doing and shifting what its “goal” is. (There’s not really a clean practical or conceptual distinction between the two.) It would also probably require making progress on some sort of “alignment” technique, since we can’t (e.g.) write down a hand-coded reward function that quantifies novel quality.
I don’t think I caught the point about GPT-3, although this might just be a matter of using concepts differently.
In my mind: To whatever extent GPT-3 can be said to have a “goal,” its goal is to produce text that it would be unsurprising to find on the internet. The training process both imbued it with this goal and made the system good at achieving it.
There are other things we might want spin-offs of GPT-3 to do: For example, compose better-than-human novels. Doing this would involve shifting both what GPT-3 is “capable” of doing and shifting what its “goal” is. (There’s not really a clean practical or conceptual distinction between the two.) It would also probably require making progress on some sort of “alignment” technique, since we can’t (e.g.) write down a hand-coded reward function that quantifies novel quality.