Overall, I think the degree to which intelligences of whatever kind, looking at the same phenomena, will converge on the same concepts, is significantly greater than you make out. To take one of your examples, we may not be able to read off the concept of “car” from the laws of physics, and it is a concept that might be lacking in an alien civilization. However, an alien anthropologist who chooses to study 21st century humans will necessarily develop the concept of “car”, because that actually is a meaningful cluster of the physical objects that humans interact with, that actually is one of the joints of 21st century human reality.
That said, I’m still pretty confused about what the implications of any of this for alignment are. You seem to agree that so long as we train models on the same sorts of data we are now, they are likely to converge on the same concepts. But when do you foresee us doing anything else? Training a frontier model is expensive. Nobody is going to put in the investment for a model that isn’t expected to be useful to humans, and so every model will continue to be trained on human scale data for the foreseeable future. What change do you imagine might happen in how we build LLMs that might shift them to a different basin of attractors?
Overall, I think the degree to which intelligences of whatever kind, looking at the same phenomena, will converge on the same concepts, is significantly greater than you make out. To take one of your examples, we may not be able to read off the concept of “car” from the laws of physics, and it is a concept that might be lacking in an alien civilization. However, an alien anthropologist who chooses to study 21st century humans will necessarily develop the concept of “car”, because that actually is a meaningful cluster of the physical objects that humans interact with, that actually is one of the joints of 21st century human reality.
That said, I’m still pretty confused about what the implications of any of this for alignment are. You seem to agree that so long as we train models on the same sorts of data we are now, they are likely to converge on the same concepts. But when do you foresee us doing anything else? Training a frontier model is expensive. Nobody is going to put in the investment for a model that isn’t expected to be useful to humans, and so every model will continue to be trained on human scale data for the foreseeable future. What change do you imagine might happen in how we build LLMs that might shift them to a different basin of attractors?