On your part about ingenuity, I immediately thought about AIs making up new languages to communicate—possibly to hide their work from human supervisors, but possibly to invent more efficient ways to reference recognized patterns. Are they adding anything new when they do this? And a similar question: Does Newton add anything when he recognizes and proves gravity via math?
If AIs mirror human reasoning, and intelligence is nothing but recognizing causal structures and predicting the next best result, adding things over and above causality is only connecting to what actually exists. New experiences add to causal web, so experiencing novelty will be a very important part to creating superintelligence.
On your part about ingenuity, I immediately thought about AIs making up new languages to communicate—possibly to hide their work from human supervisors, but possibly to invent more efficient ways to reference recognized patterns. Are they adding anything new when they do this? And a similar question: Does Newton add anything when he recognizes and proves gravity via math?
If AIs mirror human reasoning, and intelligence is nothing but recognizing causal structures and predicting the next best result, adding things over and above causality is only connecting to what actually exists. New experiences add to causal web, so experiencing novelty will be a very important part to creating superintelligence.