This proves to me that you are smart.
Interesting dialog, thanks for going into the trouble.
cveres
Karma: 48
A strange twist on the road to AGI
My argument against AGI
It is from a newspaper from the 1800s, probably on microfiche. Humans can sort of cope :)
The passage:
THE DEATH OF LOGAN FONTANELLE, THE OMAHA CHIEF.
The following interesting narrative we have just received from America, and is an apt illustration of the oft repeated statement that “truth is stranger than fiction.”
Logan Fontanelle, chief of the Omahas, has just been slain and scalped at Loup Fork, by a band of Sioux.
Logan was a noble fellow, and in this last mortal conflict he despatched several of the enemy to the spirit land before, to herald the coming of his own brave soul.
He fought long, desperately, and with great effect, but numbers finally overcame him, and his life departed through a hundred wounds.
He died a martyr for his people, and his name should be carved upon fame’s brightest tablet.
He was on his annual hunt with his nation.
A number of his lodges were pitched upon the plains near Loup Fork.
As a young warrior rode around the adjaent hills, he espied a powerful band of Sioux encamped along a stream in a sequestered vale.
Ile hastened to inform Logan of the propinquity and power of their natural foe.
Logan ordered his people to pack immediately, and proceed in a straight line and with all speed for home, while he would remain behind, and divert the Sioux by false camp fires and other devices, from a direct pursuit of them.
This was about twilight.
The people got under way as quickly as possible, but not too soon ; for scarcely had they turned a highland, when several Sioux warriors came in sight and discovered the place of their recent encampment.
They examined it, and found that Omahas had been there, and then they returned to notify their chief, and bring an adequate force to pursue and slaughter them.
Logan, from a biding-place, saw all, and knew that no time was to be lost in drawing their attention from the trail, which they would soon discover and follow, and mounting his horse, he dashed away at full speed across the prairie, at right angles with the route his tribe had taken, and struck a fire about eight miles distant, on an eminence where the Sioux could distinctly see it.
He had scarcely done so before a powerful band were upon the spot that he and his people had so lately left, and who, without stop—ping to distinguish the trail, started for the tire which they saw rising against the clear, blue sky, and where they expected in another moment to imbrue their hands in the gore of their unguarded victims.
But Logan had not been unwary.
As soon as the fire was lighted, he again mounted and rode on eight or ten miles fur—ther, and kindled another fire just as they reached the first.
This rather bewildered them.
They dismounted and examined the ground.
Logan, anticipating this, had trotted and walked his horse around it, so as to make the appearance upon the grass of the treading of a dozen horses ; and this drew them into the belief that a small body had lingered behind and kindled this fire, and then gone on to where they could see the new fire burning ; and so they followed with renewed avidity.
The same thing happened as before.
Logan had gone on, and another fire met their astonished gaze, while the were now gathered.
Their suspicions were
They examined the ground more closely, both far and near, and discovered that a solitary horseman bad deceived them, and th knew it was for the sole purpose of leading them off from the pursuit of the party whose encampment they had first dis-
Logan saw them going round with glaring torch, nd understood their object, and knew that his only chance of safety w that by the time they home; and he further kn could retrace their way to their place of startin find the trail that his own people had taken, they would be beyond the reach of danger.
The Sioux, in the meanwhile, had divided into pursue the Omahas, and the others to endeavour to cap—ture the one who had misled them.
They knew e must be an Omaha, and that he would eitht further and kindle ,nother watch-fire, or start for hi nation in a straight line ; and, therefore, one party went on a little further, and the others spread out towards the Omaha country for the purpose of intercepting him.
Logan pressed forward as rapidly as his jaded steed could bear him, until he thought he had entirely eluded vithin its verge he met.some predicates that would be nice to have, in order to create a knowledge graph. Note that there should be some semantic knowledge like for example slay = kill. This is just a quick fanciful set, but I think any person could pretty quickly learn to extract key relationships and properties from the passage, and handle pronoun resolution at the same time.
chief(The Omaha, Logan Fontenelle)
received_from(America, narrative) or even better, origin(America, narrative)
topic(death of Logan Fontenelle, narrative)
slay(Logan Fontenelle, Sioux) = killed(Logan Fontenelle, Sioux)
scalped(Logan Fontenelle, Sioux)
fought(Sioux,Logan Fontenelle)
property(martyr,Logan Fontenelle)
warned(Logan Fontenelle, young warrior)
I have found a cute passage from an old newspaper which I was using in a project with the British Library newspaper archives. I will post it in the next reply, but first another challenge from my longer paper. I asked GPT-3 to check the truth of an implication but it got the wrong answer. Perhaps you can find a better way to ask? Here is the relevant snippet:
“Is it true that if the moon is made of green cheese, and cows chew cud, then the moon is made of green cheese?”, and GPT-3 answers “No, it is not true that if the moon is made of green cheese, and cows chew cud, then the moon is made of green cheese.”, which is not correct. GPT-3 appears to focus on whether or not the consequent itself is true or false.
I have also tried prompting as you suggested earlier but could only get this far:
You are an expert in logic and you are asked to verify the truth of the following implication. “If the moon is made of green cheese, and cows chew cud, then the moon is made of green cheese”.
GPT-3: The moon is not made of green cheese, therefore the implication is false.
Thanks I will check it.
Sorry my sentence was not sufficiently well specified. What about:
“The fact that a language model can reproduce Python strings without any operations that are isomorphic to the production rules in the BNF specification of Python grammar shows that they are not informative about Python grammar.”
Now the contraposition is (I think)
“If a language model is informative about the grammar of Python then it cannot produce Python strings without any operations that are isomorphic to the production rules in the BNF.”
This then becomes the argument about implementational connectionism I made in my longer paper. If the network is implementing the BNF then it becomes far less interesting as a radical new model of Python in this case, but generally as language or cognition.
You are right, this is not a very strong form of argument, which is why my main argument is based on what the models CAN do. That is, the fact that they can reproduce Python strings shows that they are not informative about the grammar of Python (or natural language).
But I will test some more examples against GPT-3 and see if they work.
I would also LOVE IT if you figured out how to make it do open relation extraction. That would be very useful.
Right .. but I see that as a problem if you claim that Python doesn’t have a relevant grammar. Of course everyone knows it has a grammar so no one claims this. But people DO claim that natural language does not have a grammar. This is what I have a problem with. If they said natural language has a grammar and “neural networks can check a natural language string against the relevant grammar”, I would have no problems. But then these people would not be in a position to claim that they have discovered something new about language, just like we are not in a position to claim that we have discovered anything about Python.
Thank you. I was also very confused and disappointed, especially because the downvotes did not come with comments. In fact from my follow up post you will see that I have been begging for someone to tell me what’s wrong with the argument. Yann LeCun and Christopher Manning on Twitter have not been able to.
So I am happy with substantive and convincing counter arguments, which I will of course try to answer. In the best tradition of the scientific method.
Thanks David. Indeed, I completely agree that humans use external symbolic systems to enhance their ability to think. Writing is a clear example. Shopping lists too.
And to answer your last question—indeed I am saying exactly that DL based language models CAN do this. i.e. they can classify grammatical strings. But by doing this they act as a tool that can perhaps simplify the task. The correct way to check the grammar of a Python string is to look up the BNF. But you can also take shortcuts especially with simple strings.
I don’t really understand your comment, but I’d like to point out that I didn’t invent the “symbolic” idea. Leading people on both sides (LeCun, Bengio, Marcus) agree that some form of symbolic reasoning is necessary. It IS a very complex problem I agree, and I encourage everyone to engage with it as the top researchers in the field already have.
Thanks for mentioning the self programming AI paper. It seems that you place a lot of faith in how far such approaches can take us on the way to some sort of human-like AI. But when I read the paper I was not so convinced. The critical part for me is how the AI is allowed to change its program: “The possible modifications include adding convolutional layers, changing the size of convolutional or hidden layers, and increasing the number of hidden layers.” But this is quite trivial. The recent history of AI shows us that big breakthroughs happen with advances in architecture. The transformer model is a prime example. But this program will never come up with a new architecture if all it can do is modify trivial aspects of an existing architecture. We can of course dream that if an existing architecture becomes good enough it can break its chains and modify anything at all about its code base and even create new architectures. But this seems like a very big stretch of the imagination from what is actually presented in the paper.
Excellent question. There are many ways to answer that, and I will try a few.
Maybe the first comment I can make is that the “godfathers” LeCun and Bengio agree with me. In fact the Browning and LeCun paper has exactly that premise—DL is not enough, it must evolve a symbolic component. In another Twitter debate with me LeCun admitted that current DL models cannot write long complex Python programs because these require planning, etc., which non symbolic DL models cannot do.
There is a lot of debate about what sort of “next word prediction” is happening in the brain. Ever since Swinney’s cross modal priming experiments in 1979, we have known that all senses of ambiguous words become activated during reading, and the contextually inappropriate becomes suppressed very quickly afterwards as it is integrated into context (https://en.wikipedia.org/wiki/David_Swinney). Similar findings were reported about syntactic analysis in the years since. But these findings are never taken into consideration, because DL research appears to progress on stipulations rather than actual scientific results. Clearly we can predict next words, but how central this is to the language system is debatable. What is much more important is contextual integration, which no DL model can do because it has no world model.
But let us take equally simple language tasks on which DDL models are terrible. One really useful task would be open relation extraction. Give the model a passage of text and ask it to identify all the key actors and their relations, and report them as a set of triples. I try and do this in order to create knowledge graphs, but DL models are not that helpful. Another simple example: “The dog chased the cat but it got away” vs. “The dog chased the cat but it tripped and fell”. I have not found a single language model that can reliably identify the referent of the pronoun. What these two examples have in common is that they need some “knowledge” about the world. LeCun does acknowledge that we need a “world model” but again this is where I disagree on how to get it.
One final point. Current language models do not have a theory of language acquisition. Chomsky rightly pointed out that a good theory of language has to explain how children learn. It is pretty clear that the way language models are trained is not going to be able to explain anything about human language acquisition. Another point that is conveniently excluded from discussion.
There are many many other reasons why we should doubt how far these models will get. People tend to focus on progress on largely made-up tasks and ignore the vast landscape of unknowns left in the wake of these models. (Emily Bender on Twitter has some interesting discussions on this). Hardly a confidence inspiring picture if you look at the entire landscape.
Thanks for the comment. You are right, I forgot to put the word “intelligent” into quotes. Because as Turing already pointed out, it is not a well-defined term. So I am using it somewhat colloquially.
But I do take issue with your second point. First of all, there is nothing insubstantial about semantics. Semantics is meaning, and meaning is everything. So my point is that current AI is not building models that are capable of carrying out the full range of human cognition that is necessary for planning, creativity, etc. This is exactly what they want to know, I think—because this is what is needed for “AGI”. But this question is to a large extent orthogonal to when they will have capability to destroy us. They already do. Hook up a DL based decision system to the nuclear arsenal and sit back. In terms of the FTX future fund, I just want to point out that they have an entire section devoted to “semantics”: “What do you mean by AGI?”
Don’t expect AGI anytime soon
Interesting thoughts. But do you think AI will, on balance, help or hurt us in this quest?
Interesting post Yuri, but I am very confused about your claim that Pavlov’s ideas were ignored: “this mechanism has been neglected by the mainstream of psychologists”. My understanding is that the ideas inspired the U.S. school of Behaviorism where Watson and then Skinner pretty much ruled American psychology from 1920 to the mid 50s.
The Cognitive Revolution spearheaded by for example Chomsky, showed that simple rules of learning were not sufficient to explain adult competence. The debate has been revived in a modern form by deep learning, of course,
The reductio is specifically about Python. I show that the argument must conclude that Python is not symbolic, which means the argument must be wrong.
So your alterenative would be that BL shows that Python is not based on classical symbol systems.
We don’t know how human cognition works which is why the BL argument is appealing. But we do know how Python works.