GPT's Very Inhuman Mind

Fernando Kaskais

Mar 31

Like Narcissus and his reflection, too many AI researchers are searching for the essence of intelligence when it doesn't exist.

BY REUBEN COHN-GORDON - Reuben Cohn-Gordon is an AI researcher and writer. He is soon starting a postdoctoral position at the University of British Columbia in Bayesian inference and machine learning. Previously, he studied classics.

– ONE –

"He loves a whim without substance — what he thinks to be a body is a shadow."
— Ovid, "Metamorphoses," Book 3, line 417

The phrase "mere pattern recognition" tends to appear over and over again in discussions of modern advances of AI. What it gestures at is a perceived dichotomy between true understanding and a shallow surface-level trick that gives an interlocutor the impression they are talking to an agent that understands.

The apparent distinction is neatly embodied by the difference between an echo and Echo. An echo doesn't understand what you said — you could scream in Amharic, Swahili or English, and the distortion produced by an echo would be much the same. Echo, on the other hand, is a person: a figure from Greek myths made famous in Ovid's "Metamorphoses" who lurks in caves and repeats back what she hears. (The traditional story is that she does this because of a curse, but she prefers to think of it as an elaborate piece of performance art in the style of Oulipo.)

Echo understands what the sound you made means. When you say, "Where are you?" she apprehends more than just a series of vibrations in the air. She gets the information, the meaning, the content. "Where are you?" she replies, tauntingly, implying that you're a busybody.

In an abstract sense, Echo and an echo are similar, both being machines that receive a sound and spit out a sound in return. But in another sense, they couldn't be more different. One merely gives the impression of being a mordant interlocutor, the other actually is one.

As for a conversational AI system like the now terminally over-discussed ChatGPT, it too has the trappings of a real interlocutor, returning back words in answer to the words you give it.

Me: If computers were made of wood, would they work?

ChatGPT: No, computers need electricity and metal components to function.

Me: Why?

ChatGPT: Wood is an insulator, meaning it does not conduct electricity. Without electricity, a computer cannot function. Additionally, wood is not strong or durable enough to withstand the physical pressures of a computer's components.

Me: What about a conductive hardwood?

ChatGPT: While some hardwoods may be slightly conductive, it would still not be enough to power a computer. The conductivity of the wood would be far too low to support the electrical components of a computer.

And so on. If deep learning systems from the 2010s onwards began to gesture toward semantic understanding, ChatGPT practically flaunts its flexible, fluent ability to reason counterfactually, smoothly taking into account both linguistic style and world knowledge.

But the perennial accusation against modern statistical AI, no matter how impressive it appears, is that it is less like Echo than her lowercase cousin, an operation on form when it should be an operation on content. Or like ELIZA, a program from the 1960s that rearranged what you said as a question, giving the impression of being uppercase Echo, not unlike the way two children stacked on top of each other in a trench coat gives the superficial impression of an adult. The meaning you perceive from ChatGPT's responses, in this view, is only what you project onto it, a more elaborate version of perceiving a mocking tone in an echo.

The warning given by a host of like-minded linguists and cognitive scientists is that purely statistical approaches, being conceptually flawed, will hit a ceiling soon enough, with all the hubris of Icarus flying too near the sun. This will trigger an AI winter as the stock of OpenAI plummets into the chilly Adriatic.

That this criticism is so pervasive, despite the extraordinary behavior of modern AI systems, is a testament to an enduring analogy between the mind and software. As far as computer programming is concerned, the difference between echo and Echo, and the concomitant danger of confusing the two, is a familiar theme.

Programs are sequences of characters, while their meaning, or their content, are numbers or functions or other sorts of mathematically exact entities. One could imagine the forms of all programs living in one space (picture Borges' infinite library from the short story "The Library of Babel," if you are that way inclined), which contains every possible sequence of characters. One could also imagine the meanings of all possible programs living serenely in another space, one of mathematical functions.

The relationship between the two "libraries" is at the bedrock of computer science and mathematical logic, but it is equally relevant to everyday software writing tasks: Programs work by manipulating content, not form. For example, a chess program might take a string of characters like "8/8/8/4p1K1/2k1P3/8/8/8 b – – 0 1" as input (form) and extract from this the state of a chessboard (content). It might then perform some calculation on the basis of this content, like determining whether black can capture any pieces (yes/no).

This is a machine that takes form (string of characters) and produces form (yes/no), but trying to write this function without routing it through the content (the state of the chess board) and the surrounding concepts (the rules of chess) would be a doomed exercise.

In fact, a program that commits this sin has a name: It is a hacky program...

more...