The Sente Playbook

GenAI Is Not Stupid, But It Is Ignorant

Ryan McClead

For the entire history of human civilization, the ability to put words together intelligently, whether spoken or written, has indicated an underlying level of understanding and a general level of intelligence of the speaker or writer. The development of Generative AI may be a major milestone in the creation of artificial intelligence, but it also represents the decoupling of language from intelligence.

To say that “GenAI has passed the bar exam”, is actually saying that GenAI can string together words that, if they were written by a human being, would indicate that the writer had a baseline understanding of the law. But that is fundamentally different from actually having a baseline understanding of the law. GenAI doesn’t understand “the law”. It doesn’t understand the meaning of the word “law”, any more than it understands the meaning of the word “fish”. GenAI understands exactly one thing and one thing only, the probability that the next word in the sentence is “law”. Or “fish”. Or “understanding”.

Okay, there is a bit more going on under the hood, but at its heart, that is what a large language model does. It chooses the next word to write from the most probable next words available.

I won’t say that GenAI “knows nothing”. The probability of the next word that will come, within the context you’ve been given, is most definitely “something”. It’s a “something” that we probably all do unconsciously in simplified form as we write and speak sentences. In fact, AI scientists may have unlocked the digital equivalent of the biological mental processes we use as we write and speak. Except that for a human to generate coherent text or speech, requires a baseline understanding of the world around them; a minimal passing knowledge of a language’s grammar, rules of sentence construction, and vocabulary; and a simple comprehension of the difference between things and symbols that refer to those things, even if the human can’t actually articulate any of that. This means that requiring a human being to write or speak about their knowledge is a pretty good way to test whether they actually know what they say they know. Asking a computer to do the same tells you how good the computer’s language probability model is.

Generative AI bypasses all of the standard human language requirements and just knows symbols and how likely people were to write one particular symbol after a series of other symbols in the past. There are several examples of still

undeciphered writing systems⁠

around the world. I have no doubt that given a sufficient quantity of text written in those languages an LLM could generate volumes more content that we still could not decipher, but a native reader would probably understand. Still, GenAI could not translate that text and tell us what it was writing, because it simply calculates probability of recurring patterns, it doesn’t understand language. Or anything else. That remarkable skill to calculate probability means that GenAI can write a perfectly coherent and intelligent essay on the differences between laws and fishes, without knowing a law from a fish in any meaningful sense. Whereas, to do the same, a person would have to have a substantial understanding of both laws and fishes, or they would quickly be called out by people who knew a lot about one or the other.

We humans have historically conflated strong language skills with intelligence and understanding for a very good reason; we have only ever existed in a world where the two were inextricably linked. Prior to the development of GenAI, if you could use language skillfully to make your points, it indicated that you were knowledgeable in that particular subject. Now it indicates you’re either knowledgeable in that subject or you’re skilled at prompting the AI to appear to be knowledgeable in that subject and, demonstrably, those are not the same.

Those of us who rely in part on our language skills to make a living, whether we’re lawyers, consultants, or writers are not in trouble in the way that the doomsayers claim. GenAI will not replace a knowledgeable person, because it’s their knowledge and their ability to apply that knowledge in novel situations that is actually valuable. The language is just how they express that knowledge. GenAI as a tool in the hands of a knowledgeable person will only increase the value of that person’s knowledge. However, those of us who rely on language skills to prove we’re the smartest kids in the class, on the other hand, we’re kinda screwed.

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.