I'm sure you've heard the argument that LLMs aren't really thinking because, according to them,
LLMs are just predicting the next token...
And that output is also non-deterministic.The problem with this argument is the same exact thing applies to the human brain. You can just as easily say,
And Shakespeare is just vibrations of vocal folds.
The human brain is nothing but a collection of neurons that finds patterns and generates output.
The reason output argument There's a related argument for something's ability to reason, that says something like,
If you give a problem and the output could only have been the result of reason, then the thing you gave it to must be able to reason.
This was by John Searle in a 1980 paper called Minds, Brains, and Programs.And one counter to that argument is called the Chinese Room Argument, which basically says that computers can't ever really be thinking because they don't truly understand what they're doing.
The paper's analogy is a person inside a room with a set of instructions on how to translate Chinese characters into English. The person inside the room doesn't actually understand Chinese; they just follow the instructions to produce the correct output.
The paper argues that this room itself is like a computer—any and all computers actually—where there is complex work coming out of it, but at no point is there actual thinking or understanding happening.
The problem with the Chinese Room Argument I'm not sure why this isn't brought up more.The extraordinary mistake in this argument is not realizing that the brain of a Chinese-speaking human—is also a Chinese machine.
When Chinese goes into the ears or eyes of a Chinese speaker, what comes out is understanding of what that Chinese meant. The speaker has no idea how that happened. They simply look at the characters on the page, or hear the words, and they somehow know what they mean.
This is exactly as if those Chinese characters are being passed into another room inside of the Chinese speaker’s brain, and inside the brain there is someone in a room with actual Chinese understanding.
We know for certain that this is what's happening inside the brain of every human who understands a language. Or anything else.
What if your black box got replaced with another? Would you even know?Most people don't think about the fact that we have no idea where our words, or sentences, or ideas come from. They just stream out of us from some mysterious place.Once we understand something, the ability to produce out output from that understanding turns into a Chinese room. We throw input, and our brain throws back output. From where? We have no earthly idea.
Our brains are Chinese rooms.
Somehow, we have forgotten this. We've completely overlooked the fact that we haven’t a slightest idea where all this knowledge, and understanding, and creativity, is coming from when we pose problems to our own minds.
Confusing substrate and output Not strictly true, but you get the point.If you slice into a human brain and look at what it's made of, you will find nothing but neurons, axions, and synapses. Just as you will find nothing but layers of weights in the brain of a Transformer. Two different substrates, but ultimately an architecture of connections and activations.
Mostly opaque, these days.In both cases, you have a completely opaque, non-deterministic, and highly mysterious process around how it generates output.
If you want to know if something can reason, I propose the only way to find out is to:
Anything less is dishonest at some fundamental level.
Yes, humans can reason. But we're really sloppy with how we think we know we can. And as a result, we're similarly sloppy with our definition of reasoning.
Most attempts end up being No True Scotsman appeals that basically say,
And of course they can't actually explain how humans do.
It's truly reasoning if and only if it reasons like I think humans do.
We have no idea how humans are able to reason or be creative.
Very much like Transformers, our brains are black boxes that produce wondrous output through a vast network of small nodes that light up together when given input.
Any materialist who claims that we are fundamentally different than Transformers has all their work in front of them, since they neither know how humans do it nor how transformers do it.