What Penrose Couldn't Name

Friday night and I was on X scrolling the way you do when the day is over, and a video stopped me. Two minutes cut out of a longer interview, the kind of thing that gets passed around with a caption telling you what to think about it. Twenty-three thousand people had already watched it. The title promised that artificial intelligence will never be conscious.

I watched it twice. The second time I had stopped listening to the argument and was listening to a pause.

They ask him whether AI might be a new sphere of human existence, and he says: "No. It's missing something I think. Yeah, that's the trouble." Then he stops.

The man stopping is Roger Penrose, ninety-five years old, mathematical physicist at Oxford, Nobel laureate in 2020. The prize was for something with no bearing on machines whatsoever: proving on paper, with mathematics rather than a telescope, that black holes are a forced consequence of general relativity instead of an exotic possibility. Space and time, no minds involved. Minds are his second trade, and he has been at them since 1989. In the clip going around, the word (Nobel) sits in brackets beside his name, and the brackets are propping up the wrong question.

All the same, that I think and that that's the trouble are the only words in the whole two minutes that close nothing. I went looking for what he finds missing, and what I found is that he has been looking for thirty-seven years.

He did build a machine for answering this, and running it is more interesting than summarising it. It starts with a theorem from 1931, proved by Kurt Gödel, an Austrian logician who was twenty-five at the time and who with that one result ruined the sleep of a generation of mathematicians busy setting all of mathematics on complete and secure foundations. Gödel showed it cannot be done, and here is how. Take a formal system: a list of axioms and mechanical rules for grinding out theorems, the sort of thing you execute without understanding it, the way a clerk applies a regulation without asking why. Inside that system Gödel builds a sentence that says of itself I am not provable in here. If the system proved it, the system would be proving a falsehood. So provided the system is consistent, the sentence is true and the system cannot reach it. Add axioms, and the stronger system finds itself with a fresh sentence of its own.

Penrose aims the theorem at us. Suppose human mathematical understanding were a system of that kind, call it F. Then we, looking at F from outside, would see that its sentence is true: we would see something F cannot reach. So F fails to describe us, and since F was any such system at all, nothing of that kind describes us. The thing we would do and the machine would not is step outside the system and judge it sound, which is exactly the operation Gödel forbade a system to perform on itself in a second theorem.

It is worth watching this run all the way down, because it holds. What it does is narrow. We see the sentence is true on condition that F is consistent, and the version with the "if" in front is one a machine derives perfectly well: to beat it we would have to know, unconditionally, that F is consistent, and about a system as large as human mathematics nobody knows anything of the kind. Then the rest. We have no idea which system we would be, so we cannot write our own Gödel sentence, because writing it would need precisely that outside view of ourselves which is the one thing we are missing. At every step the argument stays standing and gets tighter, until what it started out holding, a difference, has become a conditional.

What stopped me is that Gödel saw this himself. In a 1951 lecture drawing the philosophical consequences of his own theorem, he concluded a disjunction: either the human mind surpasses every finite machine, or there exist absolutely undecidable mathematical problems. He left both doors open, because the theorem on its own closes neither, and he never claimed the first one. The man holding the proof was more careful than anyone who has cited him since. Penrose picks a door, and keeping it costs him: he has to go looking for non-computable physics inside the brain, in the microtubules of nerve cells. That is the price of closing, and you can see it being paid.

At that point I turned the test on myself and got stuck. Knowing what you are doing. In 1977 two psychologists in the United States, Richard Nisbett and Timothy Wilson, gathered up what was known about how much access people have to their own mental processes, and the answer is that we confabulate. Ask someone why they chose what they chose and they answer with confidence, describing steps they have no access to. An experienced surgeon cannot tell you what her hands are doing; she watches them do it. I cannot tell you why I chose this word over that one. I can give you an account, and the account would be plausible and unverifiable. The test I use to rule the machine out, applied to me, I fail.

Alan Turing had seen this in 1950. He is the English mathematician who broke German codes during the war, and who before that invented the very idea of a machine able to compute anything computable. In the clip Penrose writes him off as "a little bit confused person in a way". Turing wrote that the question of whether machines can think was too meaningless to deserve discussion, and replaced it with a game: a public procedure where the matter is settled from outside, by watching how the thing behaves. It is the most elegant move of the century, and it is also a change of subject, and he knew it. A procedure tells you how something behaves. It leaves the thing you wanted to know sitting where it was.

So I went back to the pause. Whoever cut that video took three seconds of hesitation and got a verdict out of them, with the Nobel in brackets beside the name and twenty-three thousand people going to bed reassured. I was about to take the same three seconds and get a diagnosis out of them: an unnamed remainder, unfalsifiable, therefore a consolation. Two gestures opposite in intention and identical in mechanics. Both of them stop the only thing in there that was still moving.