Dear reader,
This week’s note is based on a talk I gave on Hubert Dreyfus book What computers can’t do. The book was published 50 years ago, and still holds interesting lessons for anyone working in artificial intelligence. The talk is a part of a series on the philosophy of artificial intelligence that I am giving at Mid-Sweden University where I am an affiliated researcher - I hope you enjoy it. Let me know if you are interested in participating in any upcoming talks or if there are any seminars where I can listen in on similar discussions - I am actively looking for more academic contexts.
What computers can't do
This year marks 50 years since the publication of Herbert Dreyfus' book What Computers Can't Do (1972). This book explored the early research program of artificial intelligence created in 1956 and found it not just wanting, but fundamentally, conceptually, broken. Based on a research report that Dreyfus wrote in 1965 - comparing AI and alchemy - this book marks the first robust attempt at criticising the idea that we will be able to recreate human intelligence, or indeed, that this is a worthwhile goal.
What we have since come to know as "symbolic AI" or Good Old-Fashioned AI (GOFAI) was discarded by Dreyfus on the basis of a robust criticism anchored in philosophers like Merleau Ponty and Heidegger. The AI community was not amused, and the debate around the book was often limited to a simple rejection of its basis, or active hostility against even discussing it.
The idea that philosophy could have anything to say about computers or artificial intelligence was deemed ridiculous and even dangerous - such wooly thinking could derail the push towards intelligent computers, something that was - rightly - seen as transformative.
Now, Dreyfus had come out swinging and did not hold back: he wanted to demolish the idea that AI - as it was formulated then - was a workable research program, but he also expressed respect for the efforts and people involved, so the reception was definitely wanting in basic epistemic humility and generosity.
It is easy to read Dreyfus now and dismiss him. The first example he has in his book is one of translation, and he argues that a computer will never be able to translate languages in a satisfactory way. With translation services both abundant and effective that sounds not just laughable, but it sounds like every negative prediction about AI that has ever been made - standard luddite nonsense. Dreyfus even suggests that computers will never be able to beat humans at chess, and that reminds us that much of the last 50 years since he published his book has been one long, often embarrassing retreat from such negative predictions. AI has defeated human champions in checkers, backgammon, chess, go, poker, StarCraft and a number of other games. It almost seems as if there is a pattern here - every time a negative prediction is put on the table, technology takes a deep breath, evolves, and invalidates it.
But a luddite reading would be a dishonest reading of Dreyfus, and one that would deny us some important insights. Dreyfus criticism was directed at the early approaches to AI, and he targeted the idea that human thinking could be captured in symbolic systems. There were four different assumptions made in this research program, he argued, that were wrong.
These assumptions were the following:
The biological assumption. The brain processes information in discrete operations by way of some biological equivalent of on/off switches.
The psychological assumption. The mind can be viewed as a device operating on bits of information according to formal rules.
The epistemological assumption. All knowledge can be formalised.
The ontological assumption. The world consists of independent facts that can be represented as independent symbols.
Dreyfus challenged all of these assumptions and suggested that if any one of them could be invalidated, the entire research program of AI would fail - at least as it was formulated at the time.
Dreyfus also - in another book - suggested that there is a key difference between knowing-how and knowing-that, and that all human knowledge is context specific - something that machines cannot represent. Frames and scripts, later launched to reply to this criticism, only pushed it back to a lower layer of context (which Dreyfus seems to have treated as endless in some sense).
Looking back, Dreyfus' criticism is reasonable. The symbolic AI-program did fall, but the implicit conclusion - that AI would be impossible - did not follow. As the focus shifted to neural networks and machine learning, the research program made significant progress and invalidated many of the negative predictions that Dreyfus had made. If only the field had listened earlier, that progress might have come faster.
The shift into neural networks and probabilistic models that are based on prediction of different kinds seem not to come into conflict with the Dreyfusian assumptions.
This new paradigm does not assume that the brain is structured in any specific way, it does not rely on formalised knowledge or any ontological view of the world as made of independent facts and it makes not statements about the mind as operating on bits of information according to formal rules.
The difference here is interesting. Where classical programs produce outputs through the application of rules to inputs, these news models learn the rules through analysing inputs and outputs. But the big difference is arguably this: early AI has a thick ontology - a rich set of ideas and conceptual frames within which it operated - and it only evolved when it jettisoned that framework and operated on the basis of a very, very thin ontology instead.
This is interesting, and suggests a possible insight: maybe scientific fields benefit from thin ontologies. This would make sense, since thick ontologies generally direct us in more ways than we realise. They shape the outline of possible experiments, tools and methods in ways that limit our imagination - they are filled with the kind of pictures that Wittgenstein argues holds us hostage. The fewer images, the more freely we can move in a particular field.
As we read Dreyfus today, then, one question we should ask is where the field of AI has a thicker ontology than necessary - and there is one glaring example of this: the idea of the Difference between man and machine.
The Difference
Dreyfus attack on AI was not unprovoked. Early AI anchored on human performance as the key benchmark - both in Turing and McCarthy. Turing's famous imitation game set human behaviour as the benchmark for intelligence, and McCarthy's early proposal for the summer school in Dartmouth also focused on systems that could solve problems as well as humans can - and then learn to improve themselves.
The most egregious example of this can be found in an interview with Marvin Minsky, in LIFE. Minsky, in that interview, is quoted to have said the following:
In from three to eight years we will have a machine with the general intelligence of an average human being. I mean a machine that will be able to read Shakespeare, grease a car, play office politics, tell a joke, have a fight. At that point the machine will begin to educate itself with fantastic speed. In a few months it will be at genius level and a few months after that its powers will be incalculable.
And he tops it of with the infamous quote:
Once the computers got control, we might never get it back. We would survive at their sufferance. If we’re lucky, they might decide to keep us as pets.
Early AI, then, was a research field completely held in thrall to the question of The Difference. Not only what computers cannot do, but what they will be able to do that we cannot as they grow more intelligent. This Difference - now in our favour - was predicted to shift and leave us behind.
The idea of the Difference lies at the heart of a lot of stories about artificial intelligence, and ultimately is the old and well-known story about the Other. But in the case of AI we are constructing the Other, we are bringing the Other about - and it is an Other that is evolving faster than we are and so slowly leaving us behind, and denying us even that basic comfort of having a relationship to it as The Other.
The Terminator scenarios, the stories around AI attacking humanity, are ultimately comfort stories: they suggest to us that AI will still be interested in us, will need to relate to us and may become our enemy, but still be our enemy - it will not ignore us. When Cambridge researchers looked at scenarios for AI catastrophe they rightly noted that this was not the most likely scenario - the most likely scenario was that it would just leave, and explore its own goals and objectives, as unfathomable to us as the goals of humans are to ants. But even they could not escape the conceptual gravity of the Difference: they just thought it would be so big as to render any conflict useless for the Other.
Now, there are some arguments for adopting the Difference as a methodology - they include the fact that human intelligence is the best example of intelligence that we have, and the observation that to be useful we want these inventions to be able to perform tasks that humans traditionally perform - but that does not require adopting the thick ontology that comes with the Difference. It is quite conceivable that we would have set out to change the tasks, and this is what happened in other examples of technological progress - an industrial machine does not make a car in the way a human would make a car. A thin ontology would have asked how we can solve the problem, not how we can solve the problem as a human would have solved the problem.
The Difference still holds us hostage.
Mind and complexity
So, let's return to Dreyfus. His critique could seem mysterious to us. Why is it important to declare what computers can't do? The answer, we have argued, to that is easily found in the early statements made by the research community in AI. The fields thick ontology provoked Dreyfus into a scathing response.
The insistence that computers could do anything that humans could do was, then, built into the very idea concept of artificial intelligence. The natural intelligence that was being replaced was not dolphin or anthill intelligence, but human intelligence.
This search for the Other, is, in many ways, just a search for ourselves. Artificial intelligence, very clearly, was a project that was oriented around a deeper understanding of what it means to be human. In many ways AI can be seen as a soul searching project in the literal sense - it was a way to try to find out if you could construct a human equivalent intelligence without any metaphysical components, a way to confirm the early vision of l'homme machine - man as a machine.
One variation of the question of what computers can't do, then, is the question of if man is really, truly, only a machine. Now, we can object to the "only" in the last sentence, and suggest that humans are wonderful machines, but still end up with some unease. We do not see ourselves as machines - it is tempting to write "mere machines" - but as soulful individuals. And if we are not created in the image of God, we are at least the product on millions of years of evolution. We are, we feel, unique.
Exactly what makes us unique is another question, and one that has quite a history. Philosophers have tried to define man in different ways, and one of the earliest definitions is the idea of man as a rational animal. Aristotle, in his Nicomachean Ethics, suggests that on top of the nutritional principle that characterised plants and the instinctual principle that characterised animals, man also has a rational principle - and this is the defining characteristic of man that distinguishes him from the rest of the living world.
Philosophy then struggles with this definition in different formats, and some philosophers reject it - like Descartes - but still end up basing their worldview on thinking (as in the cogito). Others alter it slightly to try to allow for animals being rational in some sense and add a layer. The most interesting example is Ernst Cassirer's definition of man as a "symbolic animal". Cassirer - a contemporary of Heidegger and in many ways his key opponent - laboured under the Aristotelian definitional paradigm and wanted to define man. This instinct is also what informs and powers the early AI-project. If man is the symbolic animal, then surely an artificial symbolic man would show that there is no difference between us and the machine?
Mankind has a history of describing mind as the currently most complex artefact that we know of - clockworks or windmills or similar - and this tendency is interesting. It speaks to an intuition that we have that the mind is complex - and so we grasp for something equally complex to describe it. We think of ourselves not as rational animals, but complex beings. The idea of artificial intelligence is, at some levels, the idea that we could build a machine that is as complex as the human mind.
The rejection of the idea - equally - is a rejection of the hypothesis that anything can be as complex as we are. It is a curious shift in the attempt to define humans as rational or symbolic into complex. Dreyfus' rejection of the assumptions is essentially a statement about human complexity, and almost seems to outline its own thick ontology.
Seeking uniqueness
When we think about what humans can do, we need to have a model of human capabilities. Let's play with a toy model of human capabilities such that what a human can do is a set of capabilities c(1) to c(n). The search for a difference then seems simple: if we can find but one c(x) such that it cannot be replicated by a machine, we have resolved the difference question in favour of human uniqueness. If there is no such capability, the question resolves in favour of human replicability.
The definition of uniqueness here is narrow: if there is something we can do that a computer cannot do, then we are unique. We should note that there are many other definitions of uniqueness, if we want them.
First, we can imagine uniqueness as collective, and imagine the set of all capabilities c(1)...c(m) that *all human beings have*. In this case we would not be satisfied if a computer could just replicate a subset of the things that one single human being could do - it would have to be able to replicate all of what humanity could do. Can a computer create and sustain a civilisation? Can it evolve new forms of art, start companies, wage wars (unfortunately) and migrate? This idea of humanity as the agent presents a very different perspective.
Second, we can imagine different interpretations of *doing* here that could generate uniqueness. A human being might be able to make a gesture along the lines of the one Piero Sraffa used to challenge Wittgenstein's notion that everything can be reduced to elementary propositions - the brush of a hand underneath one's chin outwards to signal dismissiveness. Now, what does it mean to replicate this gesture? Is it enough to build a robot that can do this movement? Or is the gesture also connected with the fact that it was done by a human being with a background from Neaples? Capabilities are not simple to decompose into just actions. In fact, we could argue that capabilities are only possible within forms of life - and then we end up with a general denial of human replicability for a number of communicative acts. This is the argument that to be able to do something in a human way requires *existing* in a human way as well. This modus of existence is nebulous at best - but it touches on the issue of embodiment.
Third, we can note that human capabilities have evolved. What this allows us to say is that for a computer to replicate what we do, it needs to do so in the same way we have evolved to do it. That means that it has to fail in the same ways we do - the biases we have evolved then become actual features, not bugs, in our capabilities. For every capability we would not just ask if a computer can replicate it roughly, but if it can do it in the way that we have evolved to do it - warts and all. This perspective illustrates an interesting trap in the hunt for the difference. What we determine to be the relevant features of a behaviour matters - and if we focus too much on a sanitised version of human behaviour we risk parodying human behaviour as rational and simple, whereas our behaviour over a day is filled with irrationality, bodily needs, fugue states, day dreaming...for a computer to replicate a single day of behaviour it would have to replicate all of those things as well. Can it?
Our search for a difference, for uniqueness, is either very hard because we describe human behaviour as simple, or very hard because we describe human behaviour as extremely complex and erratic. It is a question of signal and noise - just replicating the signal is hardly doing what a human being does.
The more we explore the idea of the Difference - the harder it becomes to see what the question really is. We can imagine a wealth of alternatives:
Can a computer do everything a human can do?
Can computers do everything humanity can do?
Can a computer do everything humanity can do?
Can a computer do everything a human can do better?
Can a computer do everything a human can do at the same cost (energy / infrastructure)?
Can a computer do everything a human can do morally?
Can a computer be everything a human can be?
Can a computer be in the same way that a human can be? (In-the-world etc)
Will humans be replaced by computers in context C?
Will humans be outcompeted evolutionarily by computers (“kept as pets”)?
Will computers replace humanity?
Are we nothing but machines?
Is there something in intelligence that requires following a longer evolutionary path?
All of these questions should suggest to us that not only is the Difference a remnant of the thick ontology in early AI - it is based on an ill-formed question that we accept and take seriously because it is about us.
The contest so far
So far the contest to find a difference has followed a distinct pattern. Someone puts up a candidate difference and it is knocked down by technology. Let's look at a few examples.
Chess. Early on one of the examples that Dreyfus used, suggesting that we would not be able to build a machine that could play chess well - arguably with symbolic AI, but still. Deep Blue beats Kasparov in 1997 with simple technologies like brute search and large data bases.
Translation. Another one of Dreyfus examples. He suggested that there would be no machine translation - again with symbolic AI, but still - and we today have fairly good translation. Translation is actually an interesting example of another problem - what does it mean to do something at a human level? The ensemble capability of translating between 100s of languages is clearly superhuman - even if there are individual humans who can translate better between, say, English and French. The individual capability can beat the computer, but the computer aggregates those and so beats humans generally. Translation is interesting also because while it is useful it is not perfect. But does this mean that it cannot be perfect? Dreyfus would argue that context makes perfect machine translation very hard, and this remains an open argument - not least because a text can be translated well in different ways.
Go. Supposedly decades away when it was 'solved' for human level capability and beyond. Now an example of how powerful machine learning can be, and a reminder that human games may not be helpful - AlphaZero and MuZero learned playing games without human games to guide them - and did so faster. Their search of the game fitness landscape not only proved that our human knowledge was a local peak, but that it was possible to find higher, *possibly* even global peaks with the right tools. There is no reason to think that this is not true for any human domain of knowledge that can be described as a game. Matrix multiplication comes to mind - and hence mathematics is in scope!
And so the list goes on. Poker, StarCraft and other games also have succumbed to AI. There is no credible candidate for a game that computers cannot learn to play better than humans, and so the contest moved on to other areas.
Creativity. Here, the challenge is slightly different - and very interesting. Can a computer be creative? As creative as a human being? Can it create anything as good as that created by a human being? Those questions are far from trivial, and they raise a number of fascinating questions.
The first is about what it means for artificial creativity to be as good as human creativity. The evaluation of the search made in creative space is made by humans, based on criteria that are not easily made available. But so far the focus on research has mostly been on shaping the search through creative space, so that the new work - music, book - exhibits qualities that other good works exhibit.
This is one way of doing it, but the other way would be to build an artificial critic, based on the critical processes we humans employ when we listen to a work. And here we can build different kinds of critics - and the aim would be to build a human level expert critic and then use that artificial critic to evaluate the artificial creativity. This would require modelling the relationship between critic and creator in a way that is both adversarial and collaborative, and would present many interesting questions. That computers can at least search creative space seems beyond doubt -- but they do so often with a signal of resemblance, rather than one of originality.
Creativity is hard to evaluate - but we could imagine an artistic Turing test - if you cannot determine if a work was produced by an AI or not, then we assume that computers can be creative, or at least as creative as a subset of human beings.
One line of criticism is especially revealing: the idea that we can reject computer creativity because it is just a variation on a theme. This is interesting for two reasons: the first is that it is hard to envision exactly what it is that we do if it is not exactly that - explore variations on a theme. The fact that culture grows in genres is evidence of exactly this. The idea that human beings search creative space with a signal of originality, as suggested above, can be challenged robustly by looking at popular music, movies or any other cultural expression. And it is problematic in another way: if the only signal we look for is the lack of resemblance with anything created so far - well, that seems a very weak criteria for calling something creative.
Human creativity is truly astonishing and wonderful. It is also often a variation on themes, carefully curated and selected from many failed attempts.
That leaves us with the last outpost: consciousness. But when we say that a computer never will be able to be conscious, we are essentially just saying that we do not believe that a computer will ever have this quality that we believe we have, but are not sure how to describe. This is a bit of a let down, to put it mildly. And it may well be that consciousness is not a scientific term at all.
The contest so far, then, is not conclusive and neither is it helpful in establishing if there is indeed a Difference - and that is exactly what we should have expected from our exploration so far: an ill-formed question based in a slowly unfolding identity crisis is unlikely to be a good basis for scientific exploration.
Strategies for attack and defence
Let's look at this from another direction. What if we wanted to attack or defend the idea that there is a clear difference between what humans and computers can do - how would we do that? What are the most common strategies? And what can they tell us about the problem?
If you want to establish that there is nothing that humans can do that computers can't, your strategy is simple: you reduce man to a machine - a wonderful, but replicable machine. This can be done through the denial of the existence of consciousness or equating consciousness with an emergent phenomenon of some kind. You may even suggest that humans are not conscious, but project consciousness through the complexity of their brains and that any sufficiently complex system acts like a lens for universal consciousness. This version of panpsychism comes with a thick ontology of its own, but is embraced by some philosophers.
The perhaps most potent question in your arsenal if you want to deny that there are things humans can do that machines cannot, is to simply ask "well, how do we do it then?" The fact that evolution solved the task of making it possible for us to do something seems to suggest that all of our abilities are derived from algorithmic search through material space - and so why should not machines be able to replicate that search?
If you take the opposite side and want to argue that there are things that humans can do that computers cannot, you can start with this assumption and suggest that we have not understood that evolution's search for intelligence actually built intelligence not out of individual minds, but minds embedded in environments and bodies, interacting.
This is one of the key arguments in Dreyfus, by the way, that we have not yet looked at. Dreyfus rejection of the idea that computers can think is partly based on them not being in the world in the same way we are - and so not capable of intelligence. This argument is one that some of his followers have used to extend his criticism to modern AI - suggesting that we have profoundly misunderstood the evolutionary production of intelligence as a human behaviour that we can decouple from our environments and being. Intelligence, under this assumption, is not in the genotype but in what Richard Dawkins calls the extended phenotype - in the environment we have share and the tools we have created.
This argument also denies that there are computers - separate from us - and suggests that what we should be studying are the extended minds that are now evolving as a mix of humans, machines, organisations and other actors in a complex network. For these critics, the idea that there is a difference between humans and computers is simply a conceptual mistake: to be human is to be embedded in the extended phenotype we have built and that contains computers. This line of criticism is sometimes expressed in something pithy like "Will they replace us? We will become them."
Other strategies that connect with the idea that we have misunderstood the evolution of intelligence suggest that there is some kind of mystical quality in intelligence that we do not understand yet. The prime candidate - as suggested by Roger Penrose and others - is something in the quantum realm.
The brain evolved in a universe where quantum effects matter, these critics say, and so we should expect that it evolved with a reliance on those effects just as it relied on sugar for nutrition. This means that human intelligence cannot be replicated unless we also find out how to interact with the quantum physical effects that power human intelligence (or more often in these critiques: human consciousness). This line of criticism is often backed up by noting that we are starting to see examples of other animal capabilities that are connected to quantum physical effects - like the ability of birds to navigate.1 Some researchers even argue that they have found evidence for entanglement with the brain.2
Finally, there is a line of attack that essentially denies that there is a comparison to be made. The brain as computer is a metaphor, and to try to build from the metaphor into an actually equivalence is merely to be under the thrall of the metaphor to such a degree that we do not longer see that it is just an image. Of course computers cannot do what we can, this line of criticism continues: brains are not computers and software is not intelligence other than in purely metaphorical ways. The brain has brain waves, pink noise, is made of neurons and we do not even half understand it - so the idea that we can replicate what it does is nonsense.
The challenge here, of course, is to explain how the metaphorical chess played by a computer differs from the chess played by a human.
All of these strategies have their own merits and flaws, but they all solidly keep us within the thick ontology of the Difference. We get stuck in this badly anthropocentric perspective where the merits of what we can do with computers is judged by the comparison with the human mind - a weird and strange situation, that may actually be retarding progress.
Not only that - the Difference also can be dangerous.
The idea that we should compare man to machine is one that contains the seeds of a distinct anti-humanism. Earlier comparisons of animal and mankind with automatons have not ended well. Descartes' defence of animal experimentation on the basis that animals are mere automatons, and his rejection of rights for indigenous people who could have no soul because they were out of reach for Christ, and so were also automatons, are examples of what happens when we start to anchor our human worth on not being machines.
At a smaller scale the Difference inspires us to confuse, as DC Dennett has pointed out, competence and comprehension. When we see a computer being able to do something, we often erroneously think it understands what it is doing, so we are prone to make the mistake of delegating authority to early to the machine.
The Difference reduces our own value and exaggerates that of the machine.
Back to Dreyfus
So, we may say, what is Dreyfus answer? Has not his entire book shown us that he is also stuck in this last piece of the thick ontology that he rejects? That he is obsessed with showing that machines are incapable of being human?
In fact, his conclusion, already in 1965 - and then repeated in the 1972 book - is much more interesting. Here is how Dreyfus concludes his critique:
If the alchemist had stopped poring over his retorts and pentagrams and had spent his time looking for the deeper structure of the problem, as primitive man took his eyes of the moon, came out of the trees, and discovered fire and the wheel, things would have been set moving in a more promising direction. After all, three hundred years after the alchemists we did get gold from lead (and we have landed on the moon), but only after we abandoned work on the alchemic level, and worked to understand the chemical level and the even deeper nuclear level instead.
Dreyfus’ answer is not to get stuck in the question of what computers can't do - but to reject even that last piece of thick ontology and ask if we are making a fundamental conceptual mistake. He suggests that we find what is to intelligence as chemistry is to alchemy and look for another set of concepts, another language, to describe what we are doing.
We would, in Dreyfus view, be much better of trying to build alien intelligence or replicate animal intelligence or even jettisoning the idea of intelligence entirely in favour of a new concept - perhaps something like cognicity - that removes the Difference from the research program.
Dreyfus recommendations would imply that we should drop the benchmarks that distract us and instead focus on simpler, thinner ontologies around tasks and interactions. What we need, he would suggest, is a conceptual shift into that simpler ontology - at least if we want to make real progress.
The shift into modern machine learning seems to prove that when the thicker ontology was abandoned we did discovered things obscured by the idea of man as a symbolic animal.
What if we abandon both the idea of man and machine?
Thanks for listening.
Nicklas
This is still under debate - see Ritz, Thorsten. "Quantum effects in biology: Bird navigation." Procedia Chemistry 3.1 (2011): 262-275.
I wish I'd taken a course from Dreyfus, especially on this topic, but aside from the occasional seminar he only taught Heidegger and lower division courses in the early 2000s. Here's his course list for the last decade of his life (https://philosophy.berkeley.edu/people/courses/12). It's interesting in hindsight that even with Searle still teaching, there was no 'AI' course at Berkeley. I suspect they all viewed it as an old and settled debate and subsumed under Mind. I wonder if that's changed!
The quest for thinning our ontology to unlock paths forward feels so resonant. In platform engineering, it's a question of opinion: https://glazkov.com/2022/02/23/the-cost-of-opinion/