Unpredictable Patterns #90: Search
From listening and reading to searching, search and space, search pessimism and optimism, search landscapes and dangerous allure of a poetry of statements - and then what we should expect next
Dear reader,
In this week’s note we will continue the exploration of language and memory that we have been engage in for two notes in a row. It is a rich subject and I confess a nerdy fascination with it that goes back quite some time. So bear with me, and let’s dig into why search is such a powerful mental model - and why it can have an effect on everything from what we know to how we live together.
Search
In an earlier note we explored the idea that we have moved from an oral to a written to a recorded culture, and that we have left the archive behind for the database. This also means that the way we approach our knowledge of the world is changing - from listening over reading to searching.1 In this week’s note we will examine what that could mean, and what searching really is, and how it changes the way we produce knowledge. 2
With the Internet and search engines, search has become prevalent in our everyday life, and we hardly even think about how extraordinary it is that we can now sit down and consult a world-wide database of information as we are trying to learn about something. In many ways, we have become blind to the magnitude of the shift that we have gone through as a culture. Sure, information retrieval is not new, you can argue, but the way it has become a part of our overall knowledge production now is a change on an order of magnitude that marks a civilisational shift.
Is that an exaggeration? If it is, I would argue that it is so only by a very small margin.
We still move tentatively in this new world, and we even sometimes think that searching cannot really figure in knowledge building - it is too easy, and what do we know about the quality of the information we find? - but it seems clear that something fundamental has happened - in two dimensions.
And it is not just about epistemology - or the philosophy of knowledge - it is equally about something else, more akin to ontology - or the philosophy of what the world is made out of. Searching does affect epistemology in many ways, but it has also shifted the way we understand the world at its most basic level.
The world has now become a shared space that we have the ability to explore in different ways.
Search and space
When we speak of search, we implicitly assume a space to be searched through. This space can be set up in many different ways, and even have many different dimensions - but you cannot understand search without understanding the concept of a space to search through.
We have used “the internet” as shorthand for the space we search in for a long time, but that is really a quite stifling simplification. What we are really searching through is a multi-modal space of text, video, images and sound that we co-create with other people around the world.
This space can be represented in many different ways, but ultimately we are searching through our own living language, or parole, in Saussure’s sense.3 This is no set space, well-defined once and for all, no, it is the site of knowledge production, messy and often under construction, shifting and subject to massive manipulation from a multitude of different participants.
There are so many metaphors we can use here that it is almost overwhelming. The idea of a Library of Babel, as suggested by Jorge Luis Borges, is one: here the search space becomes an infinite combination of all known signs, with all that is true hidden in the vastness of the false and useless. Borges description of how the library was received still remains one of the best descriptions of how we, as a civilisation, have reacted to the Internet:4
When it was announced that the Library contained all books, the first reaction was unbounded joy. All men felt themselves the possessors of an intact and secret treasure. There was no personal problem, no world problem, whose eloquent solution did not exist-somewhere in some hexagon. The universe was justified; the universe suddenly became congruent with the unlimited width and breadth of humankind's hope. […]
That unbridled hopefulness was succeeded, naturally enough, by a similarly disproportionate depression. The certainty that some bookshelf in some hexagon contained precious books, yet that those precious books were forever out of reach, was almost unbearable.
This dual reaction, the joyful hope replaced by the depression when we realised that the idea of a globally shared space for truth would forever be beyond us, gives us a stark image of our own relationship to the Internet. Borges also depicts humans as forever searching the infinite halls of the library, some despairing, others driven by obscure and occult ideas about patterns in infinity that only they can discern.
The space we search in here is in some senses a trap that confines us to searching without ever being able to determine the value of anything we may find. In this sense Borges is a search pessimist - one who believes that the search space we have co-constructed only looks as if it can give us information we can turn to knowledge, but in truth will resist any such attempts at extracting value from the information deserts it contains. He writes:
Infidels claim that the rule in the Library is not "sense;' but "non-sense;' and that "rationality" (even humble, pure coherence) is an almost miraculous exception. They speak, I know, of "the feverish Library, whose random volumes constantly threaten to transmogrify into others, so that they affirm all things, deny all things, and confound and confuse all things, like some mad and hallucinating deity."
Borges metaphor aligns closely with language, and it was his genius to understand that in many ways human knowledge can be represented as the permutations of the letters of the alphabet - and that if we have access to all of those permutations we are equally lost as if we have access to none.
For Borges searching is being perpetually lost in language - but there is hope: the hope is that the Library, he says, may be infinite but periodic. He describes it in a way that manages to capture both the impossibility of searching and the possibility of pattern:
And yet those who picture the world as unlimited forget that the number of possible books is not. I will be bold enough to suggest this solution to the ancient problem: The Library is unlimited but periodic. If an eternal traveler should journey in any direction, he would find after untold centuries that the same volumes are repeated in the same disorder-which, repeated, becomes order: the Order. My solitude is cheered by that elegant hope.
This is a both funny and sad conclusion at the same time. For us, exploring the space we are searching through, it becomes interesting in another way: because the space we have built is not infinite - it is still limited, and it is limited in the way any near living environment is limited: it has a metabolism, it requires energy and other inputs, it gives off heat and it fades over time.
The metabolism of our search space is tied to simple things like the energy needed to power the databases and information repositories we are drawing on, but also the curation of information over time, the format-related forgetting and bit rot that surges through the search space as it evolves. Understanding this metabolism will also help in feeding and curating that space.
But Borges search pessimism should not be left standing unopposed - because there are also search optimists. If search pessimism is the belief that we will slowly collapse under the sheer weight of information and that search will lose its ability to produce knowledge, then search optimism is the belief that we can always structure the search space in such a way that it can help not just produce knowledge, but even produce institutional order, progress and peace.
This might sound a bit much, since we live in an age that is instinctively cynical, but it really is worth taking seriously. Among the search optimists, the most interesting thinker was Paul Otlet. Otlet’s thinking around information and search pre-dates the Internet by half a century at least, and his thinking about the future of the book remains shockingly prescient.5
Otlet’s vision was also that of “a radiated library”, but his library was not infinite - it was structured as a mirror of the world, and it was located in a city of knowledge.
This city of knowledge then evolved into the Mundaneum - a world repository of knowledge, or a search space, that Otlet then spent years trying to build, structure and operate.6 Otlet's vision was an institutional search space that was operated by international institutions and curated by them in the interest of disseminating information that would contribute to progress and peace. Here is how he describes it:7
General principles. A systematic body of regulations will ensure cooperation and coordination between all the forces working for the development of the Book or which are involved in Documentation. An organisation (General Union for Documentation) will keep documentary organisations (members) in contact with one another. It will link them to an executive body (Central Institute) by means of a system of agreements and will ensure their representation in a Congress having all the necessary authority. The organisation will be a federation in which the autonomy of its members will be respected; it will be mixed, bringing together non-governmental and governmental organisations. It will achieve coordination in a two fold way: geographic place and subject specialty. It will create concentrations and relationships at various degrees or levels. It will use what exists as much as possible, but will also proceed by fusion, elimination, and creation (amalgamation and re-organisation). It will take advantage of the best organisational experience, especially that of international organisations.
The tension between Otlet and Borges is the tension we live in today: the infinite library of despair and depression or the institutional source of progress and peace through knowledge. Search pessimism and search optimism as contrasting views not just of how we consume knowledge, but also of how we govern ourselves: and this is increasingly important.
Maybe one possible insight here is that we have, so far, paid too little attention to the organisation of our search space and the curation of information. We have, perhaps, mistaken Borges infinity for a natural state of freedom and dismissed Otlet’s city of knowledge as a utopian dream - and that needs to change.
Thinking in landscapes
There are other metaphors for our search space too - we can view search space as a landscape. This idea of a search landscape sparks a lot of different ideas - we can think of search as the exploration of an unmapped frontier, evolution progressing across a biological fitness landscape or the selection of optimal moves in game landscapes.
We can also explore how this landscape differs from earlier search spaces.
Before the Internet, our search spaces were local and rugged: we searched through a small space where there were peaks and valleys - authoritative sources and gossip - easily uncovered in the search space. One of the great challenges of digitisation was not only that the search space became vast and global, but also that it became flat. Suddenly there were fewer ways to signal and maintain authority in this new space, and the rugged information landscape with peaks and valleys became a single, infinite desert of data. This dimensional reduction - from an at least three dimensional to a two dimensional search space, was a consequence of the shift to digitised text - but it was not an irreversible change.
One way to think about challenges we need to meet in this new world is to say that we need to re-establish a ruggedness, we need to find ways of allowing for peaks and valleys to emerge in our search space, and get away from the flat data deserts.And that brings home another point: our search space is not a given, in the sense the library is found. It is a built space that we can re-imagine.
This is interesting from a policy perspective, since most policy efforts are focused on how we search, and not the constitution of the landscape. The careful and intentional conversation about how dimensions of authority and evidence can be re-introduced is drowned out by a discussion about how searching should be regulated. That is a choice, but it is important to note that the ultimate outcome we want to avoid - the nihilism of Borges’ library - is better sought in a combination of solutions for the search space and the search itself.
This idea, that there is such a thing as a governance of the overall space that we search in, is one to approach with some hesitancy. There are obvious failure modes, like ministries of truth or propaganda peaks. But the trick is not to try to re-introduce a geography of truth in our search landscapes - but a geography of reasonable disagreements.
A space organised around disagreements is a relational space - it is based on the relationship between two different narratives with differing outcomes, and creates a rugged field of possible positions between them. Knowledge is equally built on relationships between the stories we believe and the stories we do not believe - we do not know facts. When we know something we are able to locate ourselves in between disagreeing narratives.8
As we proceed to build out our search landscape we should do so to represent the tensions and disagreements as well as we can, and then allow people to search the resulting rugged landscape.
The space we search in is our intellectual fitness landscape, and it is up to us to design it well.
The poetry of search and the poetry of statement
Dorothy L Sayers, author and translator, once wrote a small essay about the poetry of search and the poetry of statement. Her argument was that there are two different kinds of poetry - one in which the poet is searching for something, for herself or for an elusive meaning, for example, and one in which the poet is stating things for the reader to learn. These different modes of poetry, for her, roughly corresponded to romantic and classical poetics, and she held that both were important.
Her distinction can be expanded to our time, by noting that this is not just true of poets anymore, but everyone who is dealing with knowledge. We can talk about an epistemology of search and an epistemology of statement.
In the first case, knowing is searching - being aware that the answer is always changing and that the search can come up with different results in different situations. Knowledge as searching is not arbitrary or relativistic, but it is relational: what we know is related to what we search for and how we search.
In contrast, knowing by statement is the classical form of knowledge. The world can essentially be presented as a list of facts, and those facts can be accurate or false. In this world your knowledge is absolute and largely unchanging.
It may be felt that searching is a shaky foundation for knowledge, and that if we want to build a just and fair society we need something more like statements - like facts - in order to be able to make decisions, and that is partly fair. The reality is that we have always progressed by using a combination of search and statement - and what we need to do is to perhaps shift more of the mix towards search.
The way we conceive of our knowledge also impacts the way we see our own role in it. A society that sees knowledge as a continuous search in unexplored spaces will see knowledge as a project, whereas a society that believes that knowledge is a set of statements is more likely to approach knowledge as a system.9
We participate in projects, we use systems. Projects build, systems operate. The difference is salient when we look at modern societies - and one of the core problems of democracy today is that it is not viewed as a search, but as a series of statements.10
Reimagining our societies as search might be a way to revitalise our interest in and commitment to democracy then - since the society of the statement increasingly seems to be drifting into authoritarianism.
So what?
What should we expect to see happen with search if these observations are true? Here are a few possibilities.
We should expect to see new forms of search emerge that build on a new conception of the search space and how that is built. The age of the index may be replaced by the age of the model (yes, this is a simplification). This also suggests that the investment formula for how much is invested in the data structures and the design of the search space and how much is invested in search algorithms could change over time (and the search stack could change as well).
We should expect to see new technologies, business models and regulatory initiatives to re-establish a rugged information landscape. The information deserts that we live in are dangerous places, where all narratives are created equal and facts are made subservient to tribes.
We should see new political movements emerge that draw on the idea of our society as a search for new and better ways of living together, and solving our challenges - or else see societies slowly disappear in the disempowered resignation of statements.
We should see a shift in focus in education from statements to search. This is anathema to many classical educators, and highly controversial, but if the basic mode of knowledge in our societies is changing from listening to reading to searching, should we not expect search to become a key skill? In Vernor Vinge’s remarkable Rainbow’s End, two key subjects in school are search and visualisation.11 If we are forced to identify to core skills needed in the future, these are not a bad bet.12
More than anything, search is powerful mental model when approaching anything from creativity to democracy and evolution13. Just applying a search framing to a problem allows us to ask productive questions about the space we search in, local and global optima, evolution as a search algorithm and new dimensions in our problem as well as much more.
After all, we are much more homo perscrutor than we are homo sapiens.
Thanks for reading,
Nicklas
It is probably safest to state clearly that while I worked many years at Google, this is not about any search engine or company, this is about search as a model for knowing in much more general way — and my views are my own here, of course.
Borges, JL The Library of Babel (1941)
See this video. Otlet’s vision of the multimodal Internet is astonishing - and at the same time obvious to us today.
See Otlet, P International Organisation and Dissemination of Knowledge: Selected Essays of Paul Otlet (1990)
Ibid. p 197
There is a deeper point here about how we think the world works: is it composed of single components, or does it exist in relationships between things? Is the world things or relationships? A world that is construed as a set of relationships between things, a network of interacting forces, is very different from one that is atomistic.
See e.g, Calhoun CJ Gaonkar DP Taylor C. Degenerations of Democracy. Cambridge Massachusetts: Harvard University Press; 2022. https://www.jstor.org/stable/10.2307/j.ctv2k4fwtt. Accessed September 25 2022.
Vinge V. Rainbows End. London: Tor; 2011. Accessed September 25 2022.
But will we not need facts and a baseline of memorised knowledge? We probably will - but it may well be new mental models and facts that we need to internalise. The balance between the internalised and the extended mind is always shifting.
The idea of evolution as search is surprisingly powerful. See e.g., Wagner A. Arrival of the Fittest : Solving Evolution's Greatest Puzzle. New York New York: Current; 2014. This is worth its own note at some point.
We gotta chat about "age of the model"!
This is fascinating. Thank you for sharing, Nicklas 🙏