I was listening to this interview of Iain McGilchrist. He was explaining how the mechanistic metaphors that we often use in our daily lives could be more harmful than they are helpful. When we think of our bodies in terms of machines, it will consequently trickle down to our reductionistic approach in problem solving.

We’re not machines. And we will completely miss and can’t explain many things if we think of ourselves as such. I think I give at least eight ways in which we differ from machines and look over the evidence in some detail, so I recommend people to read it if they’re interested…..

That statement piqued my curiosity. What are those eight things? Where to read? Now, because:

1. Iain didn’t proceed to elaborate on the eight characteristics in the interview,

2. I didn’t have immediate access to the book (and I was not sure which book Iain was referring to), and

3. the two questions of “what LLMs can and can’t do” are still top of mind for me,

I decided to use this opportunity to launch an LLM experiment and investigate this question: “how reliable are LLMs’ responses when you’re interested in the content of a book that’s relatively obscure?“

For those of you with attention and patience deficiency, feel free to jump straight to Conclusions to find out what I discovered about LLMs’ capability and reliability when querying about ideas contained in books that are not part of the public domain. If you want to go on the LLM-investigation path together with me, then read on.

Enter LLMs

I tried asking Perplexity.ai and Chat GPT (GPT-4o) this lazily phrased prompt: “iain mcgilchrist eight ways in which we differ from machines“

---

Perplexity’s response

This is what Perplexity gave me:

The Master and His Emissary Book on the specialist hemispheric functioning of the brain
Author: Iain McGilchrist
Genre: Psychology, neuroscience, philosophy of mind, sociocultural evolution
Publisher: Yale University Press

Iain McGilchrist presents eight ways in which humans differ from machines in his book “The Matter with Things.” These differences highlight the unique aspects of human consciousness and experience compared to the mechanistic nature of machines. Here are the eight ways:

1. Holistic vs. Analytical Perception: Humans have a holistic way of perceiving the world, whereas machines are designed to analyze and break down information into discrete parts Ref 1 | Ref 2.
2. Context and Relationships: Humans understand the world in the context of relationships and interconnectedness, whereas machines operate based on isolated data points Ref 1 | Ref 2.
3. Consciousness and Self-Awareness: Humans possess consciousness and self-awareness, which allows them to reflect on their own existence and experiences. Machines lack this introspective capacity Ref 1 | Ref 2.
4. Emotional Intelligence: Humans have emotional intelligence, which enables them to understand and respond to emotions. Machines do not possess this capability Ref 1 | Ref 2.
5. Metaphorical Understanding: Humans understand the world through metaphors, which allows for a deeper and more nuanced comprehension. Machines operate based on literal interpretations Ref 1 | Ref 2.
6. Creativity and Imagination: Humans are capable of creativity and imagination, which allows for the generation of new ideas and solutions. Machines are designed to follow predetermined algorithms Ref 1 | Ref 2.
7. Contextual Understanding: Humans can understand the context and implications of information, whereas machines process information based on specific instructions and data Ref 1 | Ref 2.
8. Spiritual Intelligence: Humans possess spiritual intelligence, which is an attribute traditionally seen as distinct from machines. This includes the ability to perceive and understand the sacred and the divine Ref 1.

These differences underscore the complexity and richness of human experience, which is not replicable by machines.

Alright…. These look convincing and reasonable enough. Let’s see what GPT-4o says.

---

GPT-4o’s response

GPT-4o’s response to the same prompt of “iain mcgilchrist eight ways in which we differ from machines“

Iain McGilchrist explores the differences between humans and machines in his works, particularly highlighting how our unique qualities set us apart. Here are eight ways in which McGilchrist argues we differ from machines:

1. Embodiment: Humans experience the world through a body, with senses and emotions that provide a rich, nuanced understanding of reality. Machines, on the other hand, lack this embodied experience and rely on data input without subjective sensory feedback.

2. Consciousness: Human consciousness is complex and multi-faceted, involving self-awareness, introspection, and the ability to experience a wide range of emotions. Machines, even advanced AI, do not possess true consciousness or subjective experiences.

3. Intuition: Humans have the capacity for intuition, an immediate form of understanding that doesn’t rely on explicit reasoning or analysis. Machines operate based on algorithms and data processing, lacking the ability to ‘just know’ something without computation.

4. Creativity: Human creativity involves the ability to generate original ideas, see new connections, and imagine possibilities beyond current realities. Machines can mimic creativity through pattern recognition and recombination of existing data, but they do not have the genuine inspiration or the capacity for true novelty.

5. Contextual Understanding: Humans excel at understanding context, reading between the lines, and grasping subtleties and implicit meanings in communication and situations. Machines often struggle with context, requiring explicit programming to handle different scenarios and often failing in unstructured environments.

6. Moral Judgement: Humans can make moral and ethical judgments, considering the consequences of actions and the well-being of others. Machines lack moral reasoning and can only follow programmed ethical guidelines, which may not cover all nuances of human ethical dilemmas.

7. Purpose and Meaning: Humans seek purpose and meaning in life, striving for goals, values, and a sense of belonging. Machines operate without any inherent purpose or understanding of meaning, functioning purely based on tasks assigned by humans.

8. Interpersonal Relationships: Human relationships are built on empathy, emotional bonds, and complex social interactions. Machines can simulate certain aspects of interaction but do not form genuine relationships or experience empathy.

McGilchrist’s perspective emphasizes the depth and richness of human experience that cannot be replicated by machines. These differences highlight the irreplaceable nature of human qualities in an increasingly mechanised world.

Seems a bit too generic and imprecise to me, but still highly plausible, slightly probable, and commonly sensible.

---

What the book said

Now, for the moment of truth. Let’s compare it with what I managed to find outlined in The Matter with Things:

Meta commentary: I initially searched The Master and His Emmissary for the keyword “not machines“. This keyword-based search did not manage to surface anything that seems to be what Iain was referring to. So I moved on to The Matter with Things. And voila, here it is, in chapter 12:

WHY ORGANISMS ARE NOT MACHINES

1. On-off. First, a machine is static until switched on, and may be switched off without ceasing to exist. Organisms, as Nicholson points out, much like waterfalls or tornadoes, do not have an off switch. The very existence of an organism is, from beginning to end, one unceasing flow of matter and energy…. In biology all is becoming, never just being. No machine model can make sense of this, since a machine must be built before it can be set in motion.

2. Motion vs stasis. When power is applied, one otherwise static and self-contained component transmits energy to another static and self-contained component, and so on, in a linear chain. Then it is switched off, and it returns to equilibrium, where it can remain indefinitely. In an organism, by contrast, what has to be explained is, not how it changes, but how it remains stable, despite constant change on an unimaginable scale. The stable continuance of a stream is owed to change.

3. Non-linearity. … no organism develops as the result of the execution of a sequence of predetermined steps. Each developmental ‘step’ is not simply computable from the immediately preceding one. … Everything does everything to everything.

4. Not one-way action – maybe not even interaction? … machine model suggests a direction of action of one thing on another… In organisms there is never just action without both interaction and mutual construction. Cause and effect in organisms, if it can safely be applied, is never unidirectional, but reciprocal. When an organism interacts with its environment, both parties change

5. The ‘parts’ are themselves changing A machine is made of parts that do not typically alter with their context. A tappet, a widget or a gasket continues its existence effectively unaltered wherever it is put. In an organism, unlike a machine, the ‘parts’ are continually engaged in changing themselves, sometimes radically, depending on context. One of the most obvious examples is that each cell has precisely the same DNA, yet that same DNA results in dramatically different kinds of cells arising, and hence different kinds of tissue, depending on context. … What any molecule does changes according to what is required by the context, and at many levels throughout the organism.

6. The influence of the whole While a machine has clearly defined parts, this is not, then, the case in
   an organism.139 A process arguably has no parts and is, in reality, an indivisible unity. As Scott Turner puts it, ‘integrity and seamlessness seem to be the essence of an organism’. To the extent that one can speak of an organism as having ‘parts’ at all, we find them by dismantling the whole in an inevitably somewhat arbitrary fashion. They are ultimately a product of human attention, a function of the way we choose to attend to the organism for a particular end of our own, and the parts we choose to define change depending on our focus of interest at the time. ….

7. Imprecise boundaries A machine has clear boundaries; a natural system does not. … Processes ‘have boundaries that are fuzzy or indeterminate’, write Dupré and Nicholson:

   Processes are individuated not so much by where they are as by what they do … At no level in the biological hierarchy do we find entities with hard boundaries and a fixed repertoire of properties.

   Whitehead pointed out the artificiality of drawing boundaries to organisms:

   The truth is that the brain is continuous with the body, and the body is continuous with the rest of the natural world.

8. Boot-strapping … the instructions for making the machine cannot themselves be the product of the very machine they are designed to make. Even in a computer, the software is separate from the hardware: the hardware has to be finished, before the software can be extrinsically inserted into it. The code for making the machine is not being simultaneously written by the machine in the very process of beginning to form itself as a computer.

How much did LLMs get right from this list? Obviously not much, if any at all.

Conclusion

N-gram hallucination / statistical-confabulation is still guaranteed for non-digital corpus where the content of a book is not publicly and legally available on the web. And hallucination still can’t be guaranteed to be non existent even for corpus that are part of the training dataset (as they must be somewhat dilluted).

So where do I stand on this? Don’t bother using LLMs to get information about books. Unless:

1. You don’t particularly mind the accuracy and granularity.

2. You have access to the book to cross-check.

3. The book is part of the “public knowledge” — and therefore likely to be part of LLM’s training data (for example, the ideas in Dale Carnegie’s How to Win Friends and Influence People). And even in this case, you cannot know how much of what is returned is actually in the book and what is not. More on this point below.

4. You have read the book and are looking to enrich the ideas you have absorbed from the book with additional adjacent angles that a) the book might have missed, and b) that the LLM might confabulate. The only time you could be misled by words is when you bet your thoughts, feelings, or actions on their truth and validity. But words are just words and you’re free to do whatever with them.

Put another way: navigating a book via LLM is only worth it when:

1. you have read the book and have grown enough trunks & branches to stick the embellished leaves to, or

2. you are planning to read the book anyway

Interestingly, if these two criteria are valid, I’d argue that combining reading and LLM-ing your books will allow you to better absorb the ideas in the books because it allows you to engage with them in a two-way direction and gives you more words to alchemise these ideas.

If you need accurate and reliable book summaries, you’d be better off reading blog posts of book summaries (which from now on will inevitably be poisoned with LLM-generated summaries). Remember, LLMs are not search engines. Use the right tool for the right task.

---

Even RAG is rarely worth it for a corpus that is at a book’s length because you’d need to play with chunk sizes and the different parameters that the LLM supports to make sure it’s not just spewing some statistically adjacent string of plausible words and you’re not losing important points presented in the book.

What’s concerning is that even when I attached the full text of the book to the gpt-4o and claude.ai (the two providers of state-of-the-art-cloud-hosted LLMs), they are still not be able to fully extract the relevant information that I knew I needed from the book.

I find that asking specific questions about certain parts of the book that you’d like to interact with is also less risky than asking a generic questions like, “what is the book X about?” or “please summarise each chapter of book X for me“. Working chapter-by-chapter is also a reasonable approach, but you still need to experiment with chunking and LLM parameters.

But basically, folks, with all the tweaking, tuning, and prompting work needed to get LLMs to properly represent ideas in a book, at this point in time it’s far easier to just read the damn book.

---

Have I just been unable to find the right book, the right genre, the right model, the right technique, or the right prompts to make this work? Maybe, but I don’t think so. Because I’ve observed the same with different books in the past 16 months, using different evolutions and versions of models, across different genres. McLuhan’s, Joseph Campbell’s, Christopher Alexander’s. And more recently, I arrived at the same conclusion when I tried working with the book Elements of Eloquence by Mark Forsyth.

Different book has different level of failz. But that’s exactly the point: you cannot know beforehand how much of what you are reading is made up and how much are words, ideas, or points that the authors actually intended to convey and in what consistent ways will the LLM “fail”. Yes sure, this is an inevitable problem in any form of communication in human language, but this is more dangerous now, as Sarah Konstantin wrote,

… in a post-GPT2 world, instead of “default to humility” (assume that when you don’t understand a passage, the passage is true and you’re just missing something) the ideal mental action in a world full of bots is “default to null” (if you don’t understand a passage, assume you’re in the same epistemic state as if you’d never read it at all.)

We can work with biased results but we can’t work with noisy results. Noisy means there is no consistency or observable pattern. While biased means results are always deviated in a certain way and to a certain degree when anchored against a clear parameter. When the result is too noisy, it’s better to assume ignorance. It’s safer to default to null rather than default to humility.

---

Closing thoughts

These conclusions seem obvious in hindsight:

• The content of copyrighted books are not likely part of the training dataset (sustainably, or in a legal way)

• LLMs will always try to say something. They can’t say they don’t know.

• Sylistically, the words generated by these LLMs always carry authoritative, plausible, and articulate. The concepts are also sort of statistically coherent.

But frankly, knowing the principles of how current LLMs are built, it all tracks. Of course they can’t say they don’t know. Because they do “know” all the words, all the relationships, and all the styles. But they don’t know that they don’t “know” the concepts, reality, and other instance level-truth.

But if you’ve had decent success with books, I’d love to hear your experience.

One open question I have: what about some non fiction articles. LLMs seem to be able to do NLP tasks on it pretty well. At least much better than when it’s dealing with books. Is this a problem that will be solved with near-infinite context-window size? I doubt it, but let’s see.

---

Footnotes

• Daniel J. Nicholson, John Dupre´ – Everything Flows. Towards a Processual Philosophy of Biology-Oxford University Press (2018)