Lately I’ve been revisiting this idea that reason is not the best path to breakthroughs. The core argument is that: we arrive at breakthroughs through intuition and imagination, and then we find our way to a proof or an artifact of that innovation through logic and reasoning. Aha-moments are intuited rather than arrived at procedurally.

That statement is certainly is a bold and an unintuitive (ahem) one. Yet the same time, I feel there’s a deep truth to it^[1]. And I’ve heard enough variations of this idea from people I respect that today I’d like to take some time to investigate all these doxa and find out where I land on this.

Just looking at the sentence, “reasoning our way to breakthroughs“, both of these are widely accepted as important things in modern society, especially since the Scientific and then Industrial Revolution. The first as the best method for problem solving and the latter as a valuable goal to aspire to. So to suggest that reasons cannot get us to breakthroughs seems blasphemous. Surely analysis, science, and reason can solve all of our problems? Yes, yes, they are necessary, but they are insufficient.

Intuition tells us where to go. Reasoning helps us get there. Intuition helps us generate hypotheses. Reasoning helps us test (or rationalise and justify 😉) our hypotheses. Intuition gets us 10x results, iterating through reasoning gets us 10% improvements.

Right now if you think “no no, sure, that feels true to me, obvs, ofc intuition is important, any good problem solvers know that. I agree“, I’d suggest you keep reading because I trust there will be interesting and useful bits here that might tickle your gnosis and feed your intuition in the future.

You might also be wondering, “Why is this diagnosis even needed in the first place? What is the point of this piece?“. There are three answers to this:

1. We live in an era where imagination and intuition have been unfairly deemed to be less respectable and less favourable than logic and reason in their contribution to progress, especially in “hard” domains like science and engineering.

2. To call your attention to how much you are already intuiting-then-reasoning (and rationalising^[2]) your decisions. As a consequence of point #1, you might feel more justified and secure in claiming that you honor cold logic and hard reasoning over the “woo woo” and supposedly artsy-fartsy imagination and intuition.

3. I’d even argue that in innovation, well-developed intuition and imagination are even more valuable than logic and reasoning. Logic and reasoning get you faster horse-carts. Intuition and imagination get you automobiles.

To help put intuition and imagination back up there where they belong, in this essay I’ll lay out the different arguments from different people with respectable epistemology that led me to consider this argument more seriously.

Before we move on, let’s be clear that we are talking about breakthroughs In the context of innovation — often to achieve ambitious goals or non obvious results. This includes but is not limited to business innovation, scientific innovation, engineering innovation, and artistic innovation.

I also won’t start by defining^[3] intuition, logic, reasoning, because these are concepts that we can formulate for ourselves by metabolising multiple articulation — the thing that I hope to achieve here at the end of your reading.

Science and philosophy

Iain McGilchrist

Dr McGilchrist is one of my intellectual heroes. Initially I was slightly put off by the many worshipp-y comments on his videos and interviews. But the more I follow his research and attend to the many many articulation of his thinking, I only get his intellectual honesty, eruditeness, and holistic wisdom.

He is most known for the The Divided Brain theory.

In a nutshell, Iain is a British psychiatrist, literary scholar, philosopher, and neuroscientist. In 2009 he published The Master and His Emissary: The Divided Brain and the Making of the Western World. Then in 2021 he published a book of called The Matter with Things.

Both books approach the complex and serious subjects of the nature of life, reality, our existence, and most importantly, them in relation with one another. But they are written in such an accessible way that it is a perfect entrypoint for anyone without formal training in philosophy, neuroscience, epistemology, or metaphysics who wants to familiarise themselves with the researches in these fields.

If you want to ease yourself into the books, Iain ran a series of discourse for each chapter in TMwT (it’s fabulous and Alex Gomez-Marin did a great job at hosting it). And in this one on intuition (chapter 19), he quoted Henri Bergson:

It cannot be too often repeated that one can move from intuition to analysis, but not from an analysis to intuition

In another video interview with Rebel Wisdom, Iain talked about how rationality begins in intuition and ends in intuition

Q: you talked about that rationality begins in intuition.

A: well yes it has to. it begins in intuition in two ways: one is that we we can’t rationally prove why reason is a helpful tool, (rather,) we intuit that it is a useful tool. But secondly, when we begin reasoning, we have to begin from quite a host of assumptions. Science officially claims to make no assumptions, in fact makes quite a few assumptions before it gets started. It has to. That’s not a criticism. The system can’t work without somewhere to start. Because nothing could be supported on nothing. And it also ends in intuition, because the results of reasoning have to be reintegrated into a broader picture. So it’s an intermediary tool. The left hemisphere is an intermediary tool.

Then in this interview with Jordan Peterson, Iain and Jordan spent some time elaborating on intuition, imagination, and scientism. A couple of snippets from it.

Jordan Peterson and Iain on why we shouldn’t be more skeptical of intuition any more than we should be skeptical of science, reasons, or logic

just because we can come up with contrived situations where intuition fails doesn’t mean that intuition doesn’t function properly in a huge range of appropriate contexts

Iain then points at the different constituents of intuition^[4]

I use about eight different categories of things, as possible constituents of intuition, things like: 1) instinct, 2) ready to go knee-jerk reactions, 3) heuristics, 4) prejudices 5) the aha moments, 6) scientific and 7) philosophical insights, and then 8) imagination

From 40:40 to 45:51, they shared examples of how our impression of science as a linear or algorithmic process is often a myth, on intuitive leaps, and Einstein. Pulling a small excerpt here but the whole section is worth listening to for full context.

You’re absolutely right that most of the great discoveries in science and maths were made intuitively through pattern recognition, through seeing gestalt. They weren’t made by following a linear sequence.

This is a point that’s made by George Gaylord Simpson who’s one of the founders of the modern synthesis. He says the scientific method is more or less a fiction, in that, it’s more honored in the breach than the observance. Although it is a useful paradigm to have at the back of your mind for a lot of the rather plotting early work in science and reason.

… and in the book i look at so many examples of how this is true, that what we need is this broader combination of intuitive work and more routine humdrum work … and einstein himself famously used to say that it took him a long time to explain in words how he reached conclusions that he found came to him sometimes while playing music

Lastly, on another interview, Iain remarked that progress in philosophy is more akin to gestalt:

It’s more like seeing a Gestalt, seeing a form, than it is culminating an argument in which “step A leads to B through to Z and then you switch off and say, right, proved”. That’s never going to be a success in philosophy in my view, I say rather controversially, because the world is not like that.

Kenneth Stanley

Professor Kenneth Stanley is a notable figure in the field of AI^[5]. In his book, Why Greatness Can’t Be Planned, one of his key arguments is that objectives are harmful, especially when we talk about ambitious goals. If you’re looking to innovate, then objective-based approach is the least helpful approach.

Logic and reasoning follow from objectives. The more interesting questions now are: Which parts of the cognitive processes set the objectives? And how to set good objectives?

In this interview with Machine Learning Street Talk, he shared his views that the current state of AI research should not be treated as a strictly scientific endeavour and how working in AI requires an equal amount of sensibility in art. Essentially he’s reflecting on this question: is AI only science?

We can borrow his frame here for innovation. Is innovation science or art? But even science will not stand on logic and reasoning only. It needs intuition and imagination.

An artistic inclination can help you as a scientist. This is not new to point these things out, but I’m trying to make a much more literal point: that AI itself, specifically AI with the word Artificial in front of it, is really about art in a strong way. … (to be creating) these new creations, new ways of thinking about things, these were really inspired from a more artistic sense.

And he was making the same point that to kind of say like, “well no if it doesn’t have a certain level of rigor”. Well, in order to get to those those forms of rigor, you first have to have that stepping stone, that inspiration that generates something brand new.

Be sure to listen to the full podcast episode if you’re interested in hearing the fuller context of what he’s saying on this art/science part of the conversation, including many more interesting reflections and points related to the nature of AI and scientific research.

Related concepts

• The premise of Why Greatness Can’t Be Planned reminds me of John Kay’s book, Obliquity: Goals are best achieved indirectly.

• It also reminds me of Saras Sarasvathy’s theory of Effectuation which describes “an approach to making decisions and performing actions in entrepreneurship processes, where you identify the next, best step by assessing the resources available in order to achieve your goals. Effectuation differs from the causal logic, where there is a predetermined goal and the process to achieve it is carefully planned in accordance to a set of given resources. Sarasvathy argues that the causal logic is not suited for entrepreneurship processes that are inherently characterized by uncertainties and risks“. (source. Here’s the official website of the method where you can learn more.

Humanities

GK Chesterton

You can find truth with logic only if you have already found truth without it.

It took me a bit of time to find the full text beyond that compressed catchy aphorism.

Logic and truth, as a matter of fact, have very little to do with each other. Logic is concerned merely with the fidelity and accuracy with which a certain process is performed, a process which can be performed with any materials, with any assumptions. You can be as logical about griffins and basilisks as about sheep and pigs. On the assumption that a man has two ears, it is good logic that three men have six ears; but on the assumption that a man has four ears it is equally good logic that three men have twelve.

…

The relation of logic to truth depend than, not upon its perfection as logic, but upon certain pre-logical faculties and certain pre-logical discoveries, upon the possession of those faculties, upon the power of making these discoveries. … Logic, then, is not necessarily an instrument for finding truth; on the contrary, truth is necessarily an instrument for using logic, for using it, that is, for the discovery of further truth and for the profit of humanity. Briefly, you can only find truth with logic if you have already found truth without it.

GK Chesterton was an English author, philosopher, Christian apologist, and literary and art critic often referred to as the “prince of paradox” as well as “The Apostle of Common Sense”^[6].

This essay by Wesley Chambers elaborates the idea further, but the overarching point remains: logic without reason and logic without pre-logical faculties such as intuition cannot get us to truths — something that any sustainable innovation need. Even though I must say, even logical thinking is not yet a well-developed skill for us as a society.

Adam Robinson

I first landed on that quote by GK Chesterton from Adam Robinson whose worldviews, thinking, and speaking I have found to be consistently expanding my own sense of the different forces that make the world the way it is.

Adam is a US Chess Federation life master, the Co-Founder of The Princeton Review, and a NY Times Best Selling author. He uses words like “Wizard. Global financial shaman. Pathfinder. Realm-Bridger” to describe himself. I reckon this can be challenging for the more rationally-minded to properly consider the substance of his messages, but so far I find I learn a lot more through taming my knee-jerk reactions. His writing and speaking style is unique and they are all worth pondering.

This Twitter thread^[7] essentually calls us to be more aware of how limited our approaches, tools, and faculties are in terms of approaching Truths.

https://x.com/IAmAdamRobinson/status/949641301028364288

Applying it to business, tech, and engineering

Peter Thiel wrote in Zero to One:

As a good rule of thumb, proprietary technology must be at least 10 times better than its closest substitute in some important dimension to lead to a real monopolistic advantage. Anything less than an order of magnitude better will probably be perceived as a marginal improvement and will be hard to sell, especially in an already crowded market

Google[x] director Astro Teller in this article written by Shane Snow for his 2014 book, Smartcuts:

It’s easier to make something 10x bigger than 10 percent. With higher goals, you can get to radically better solutions in about the same amount of time and resources. In order to get really big improvements, you usually have to break some of the basic assumptions and, of course, you can’t know ahead of time. Incremental progress depends on working harder. 10x progress is built on bravery and creativity instead.

Elon Musk, told by Shane Snow in Smartcuts:

Elon Musk calls this “getting to first principles.” In the 1800s 10 percent style thinking for faster personal transportation translated into trying to breed stronger horses. First principles would suggest instead thinking about the physics of forward movement, then building up from there, leveraging the latest technology—like the internal combustion engine. Most “innovation” inside industries and companies today focuses on making faster horses, not automobiles.

Conclusion

As expected, the takeaway is that to orient and move ourselves towards any goal we set for ourselves, we need to use all the logic, reasoning, imagination, and intuition that we have (and we often already do^[8]). We should equally take them all seriously, give each their own portion with our best judgement, and use each to validate the other. And they will all feed into each other to keep the same flywheel moving.

It’s almost always both-and, rarely either-or.

Where to go from here?

Why am I exploring this? Ultimately this is my attempt at getting a sense of the nature of and the limits of language, logic, reasoning, rationalisation, intuition, imagination, fantasy, instinct, dream, and their places in science, innovation, and the pursuit of truth.

We have barely scratched the surface on any of these but I hope that this essay piqued your curiosity, make you inquire the way you usually approach innovation, and motivated you to investigate further, or at least, notice more of such instances in your own life (I reckon it’s a good filtering bias to deploy every once in a while).

Yes this essay is more or less is just me pointing at things saying “here, these people say so, so there must be truth in it” but I hope the sum of all of them will land as in your being as more than the parts.

---

Relata and references

This notion of iterating and intuiting towards a goal made me think of these cases.

• Perhaps reasoning our way to innovative breakthroughs is like trying to think our bodies into health?

• The neuroscience of intuition reminds me of Barbara Oakley’s two modes of thinking: focused mode and diffused mode. It made sense that logic and reasoning sit more in the focused mode while intuition and imagination seems to be ignited in diffused mode. Would be interesting to unpack this in the future, especially on how the DMN (Default Mode Network) plays into this.

  • DMN is the neuroscience behind bathroom thoughts and how you are getting answers when you sleep.

  • When trying to remember things, you often can’t recall just by thinking harder or leaning closer. When you’re solving a problem, youre more likely to succeed if you stepped away and letting it come to you as an insight rather than chasing them down the reasoning trail. Sleeping on them is often a legit solution for problem solving.

• To learn more about how experts apply and hone their intuition and instinct, a good intro would be Gary Klein’s Naturalistic Decision-Making.

• Malcolm Gladwell’s Blink also comes to mind.

• Absolutely love Sean McClure’s articulation here “You cannot formalize nature, you can only formalize your intuitions about it.”

• I’m reminded again of how gut instinct is a huge part of our intelligence after recently seeing this tweet by Yohei Nakajima, the creator of BabyAGI. He recently quote-tweeted Hiten Shah’s tweet about improving intuition.

• Can’t end without including the tale of Archimedes’ Eureka, his reportedly perceptive insight.

Related to metrics and quantification, it reminds me of

• Goodhart’s law. When a measure becomes a target, it ceases to be a good measure. Jeff Bezos is known to have said that when we’re optimising for certain metric, it’s easy to forgot the actual goal. The goal is not the goal anymore. And deploying measurement and metrics usually led to them being abused / gamed.

  • And here Cedric Chin goes over methods available in solving for Goodhart’s Law at the organisational level.

• Dan Shipper wrote about The Optimal Level of Optimisation (how to avoid overfitting)

  Machine learning teaches us that too much optimization against a goal makes things go horribly wrong—and you can see it in a quantitative way. When machine learning algorithms over-optimize for a goal, they tend to lose sight of the big picture, leading to what researchers call “overfitting.” In practical terms, when we overly focus on perfecting a certain process or task, we become excessively tailored to the task at hand, and unable to handle variations or new challenges effectively.

---

Bonus: Call a GPT

Because LLMs can be fun some time, I asked GPT-4o to suggest some notable examples of when intuition and reasoning have been the driver of scientific breakthroughs.

What can history teach us about scientific breakthroughs?

Historically some scientific discoveries are believed to have been ignited by intuitive insights which were later validated through rigorous reasoning and empirical testing. While some discoveries are believed to be rooted in reasoning.

Breakthroughs rooted in intuition

1. Albert Einstein’s Theory of Relativity:

   • Intuition: Einstein’s development of the special theory of relativity was significantly influenced by his intuitive thought experiments, such as imagining riding alongside a beam of light.
   • Reasoning: These intuitive insights were later formalised through mathematical formulations and empirical validations.

2. August Kekulé’s Benzene Structure:

   • Intuition: Kekulé famously dreamt of a snake seizing its own tail, which inspired his proposal of the ring structure of benzene.
   • Reasoning: Subsequent structural analysis and experimental evidence confirmed the cyclic structure of benzene.

3. Niels Bohr’s Model of the Atom:

   • Intuition: Bohr proposed his model of the atom after visualising electrons orbiting the nucleus in discrete energy levels, a notion inspired by his background in both physics and intuition about atomic spectra.
   • Reasoning: The model was then mathematically refined and supported by experimental data.

Breakthroughs rooted in reasoning

1. Isaac Newton’s Laws of Motion and Universal Gravitation:

   • Reasoning: Newton’s development of his laws of motion and the law of universal gravitation was a result of meticulous observation, measurement, and mathematical analysis.
   • Intuition: While Newton had intuitive insights, his breakthroughs were primarily driven by methodical reasoning and empirical evidence.

2. James Clerk Maxwell’s Equations:

   • Reasoning: Maxwell’s formulation of the equations governing electromagnetism came from a deep and thorough understanding of existing experimental data and mathematical theory.
   • Intuition: His insight into the connection between electricity, magnetism, and light was profound, yet grounded in rigorous analytical work.

3. Watson and Crick’s Discovery of the DNA Double Helix:

   • Reasoning: The discovery of the DNA double helix was the result of combining existing data from X-ray diffraction images (most notably by Rosalind Franklin) with detailed biochemical knowledge.
   • Intuition: Their mod““el-building approach required creativity, but it was heavily informed by empirical data and logical deduction.

Of course, we don’t ever have the full story of these discoveries. But what I am certain of is, none of these scientists involve exclusively intuition, reason, or logic in their process. All of these are necessary but not sufficient.

It’s worth also considering the concept of “serendipity” in scientific discovery. Serendipity often involves a combination of intuition and reasoning, where an unexpected observation (often dismissed by purely logical processes) leads to a breakthrough when the observer recognises its potential significance. Examples include Alexander Fleming’s discovery of penicillin and the development of the microwave oven by Percy Spencer.

---

Footnotes

1. Perhaps it’s just the recovering rationalist in me fist-pumping

2. Tim Ferriss made a brief remark somewhere in this interview with Rich Roll of how he now has the conclusion and just need to find a way to get there or explain it

3. JBP: how would you define intuition and imagination?
   IM: well they are almost impossible to define and and it’s a mistake to think that we can’t discuss them until we’ve clearly defined them. Often the only way in which we can understand them is by approaching them from different points of view and and working out what they are.

   Which also reminds me of what Ken Stanley said about why he disliked “definitions discussions”.

4. Iain McGilchrist also proposed at least four primary paths of what we can rely on in trying to understand the nature of reality: science, reason, imagination, and intuition. He also encouraged us to consider how science and reason can also lead us to false conclusions, unless they are sophisticated by imagination and intuition.

5. He coined the concept of novelty search, which focuses on exploring novel behaviours rather than optimising for a specific objective. He was also a senior researcher in OpenAI.

6. In case that name sounds familiar, he is behind Chesterson’s Fence.

7. Or, a “tweet storm”, what it used to be called in the old days.

8. And interestingly. In the middle of writing this, I realised this is exactly what I am doing here. I start from an intuition, laid out the different pieces of evidences, let the pieces simmer and percolate at the back of my mind, used logic and reasons to weave them together, then imagined the final form. It’s the Right – Left – Right combo again.