Innovation is extremely difficult. Both startups and established businesses have extremely high failure rates on new innovations. Billions are wasted as smart people in companies in aggregate build products and services that nobody wants or, if they build what the customer wants eventually, it took too long and cost way too much. This isn’t news. But to make something better you must first diagnose the problem. In innovation, we have the data that failure rates and costs are very high (the patient is sick), but I don’t think we have fully explored and unpacked the issues that make finding good innovation ideas so hard in the first place. My belief is the issues are rooted squarely in the Innovation Paradox, a concept I came up with after my traditional customer research (talking to customers, focus groups, etc.) led me to alarming false positives and false negatives leading to some painful epic failures that I didn't care to repeat!
The paradox states that if the following statements are true:
People are terrible at predicting their own future behavior
People are terrible at providing accurate causal mechanisms for their own past behavior
Then how can we possibly create replicable insights in a timely fashion that can provide a framework from which we can innovate?
But are these two statements true? And to what extent? In this post I will attempt to connect the dots between various non-innovation fields including psychology, social cognition, history and evolution, to try to create a compelling argument for the validity of the paradox.
It turns out we SUCK at understanding ourselves
Nicholas Epley is a Professor of Behavior Science at the University of Chicago Booth School of Business. He studies social cognition—how thinking people think about other thinking people—to understand why smart people so routinely misunderstand each other. He is the author of Mindwise: How We Understand What Others Think, Believe, Feel, and Want. This work summarizes the vast research and state of our understanding around how well we understand ourselves, how well we understand others and how we consistently and vastly overestimate our ability to do both of those things. While this work is not about innovation per se, its implications for innovation are substantial.
Nicholas states that “No psychologist asks people to explain the causes of their own thoughts or behavior anymore unless they’re interested in understanding storytelling. You can ask people what they are thinking or feeling or wanting – the finished product of some mental processes – and expect to get a solid answer, but asking why they think or feel or want invokes nothing but theoretical guesswork.” He continues, “What’s surprising is how easily introspection makes us feel like we know what’s going on in our heads, even when we don’t. We simply have little awareness that we’re spinning a story rather than reporting the facts.”
Storytelling is what separates us from animals
Storytelling is literally built into our very success as a species. In his book Sapiens, Yuval Noah Harari posits that “Homo sapiens rule the world because it is the only animal that can believe in things that exist purely in its own imagination, such as gods, states, money and human rights.” We make up stories and we truly believe them to our core. Doing so allows us to organize in larger and large groups, allowing us to expand well beyond our genetic cousins. Chimps don’t try to talk to a burning bush because it is a vessel for a godly force. We are the only creature that has this ability and by agreeing that the bush talks, we are better able to trust each other, even if we aren’t in constant contact. But what makes humans so successful may have some negative consequences; we build stories about our own lives as well.
Nicholas believes that it is quite reasonable for the mind to have this fallibility in that “We feel in conscious control of our behavior when choosing to watch one movie instead of another, we feel unbiased when telling teachers just how smart our children are, and we know we adamantly prefer one political candidate over another.” Unfortunately, this feeling of conscious control is limited in its introspection. He explains that “…we can only guess at what’s going on inside our heads to construct those conscious experience. We can report feeling happy but are only guessing when explaining why…. And we can feel ourselves thinking through an important decision that lead to an important choice but are again guessing when trying to explain why we chose option A rather than option B.” In other words, we are quite confused as to when we are being accurately introspective and when we are, instead, making something up and believing it in. This is no doubt a sobering proposition.
Let's talk evolution - We also SUCK at understanding others
Mindwise supplies example after example of research experiments by psychologists proving this severe lack of ability for self-introspection. Yet, when we innovate, we are building products for others so we must be able to anticipate what others are thinking which logically should be more difficult than introspection. Relative to other primates, we are quite excellent at this feat of understanding the intentions of others. In fact, in 1992 British anthropologist Robin Dunbar published an article showing that the ratio of the size of the neo-cortex to the rest of the brain increased with the size of the social group typical for a primate. This implies that a big part of what makes us human is our evolutionary ability to understand what others are thinking. Unfortunately, while we are better than other primates, we are still terrible at completing this task with accuracy when trying to intuit anything beyond very simple cues. Compounding the issues, we systematically and grossly overestimate our ability in this area.
In providing one compelling example as proof, Epley describes an experiment that mimics The Newlywed Game where couples are separated and quizzed. One is asked a series of questions about themselves that they could answer with some accuracy like “How would you characterize your sense of self-worth?” The other partner then predicts their significant others answers and also provides an estimate from 1-100 of their confidence that their predictions would be correct. While the predictions were better than guessing alone (they were still poor at best), the alarming gap is that the predicting partner often estimated that their own accuracy rate was significantly higher than their realized rate, sometimes double. If we have such a troubling gap with our own partner, what might our gap be when we are innovating for others? It is no surprise we spend billions building things nobody wants.
Jobs to Be Done interrogation is the best solution today
No doubt this paints a bleak picture for innovation. The Innovation Paradox is indeed very difficult to overcome. Investors, innovators, marketers, and entrepreneurs alike have come up with a variety of processes to try to solve this issue. Some accept a poor starting point and expect to simply iterate and pivot to a result. Some rely on their internal intuition and logic. Some attempt observation, focus groups, shop alongs, home visits, intercepts, and other similar techniques. Some have looked to big data and to behavioral economics to provide answers. But these are fields of correlation, not causation. Ultimately, if one cannot even observe, focus group, or intercept their significant other, how could they expect to do this well with a stranger? False positive and negatives will be plentiful leading to initiatives started on the wrong premise.
By no means am I saying that these practices are a complete waste of time. From time to time, no doubt, an arduously won insight emerges. But if the goal is consistent and repeatable innovation, I don’t believe these techniques can result in a sufficient solution for The Innovation Paradox.
Thankfully, there is a solution that combines the Jobs to be Done Framework with a timeline and forces of progress brought together by an interview that uses criminal and terrorist interrogation techniques. Unfortunately, there is currently no comprehensive practitioners guide but some of the interrogation techniques can be found in this post. That said, I do plan on posting more about these techniques in the future!
I hope you enjoyed this post. Please sign up at the bottom of this page to have new posts emailed to you or shoot me a line if you want to talk about it further.
About me: I am a founder, builder, learner, executive, investor, strategist and innovator. An innovation and new venture expert on designing, developing and launching new products and services. A driver of transformational change through culture, strategy, team health and clarity. A life-long learner, reading a book a week and constantly seeking out new ideas and frameworks. I think big and get stuff done.