Are the Best Decisions Instinctual, or Data-Driven?

November 12, 2015

How do you develop products? Do you blindly trust your instinct, or do you hide behind a mountain of data to make decisions? Sure, these are two extremes, but I am always perplexed by the polarizing aspect of what is, in fact, a false dichotomy. Why do we have to choose one at the exclusion of the other?

Instead, I argue that you need both. Building great products requires you to be a master of instinct and data-driven decision making. Ignoring either one will inevitably lead to failure. You may have one-off successes along the way, but you won’t escape the inevitable.

For the sake of argument, let’s first assume that you can build great products and innovate by experimentation alone. Within this approach, all innovations or product iterations would be tied to the ability to experiment on them. When I was growing up, a Walkman was a great product and innovation! Today it has disappeared, and my kids only experience it as a throwback device in Guardians of the Galaxy. After the first model, the product got better and better. Different companies fine-tuned its shape, buttons, headphones, weight, etc. It just got better and better, only to finally disappear in the wake of digital MP3 players.

Its coup de grace was the advent of the iPod. If the iPod is the “natural” extension of the portable music player, could you have experimented your way from the Walkman to the iPod? I argue, no. In the space of portable music players, a Walkman is and always will be a Walkman. The parameter space (the set of things you can experiment on) that you could explore starting with the Walkman would never lead you to an iPod. To paraphrase scientific philosopher Thomas Kuhn, the iPod is a paradigm shift.

At the other extreme, relying on an instinctual drive is similar to randomly drilling holes in the desert in order to find water. You may be lucky and actually find water, but more often than not you won’t. Instinct will either tell you to keep on digging forever at a given place or to jump to another random spot.

But if you start seeing the desert as an information rich environment – start noticing the contours of the land, or where certain plants are growing – it will provide data for your instincts and tell you where water is likely to be. At that time, you will start your methodical exploration by drilling experimental holes and tracking the results that either confirm or refute your hypotheses.  

Albert Einstein once said: “Imagination is more important than knowledge. For knowledge is limited to all we now know and understand, while imagination embraces the entire world, and all there ever will be to know and understand.” Instinct is of similar nature to imagination, while experimentation is more akin to knowledge. Without working from a gut feeling you can’t wildly innovate, and without experimentation or A/B testing you can’t really fine-tune, learn and grow your knowledge. From an optimization perspective, an A/B testing platform is essential to achieve local optima. At times, however, you will need instinct or gut reaction to take you out of a local optima to potentially achieve a global optimum.

There is a symbiotic, iterative loop between instinct and experimentations. Experimentation powered by a flexible A/B testing platform (to validate hypotheses) will ultimately fine-tune your instinct over time.  Instinct in turn will inform hypotheses your platform will test.

  • Some members of the XLNT team

Some members of the XLNT team celebrating their accomplishments (left to right): Jie Bing, Nanyu Chen, Weitao Duan, Michael Zhao, Bryan Chen, Guangyi Zhou, Shaochen Huang, Kylan Nieh, Bo Liu, Chinman Yip, Steve McClung