Why product managers should not be data driven

Don’t get hit by Goodhart’s Law

The perils of Goodhart’s Law

One of the fundamental flaws of the data-driven approach is that it relies on the unshaken belief in KPIs. After rigorous analysis of the data, you define good metrics to measure that reflect the critical drivers of your product’s customer and business value, and then you tune your product development engine to keep improving those KPIs. Sounds great, doesn’t it?

Baby steps to climb a hill

Another issue with the data-driven approach is that it favors incremental improvements. In general, it is far easier to find small tweaks that have a small positive impact on the KPI than it is to find big changes with big positive impacts. The reason for that is twofold: firstly, you can test small changes much more quickly. Google’s infamous “42 shades of blue” test is a good example. Hardly any effort required, just test different colors until you find one that slightly improves the conversion rate. Success! Donuts all around, we shipped a winning feature.

Solving the hard problems

The best products weren’t built by starting with a KPI and then trying to find ways to improve it. The best products were built by focusing on a problem and banging your head against the wall trying to solve it. Being data-driven can lead to prioritizing thinking opportunistically rather than strategically.

Perfect is the enemy of done

The last major issue with being data driven is that it can lead to slowing down the organization. If decisions aren’t made until sufficient data is available, then decisions will be made later than they should be, on average. This is a bit counter-intuitive at first — after all, product analytics data is usually available in real time, whereas gathering more qualitative feedback from customers requires time to collect.

The solution: Be data-informed, not data-driven

You shouldn’t take away from this that relying on data to make product decisions is inherently bad. In fact, it can be really useful, as stated at the top of this article. Accurate knowledge of how your product is being used is really valuable, so don’t disregard that. However, don’t let the data drive you. Become data-informed, instead. Instrument your product, use that data to inform areas of the product that don’t work as well as others. Use A/B testing if you are trying to optimize something, by all means — if you are trying to find the best-performing copy on your paywall, an A/B test is often the fastest and most reliable way to get results. Also, absolutely do define success measures for product changes and go back after the fact to validate that the changes did what you hypothesized.



Head of Product at RevenueCat; previously at 8fit, Yammer, BCG.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store