Turning data into a valuable business asset is hard.
For starters, we have a complex solution space: data warehouses, BI tools, stats, ML, and more. There are countless PhDs dedicated to niche details behind each of these! Being able to ‘do’ data science is hard.
Then, even when a company has all the best data people and tech, there's still a tremendous amount of waste! Dashboards with 0 views, unqueried data tables, analyses no one acts on, or ML models waiting to go into prod.
Failed projects litter the world of data.
It pains me to feel it’s more common than not. I’ve seen more data projects fail to be useful to a business than I’ve seen projects that make an impact.
So why is this??
Why are all these amazing projects, from intelligent people, failing to be of value? Well, I alluded to why in the second sentence. Can you spot it?
Problems vs Solutions
Here’s a simple diagram I drew when I first became a data product manager.
This diagram tries to represent that:
There are problems to be solved, and solutions that can solve them
Some problems can be solved by many solutions
Some solutions can solve many problems
Some problems can’t be solved
Some solutions don’t solve any of your problems
Now, I see everything in the world of data on the right: as solutions.
Statistics helps solve the problem of making decisions under uncertainty.
Machine learning helps solve the problem of scaling complex behaviour.
Data warehouses help solve the problem of needing to store masses of data for other work.
As a data specialist, this is an important thing to be aware of: you, and the skills you bring to a business, are solutions.
This begs the question: what the hell are the problems?
The Problem-Solution Gap in Data
This is where our challenge starts: a gap between business problems and data solutions.
The problems come from business leaders. Problems like how to increase market share? How to accelerate the growth of a product vertical? Or how to prioritise among initiatives given many possible trade-offs?
These problems are valid and it seems intuitive that data can help. It is one of the greatest pitfalls of the data world: the perception that data can solve almost anything. Contrary to this, data can't easily solve these problems. At least, not when they're articulated this way. Which now brings us to the heart of why data projects fail.
The Skills Gap in Data
At the heart of why most data projects fail is a skills gap:
Most people don’t know enough about data to define problems in a way that data experts can solve them. OR...
Data specialists don’t know enough about business to connect their solutions to the right problems.
And so what do we end up with?
We get business owners asking data specialists to solve any and all problems. Data specialists who fail to deliver on those. A breakdown of trust and interaction. Data specialists losing a seat at the table. Business owners operating without data again. Data specialists trying to add value by building solutions to problems that don’t exist.
Closing the Gap
Closing the gap requires closing the skills gap. It needs people to connect high-level business problems with data solutions. Here are a few options to think about.
One option is to insert people who can be the bridge. Sometimes they come from data. E.g., it's something I do as a Data Product Manager. They can also come from the business side. For example, business leaders or finance experts who moved into and up-skilled in data.
Another option is to invest in education. Data teams need to help educate leaders on the sorts of problems they can solve. Vice versa, leaders need to educate their data teams about business problems. This two-way education helps brings both sides a bit closer.
The final option I’ll suggest here is to acknowledge the challenge. Be open that you might be on the problem or solution side. Admit there’s a gap and ask for patience as you build the bridge. Collaborate in a very iterative way to maximise continuous alignment.
Give me the summary, Simon!
Alright, summary time.
Data Science and Analytics is a complex field but it’s one of “solutions.” The “problems” to be solved lie somewhere else. There can be a vast gap between business problems and data solutions. To compound the problem, very few people have the skills to bridge this. This is the #1 reason why I believe most data projects fail.
The result? Business leaders and data teams drift apart. The promise of data-accelerated business growth diminishes. Impact becomes something that data teams need to fight for.
If you want to land more impact with data, look at the problem-solution connection. Who is defining the problems and how are they articulated? Can a bridge be built from them to a meaningful data solution? Can the solution be delivered in a way that solves the problem? You can connect them but don't underestimate the challenge.