Ask and You Shall Receive - Dealing With Hard Facts

In a quote frequently misattributed to W. Edwards Deming, one of the fathers of process improvement, ‘In God we trust. All others must bring data.’. But what if someone brings data and we don’t like what the data tells us?

Dealing with the results of analysis when it doesn’t tally with our preconceptions can be challenging. We often look to confirm some theory or back up some statement with analysis, but instead of bringing validation, the analysis confounds our theory or puts our statement into doubt. This is particularly painful when the data suggests that we might be about to go way over budget, that we have to spend considerably more time on something than we had planned, or worst of all, that we have gone down a complete dead end. It can be even more painful when an analysis that was intended to quickly confirm something instead turns our plans upside down as we rapidly approach a deadline.

There are some dubious techniques to avoid these problems; I once heard of a strategy manager who used to begin creating a presentation by adding titles to the slides outlining exactly what points they wanted to make, and then asked analyst colleagues to find the data to fit the narrative. If they data didn’t fit, the solution was to find new data that did. This approach creates the illusion of real analysis and rigour but really is just an efficient way to torch your credibility with stakeholders.

What to do instead? Borrowing a little from the scientific method, set the parameters and rules in advance, and stick to them. Perhaps take a more thoughtful approach to analytics; are you making a decision that is going to be grounded in data, and therefore comfortable with the possibility that analysis may suggest actions that are less palatable than you would like? Or are you making a ‘data-inspired’ decision; where data is just one of several inputs and can be ignored if needs be? The magnitude of the decision to be made and the impact of the consequences should inform this choice.

Regardless of which route you go, you should try to be as explicit about this in advance as possible. Agree with your stakeholders how much value you are putting on the analysis before you create or commission it. Not only is disregarding analysis because it doesn’t fit your narrative a bad way to make decisions, it also devalues the work of those creating the analysis. Sometimes, analysis will be dismissed as irrelevant because it is not perfect or has some superficial flaw that doesn’t really invalidate the core insight. In these cases, seek objective, independent assessment of the validity of the data to prevent bias. When it comes to making the decision, whether it’s driven by data, analysis or input from others, be clear with your stakeholders whether a discussion is really a discussion, or just a pre-ordained announcement in disguise.

Yet again, this is ultimately a culture issue. There’s nothing to stop business leaders cherry picking data to make decisions, but in those situations, why bother with the data at all? Ultimately, your business will suffer. By remaining open to new information, even if it is badly timed, you are more likely to make better longer term decisions and build the trust of those you work with, and those who work for you. Ask and you shall receive; just make sure you’re ready for what you get.

-- Colm