The Dark Secret of Analytics that Shouldn’t Be a Secret

CEOs get excited about HR analytics because they hope it will reveal something magical. They’ve all seen Moneyball and figure that if HR gets a data scientist some shocking insights will emerge. The fact that shocking insights emerge only rarely is the dark secret of analytics.

What happens when you apply analytics to a problem? You typically discover one of four things:

Confirmation of insights: Something that you suspected was true (or false) is indeed true (or false).
Unhelpful insights: Something you hadn’t thought of but is unactionable or has too small an impact to matter.
“It’s complicated” insights: The analysis uncovers a tangle of complications that could take years to properly unravel.
Incorrect insights: Something that isn’t true at all but appears to be in the analysis because you’ve looked at so much data that random correlations are bound to show up. (See “Is Your Data Digging Turning Up Fake News?”)

This doesn’t mean analytics isn’t useful, just that you need a clear sense of what it is likely to achieve. You’ll find analytics an excellent tool because it:

Creates confidence & consensus: Creates confidence that something we suspected might be true, really is true. This allows us to generate consensus and then start doing something that will work (or stop doing something that doesn’t work).
Establish magnitudes: Gives us a sense of magnitude of an effect so that we can prioritize what things we do.
Provides clarity. Takes a vague idea and makes it precise (e.g. we may suspect “more” vacation days would reduce stress for “many” people; analytics can help define how much vacation is needed and who would benefit).
Reduces errors. It helps us avoid a mistake, not because of a magically surprising finding, but because of something we overlooked.

And yes, just on rare occasions, it will turn up something genuinely new.

Rob Vollman, author of Stat Shot: A Fan’s Guide to Hockey Analytics says “Analytics serve best as a sober second thought to challenge or confirm your traditional opinion.” It’s a way to test beliefs, Vollman says. “Most of the time it will confirm them. The numbers usually line up with what we think. If they don’t, that’s where it gets interesting.”

The dark secret of analytics is a problem because it sets expectations that are impossible to achieve. It also puts analytics professionals in the uncomfortable position of having to dress up their useful but non-miraculous findings as if they were much more shocking than they really are. The lack of magic in analytics remains a secret because it’s easier to pretend it’s true than to try to explain the subtleties of analytics to the CEO. It may also be necessary to make unrealistic promises of miraculous findings to get the funding necessary to launch the initial analytic efforts.

I think we should have reached a point where we can reveal the dark secret of analytics. We should be confident in the value analytics does add. If analytics helps us reduce unwanted turnover a couple of percent, or detect some unduly risky incentive schemes, or convince the leadership team that we need two more specialist recruiters if we intend to grow the business as planned, well all those are excellent outcomes. Analytics in business or in sports is unlikely to be magical, but no sensible owner would skimp on it, it’s an essential tool.

Postscript: The best way to minimize the problems of analytics is to re-frame it within the broader world of evidence-based management. A topic I’m always happy to discuss.

* * *

Special thanks to our community of practice for these insights. The community is a group of leading organizations that meets monthly to discuss analytics and evidence-based decision making in the real world. If you’re interested in moving down the path towards a more effective approach to people analytics, then email me at dcreelman@creelmanresearch.com