In Big Data, Small Findings Can Have a Big Impact. Or Not

In HR analytics we are often so eager to discover something significant in the data that we hesitate to think about the impact of the finding. For example, analysis of a big data set might show a statistically significant difference in performance between baristas who like jazz and ones who like rock. That’s interesting for recruiting, but if the difference is that jazz loving baristas are, say, 3% faster on cash then it’s not very important.

This issue highlights a difference between analysis in science and business. In science we are usually interested in findings that have a high degree of certainty, so we set a high bar for statistical significance. In business we are interested in findings that have an impact on the business. If one customer gets sick after visiting our cafe, then we will take it seriously even though it’s only a single case, because, on the off chance the coffee really is poisoning some customers (or social media comes to believe it is), then the impact is great.

The challenging thing for people analytics is that the size of the impact in all the social sciences is usually small. Furthermore, if the impact is big (e.g. short people perform poorly at basketball) then chances are the business has already noticed without any fancy analytics.

If it gives any comfort, biology has the same problem with genes. There isn’t one gene for height, there are dozens, each one only having a small impact. If you are trying to genetically engineer basketball players, it is vastly more difficult than just tweaking one gene.

I wish I had a nice solution to the problem of small impacts; however, it’s built into the reality of trying to improve people and organizations. There are a few things we can do:

Be sure to think about how big the impact of a finding is, rather than worry only about statistical significance. Better you point out this aspect of your finding rather than have someone else draw attention to it.
Note that sometimes a small impact makes a big difference. In the 100 meter dash, a 1% difference in speed can be the difference between winning the gold medal and getting no medal at all.
The Japanese have shown with the Kaizen system that if you make a great many small improvements it can add up to a big impact on performance. Don’t turn your back on a small gain if you have a method for getting many of them.
If the business decision is important then even a small improvement will matter. (See my piece on why the analytics story lies in the issue not the data.) Make sure to focus your people analytics efforts on business issues that really matter.

Perhaps it is also worth saying that part of the issue is the expectations of management. They read the headlines about supposedly fantastic results from some analytics project. In our organizations we have to do a lot of work in educating management about how people analytics really works to add value.