Balancing Confidence and Uncertainty in Analytics

There’s much that people analytics professionals can learn from how data has been used in the pandemic. One of the most noticeable aspects of the expert pronouncements on COVID-19 is how radically the forecasts have changed. Experts who had the best available data have been all over the map in terms of their position on things like the infection mortality rate.

What are we to conclude from this, and how does it inform our work in people analytics?

The key insight is that there is often a high degree of uncertainty in analytics. The best available data is often not good, and any predictive models are full of unknowns. The epidemiologists giving advice on the pandemic did not mess up — things that we now know are errors in retrospect were not knowable at the time. The obvious takeaway is that we need to be aware of the uncertainty in our analysis and communicate that to stakeholders.

Communicating uncertainty, sadly, is a big problem. Leaders do not like uncertainty. They would much rather get a precise number from the analytics team than get a wide range along with a heap of caveats about unknowns. It’s easier to just give whatever number a model spits out and tell leaders, “Here is what the model says.” Yes, that definitely feels like the easy route, but only until better information comes along and you have to explain that the result was wrong. Then they have to go out and tell their stakeholders that they, too, were wrong.

Where we end up as analytics professionals is recognizing that we need to find a way to communicate something important to leaders that they don’t particularly want to hear. We also should help them communicate that to an audience that is definitely less capable of understanding the uncertainties.

The tools that statistics provide to quantify uncertainty are of limited use in this communication. Leaders and their audiences don’t have the bandwidth to learn the math. The key point is simply that with every communication, the message has to be, “This is our best estimate based on current information, and so we’ll move forward using this.” This should be followed by, “And we will change direction as needed as new information comes in.”

Everyone would prefer certainty, but part of our job as analytics professionals is to educate leaders on how much uncertainty there is and help them effectively communicate to their audience about the implications of that uncertainty.