Oftentimes when organizations opt not to conduct a criterion validation study, it’s because they may be unaware of the direct connection between validation and the bottom line business outcomes they desire. However, in reality, criterion validation is often the best path to ensuring such outcomes, as well as stakeholder buy-in, consistent application of an accurate selection process, and legal defensibility.

Conducting a criterion related validation study (e.g., administering a test to current incumbents, getting a measure of their job performance, and drawing correlations between the two), provides the critical information needed to adjust item weighting to maximize prediction of job performance within the organization. Further, the inclusion of key performance metrics (e.g., overall job performance, sales, customer service, etc.) within the study can quantify the degree to which the test predicts job performance on the key metrics that are often the basis for implementing the selection or promotional process in the first place.

Criterion validation is a win-win for organizations that employ them. Their ability to increase test accuracy, enhance legal defensibility, drive key business outcomes, and foster consistent use of a highly effective selection process is second to none.

Brad Schneider, Ph.D.
Development Dimensions International (DDI)
http://bit.ly/2sd42W
http://www.twitter.com/DDIworld

As you know, correlation is a required element of causation, but by itself it does not indicate causation absent a counterfactual. Because human events are so complex, reliable counterfactuals are hard to come by. Even very slight changes in the subject or the environment may result in a practically useless ‘validation’ of the correlation.

As you guys also know from my past comments, I believe that leadership, markets, and group-dynamics can easily overwhelm the counterfactual constructs used validate either business outcomes or job performance pre-assessments, except a certain subset of roles: those that are both very similar in performance and hired in very large numbers. I think there is almost a complete breakdown when jobs involving creativity, leadership, and judgment are the subject of the tests/validations.

Because the tactical advantages of improving the quality of hire in creativity, leadership, and teams can be large, I think the misuse of assessment can be costly. At the same time, the appropriate use of assessment for the mass-hire roles can bring good ROI with minor risk of misuse costs (which I believe are under-studied in general.)

If I/O assessment practices evolve toward testing group dynamics using multi-variant, cross-organizational environmental states for validation (I think that most but not all of the so-called “cultural” differences among organizations are stochastic and should be ignored), the change may expand the appropriate envelope of roles where assessment tests are relatively certain to provide good ROI without risk of actual harm to organizational performance- an envelope that I believe is far smaller than many suspect.

http://en.wikipedia.org/wiki/Causality – a useful guide to life ;-)

Great article Charles.
As you mention, the user is often left on their own to sort out the issue of validation. The six sigma and business intelligence are good examples. Another is the actuarial model. Many companies have or engage the practices of actuarial modeling to understand risks, probabilities, and the like. Recruiting science or I/O psychology brings similar practices and outcomes – better decisions.

Yet another way to consider assessment and validation is measurement and calibration. Assessment is a measurement discipline for a business process called staffing. Validation is the business rigor or calibrating the measurement to your company’s specific staffing requirements.

In almost every instance, when a measurement device is acquired, it goes through a thorough recalibration when installed, to ensure the accuracy of the measurement. When an assessment is used based upon the publisher’s validation, the user by default accepts the calibration to outcomes defined and valued by another company.

In the recent book, The Differentiated Workforce, the authors liken this approach to a “me too” strategy. The use of off-the-shelf assessments, validated by analysis at other organizations does little to create or drive a competitive advantage. In fact, it can minimize competitive advantage by helping a firm hire candidates more like the competition and less aligned with their internally unique performance characteristics.

So next time you are thinking about validation, think about calibration, adjusting the measurement system to your performance criteria.

Joseph Murphy
Shaker Consulting Group
Developers of the Virtual Job Tryout®