Receive daily articles & headlines each day in your inbox with your free ERE Daily Subscription.

Not logged in. [log in or register]

Validation: Practical Information for Staffing Professionals

by
Dr. Charles Handler
Sep 24, 2009, 5:48 am ET

Picture 4It is not a stretch to say that the validation of pre-employment assessment tools is both one of the most important, and one of the most overlooked, aspects of any legitimate pre-employment assessment program.

Validation is a best practice that can provide both critical information about the ROI of an assessment and the documentation required to support its legal defensibility. Unfortunately, proper validation is not the norm when it comes to the use of assessments. While many companies make use of assessments that have been validated in the past or that do satisfy some of the requirements for test validity, conducting the validation work required to fully satisfy best practices and gain an understanding or ROI is often not on even on the radar screen.

When it comes to validation, my experience shows that the biggest stumbling block is a lack of understanding of just what validation is and why it is so important. While the concept of validation definitely has its complexities, it can be boiled down to a few simple concepts which are discussed below.

What is Validation?

Webster’s online dictionary defines the word “validity” as:

….well-grounded or justifiable : being at once relevant and meaningful <a valid theory> b : logically correct <a valid argument> <valid inference>

….appropriate to the end in view : effective <every craft has its own valid methods>

These definitions definitely hold true when it comes to employment testing. Ask an I/O psychologist and he or she will tell you that validation simply means the act of establishing two key things: 1) That anything used to make employment decisions is job related, and 2) That the assessment actually measures what it is supposed to measure (i.e., that the test is “accurate”).

There are a variety of ways to document the job-relatedness and accuracy of a test as a decision-making tool; however, a working understanding of validation should focus on two general types of validation, content validation and criterion-related validation.

Content validation: Quite simply, content validation involves the documentation of the personal characteristics (i.e., experience, education, knowledges, skills, abilities, values, etc) required to perform the job.

At a minimum, claiming a selection measure is content “valid” requires an alignment of test content to job requirements so that support for the job-relatedness of the test can be documented. This means that the job or jobs in question must be carefully evaluated and that the input of subject matter experts (incumbents and supervisors) be used to create a full understanding of the various things that are required for successful job performance.

The process used to establish the job-relatedness of test content is known as “job analysis.” Once information about job performance and related characteristics has been documented via job analysis, selection measures can be mapped out to match job requirements. For instance, if job analysis shows that the job requires fast and accurate typing, then the use of a typing test to hire applicants for that job is assumed to be content-valid based on its relation to job performance requirements.

Content validation can satisfy EEOC requirements for claiming a test is valid (provided that proper procedures were followed). However, settling for only content validation is selling yourself way short. The real value proposition when it comes to validation lies in the evaluation of the ROI provided by a selection measure. This information can only be provided by criterion-related validation.

Criterion-related validation: Whenever possible, the statistical evaluation of the relationship between selection measures and valued business outcomes is desirable. This type of validation is known as “criterion-related validation” and it can provide concrete evidence of the accuracy of a test for predicting job performance. Criterion validation involves a statistical study that provides hard evidence of the relationship between scores on pre-employment assessments and valued business outcomes related to job performance. The statistical evidence resulting from this process provides a clear understanding of the ROI provided by the testing process and thus helps document the value provided. Criterion-related validation also provides support for the legal defensibility of an assessment because it clarifies the assessment’s accuracy as a decision-making tool.

While criterion-related validation may seem mysterious, it has much in common with two more well-known concepts that are used to help find value within business processes: six sigma and business intelligence. Both of these methods require that data be examined in order to help clarify relations between various process components. The resulting information can be used to help streamline business processes and uncover meaningful relationships between various streams of data. The creation of a feedback loop using criterion validation is really no different.

The 1-2 Punch

In an ideal world it is best to have both content and criterion validity evidence. Documenting content validity is a minimum requirement for any pre-employment selection measure; however, content validation alone can’t provide any evidence for the ROI associated with a test or selection measure. Adding statistical validation bolsters the legal defensibility of an assessment and provides insight into ROI. Unfortunately, most companies do not perform criterion-related validation.

There are a variety of reasons for this.

  • Criterion study not possible: A legitimate reason for not conducting validation studies is that there are simply not enough people in the job to allow for a reliable study to be conducted. The greatest shortcoming of statistical validation is the fact that its results are not credible unless the sample size is well over 100. Therefore small companies and jobs with few incumbents will preclude the ability to gather criterion-related validation evidence.
  • Lack of resources: Many companies feel that conducting criterion-related validation studies simply takes too many resources. They are unwilling to spend the money needed or take the time to collect the data required to perform such studies.
  • Lack of understanding: Many companies do not understand the concept of criterion-related validation and are not aware of the value it can provide.
  • Lack of guidance: Test providers often sell tests as being “valid” because they have been validated for jobs similar to the one in question. While this may be true, it is not always an “out” that allows one to avoid doing criterion-related validation. Many test vendors do not provide advice or services related to criterion validation, leaving the user on their own to figure out this part of the equation.

What does all this mean to staffing professionals?

  • At a minimum all measures used to select employees should be content-valid. Failure to document content validity means the legality of your selection measures is compromised.
  • Whenever possible, it pays to conduct criterion-related validation to add additional support for content validation efforts and to gain insight into the ROI of selection measures.
  • When reviewing tests provided by vendors, make sure to understand that a claim that a test has been “validated” can mean different things. Don’t assume that just because the test has been validated in another setting, that it is automatically valid for your particular situation.
  • Lack of proper validation for tests and selection measures can be costly! Citing lack of resources as a reason for not validating is like passing over dollars to pick up pennies.
  • Both types of validation can add value. Content validation provides legal piece of mind and criterion validation bolsters this and provides the ROI evidence needed to build a business case for using assessment.
  • Educate yourself about validation and the options available to you. For more information talk to an industrial psychologist about validation and legal requirements and consult resources such as the Uniform Guidelines for Employee Selection Procedures.

As with anything else, it may take a bit of extra time and resources to do things right, but the extra effort will provide value and piece of mind.

This article is provided for informational purposes only and is not intended to offer specific legal advice. You should consult your legal counsel regarding any threatened or pending litigation.

  1. Joseph Murphy

    Validation as Calibration

    Great article Charles.
    As you mention, the user is often left on their own to sort out the issue of validation. The six sigma and business intelligence are good examples. Another is the actuarial model. Many companies have or engage the practices of actuarial modeling to understand risks, probabilities, and the like. Recruiting science or I/O psychology brings similar practices and outcomes – better decisions.

    Yet another way to consider assessment and validation is measurement and calibration. Assessment is a measurement discipline for a business process called staffing. Validation is the business rigor or calibrating the measurement to your company’s specific staffing requirements.

    In almost every instance, when a measurement device is acquired, it goes through a thorough recalibration when installed, to ensure the accuracy of the measurement. When an assessment is used based upon the publisher’s validation, the user by default accepts the calibration to outcomes defined and valued by another company.

    In the recent book, The Differentiated Workforce, the authors liken this approach to a “me too” strategy. The use of off-the-shelf assessments, validated by analysis at other organizations does little to create or drive a competitive advantage. In fact, it can minimize competitive advantage by helping a firm hire candidates more like the competition and less aligned with their internally unique performance characteristics.

    So next time you are thinking about validation, think about calibration, adjusting the measurement system to your performance criteria.

    Joseph Murphy
    Shaker Consulting Group
    Developers of the Virtual Job Tryout®

  2. Laura Cordera

    Thanks so much, Charles. Please Please Please read this, corporate HR folks. As an agency recruiter with a background in psychology, I cringe when I work with clients who insist on utilizing assessments to evaluate candidates, but have no idea if they are formally valid for the specific position they are being used for. Please don’t rely alone on a good sales pitch from a testing vendor without drilling down into validation procedures and/or performing them on your own. Or fall into the trap of thinking if a test looks and sounds good, then your firm is doing a great job of assessing candidates because you “test” them as part of the process.

  3. Martin Snyder

    Always glad to read your items Charles. Joe, great points as usual: what you note about a differentiated workforce could also apply to technology, production methods, training, etc. when trying to beat the competition: when to do what they do, and when to go another way are THE key tactical decisions in many battles.

    As you know, correlation is a required element of causation, but by itself it does not indicate causation absent a counterfactual. Because human events are so complex, reliable counterfactuals are hard to come by. Even very slight changes in the subject or the environment may result in a practically useless ‘validation’ of the correlation.

    As you guys also know from my past comments, I believe that leadership, markets, and group-dynamics can easily overwhelm the counterfactual constructs used validate either business outcomes or job performance pre-assessments, except a certain subset of roles: those that are both very similar in performance and hired in very large numbers. I think there is almost a complete breakdown when jobs involving creativity, leadership, and judgment are the subject of the tests/validations.

    Because the tactical advantages of improving the quality of hire in creativity, leadership, and teams can be large, I think the misuse of assessment can be costly. At the same time, the appropriate use of assessment for the mass-hire roles can bring good ROI with minor risk of misuse costs (which I believe are under-studied in general.)

    If I/O assessment practices evolve toward testing group dynamics using multi-variant, cross-organizational environmental states for validation (I think that most but not all of the so-called “cultural” differences among organizations are stochastic and should be ignored), the change may expand the appropriate envelope of roles where assessment tests are relatively certain to provide good ROI without risk of actual harm to organizational performance- an envelope that I believe is far smaller than many suspect.

    http://en.wikipedia.org/wiki/Causality – a useful guide to life ;-)

  4. Brad Schneider

    Criterion Validation: Employers Really Can Have It All

    Oftentimes when organizations opt not to conduct a criterion validation study, it’s because they may be unaware of the direct connection between validation and the bottom line business outcomes they desire. However, in reality, criterion validation is often the best path to ensuring such outcomes, as well as stakeholder buy-in, consistent application of an accurate selection process, and legal defensibility.

    Conducting a criterion related validation study (e.g., administering a test to current incumbents, getting a measure of their job performance, and drawing correlations between the two), provides the critical information needed to adjust item weighting to maximize prediction of job performance within the organization. Further, the inclusion of key performance metrics (e.g., overall job performance, sales, customer service, etc.) within the study can quantify the degree to which the test predicts job performance on the key metrics that are often the basis for implementing the selection or promotional process in the first place.

    Criterion validation is a win-win for organizations that employ them. Their ability to increase test accuracy, enhance legal defensibility, drive key business outcomes, and foster consistent use of a highly effective selection process is second to none.

    Brad Schneider, Ph.D.
    Development Dimensions International (DDI)
    http://bit.ly/2sd42W
    http://www.twitter.com/DDIworld

Post a comment

Please log in to post a comment.

Note: You need to sign up for an account on our new commenting system if you haven't already done so — even if you have an existing ERE account. Find out why »

Login Information