It is not a stretch to say that the validation of pre-employment assessment tools is both one of the most important, and one of the most overlooked, aspects of any legitimate pre-employment assessment program.
Validation is a best practice that can provide both critical information about the ROI of an assessment and the documentation required to support its legal defensibility. Unfortunately, proper validation is not the norm when it comes to the use of assessments. While many companies make use of assessments that have been validated in the past or that do satisfy some of the requirements for test validity, conducting the validation work required to fully satisfy best practices and gain an understanding or ROI is often not on even on the radar screen.
When it comes to validation, my experience shows that the biggest stumbling block is a lack of understanding of just what validation is and why it is so important. While the concept of validation definitely has its complexities, it can be boiled down to a few simple concepts which are discussed below.
What is Validation?
Webster’s online dictionary defines the word “validity” as:
….well-grounded or justifiable : being at once relevant and meaningful <a valid theory> b : logically correct <a valid argument> <valid inference>
….appropriate to the end in view : effective <every craft has its own valid methods>
These definitions definitely hold true when it comes to employment testing. Ask an I/O psychologist and he or she will tell you that validation simply means the act of establishing two key things: 1) That anything used to make employment decisions is job related, and 2) That the assessment actually measures what it is supposed to measure (i.e., that the test is “accurate”).
There are a variety of ways to document the job-relatedness and accuracy of a test as a decision-making tool; however, a working understanding of validation should focus on two general types of validation, content validation and criterion-related validation.
Content validation: Quite simply, content validation involves the documentation of the personal characteristics (i.e., experience, education, knowledges, skills, abilities, values, etc) required to perform the job.
At a minimum, claiming a selection measure is content “valid” requires an alignment of test content to job requirements so that support for the job-relatedness of the test can be documented. This means that the job or jobs in question must be carefully evaluated and that the input of subject matter experts (incumbents and supervisors) be used to create a full understanding of the various things that are required for successful job performance.
The process used to establish the job-relatedness of test content is known as “job analysis.” Once information about job performance and related characteristics has been documented via job analysis, selection measures can be mapped out to match job requirements. For instance, if job analysis shows that the job requires fast and accurate typing, then the use of a typing test to hire applicants for that job is assumed to be content-valid based on its relation to job performance requirements.
Content validation can satisfy EEOC requirements for claiming a test is valid (provided that proper procedures were followed). However, settling for only content validation is selling yourself way short. The real value proposition when it comes to validation lies in the evaluation of the ROI provided by a selection measure. This information can only be provided by criterion-related validation.
Criterion-related validation: Whenever possible, the statistical evaluation of the relationship between selection measures and valued business outcomes is desirable. This type of validation is known as “criterion-related validation” and it can provide concrete evidence of the accuracy of a test for predicting job performance. Criterion validation involves a statistical study that provides hard evidence of the relationship between scores on pre-employment assessments and valued business outcomes related to job performance. The statistical evidence resulting from this process provides a clear understanding of the ROI provided by the testing process and thus helps document the value provided. Criterion-related validation also provides support for the legal defensibility of an assessment because it clarifies the assessment’s accuracy as a decision-making tool.
While criterion-related validation may seem mysterious, it has much in common with two more well-known concepts that are used to help find value within business processes: six sigma and business intelligence. Both of these methods require that data be examined in order to help clarify relations between various process components. The resulting information can be used to help streamline business processes and uncover meaningful relationships between various streams of data. The creation of a feedback loop using criterion validation is really no different.
The 1-2 Punch
In an ideal world it is best to have both content and criterion validity evidence. Documenting content validity is a minimum requirement for any pre-employment selection measure; however, content validation alone can’t provide any evidence for the ROI associated with a test or selection measure. Adding statistical validation bolsters the legal defensibility of an assessment and provides insight into ROI. Unfortunately, most companies do not perform criterion-related validation.
There are a variety of reasons for this.
- Criterion study not possible: A legitimate reason for not conducting validation studies is that there are simply not enough people in the job to allow for a reliable study to be conducted. The greatest shortcoming of statistical validation is the fact that its results are not credible unless the sample size is well over 100. Therefore small companies and jobs with few incumbents will preclude the ability to gather criterion-related validation evidence.
- Lack of resources: Many companies feel that conducting criterion-related validation studies simply takes too many resources. They are unwilling to spend the money needed or take the time to collect the data required to perform such studies.
- Lack of understanding: Many companies do not understand the concept of criterion-related validation and are not aware of the value it can provide.
- Lack of guidance: Test providers often sell tests as being “valid” because they have been validated for jobs similar to the one in question. While this may be true, it is not always an “out” that allows one to avoid doing criterion-related validation. Many test vendors do not provide advice or services related to criterion validation, leaving the user on their own to figure out this part of the equation.
What does all this mean to staffing professionals?
- At a minimum all measures used to select employees should be content-valid. Failure to document content validity means the legality of your selection measures is compromised.
- Whenever possible, it pays to conduct criterion-related validation to add additional support for content validation efforts and to gain insight into the ROI of selection measures.
- When reviewing tests provided by vendors, make sure to understand that a claim that a test has been “validated” can mean different things. Don’t assume that just because the test has been validated in another setting, that it is automatically valid for your particular situation.
- Lack of proper validation for tests and selection measures can be costly! Citing lack of resources as a reason for not validating is like passing over dollars to pick up pennies.
- Both types of validation can add value. Content validation provides legal piece of mind and criterion validation bolsters this and provides the ROI evidence needed to build a business case for using assessment.
- Educate yourself about validation and the options available to you. For more information talk to an industrial psychologist about validation and legal requirements and consult resources such as the Uniform Guidelines for Employee Selection Procedures.
As with anything else, it may take a bit of extra time and resources to do things right, but the extra effort will provide value and piece of mind.