Validation Deja Vu

In spite of being among the most critical hiring tools in the recruiter’s toolbox, validation remains largely misunderstood and unappreciated. Failure to validate hiring tools seriously harms both applicants and organizations. Applicants are harmed when qualified people are overlooked. Organizations are harmed when they hire unqualified people. In this article we’ll take a look at some of the most common misconceptions regarding validation ó and what the real facts are behind them. Misconception One: Validation is an abstract concept that has limited usefulness. Fact: “Validation” is proof your hiring tool (whether it be test, interview, resume, or application blank) accurately predicts job performance. If you have not validated your test, you have no assurance it predicts job performance. Misconception Two: Interest and motivation tests are good predictors of performance. Fact: Interest and motivation test scores are generally weak predictors. This is largely because their scores are only symbolic representations of future job performance. Test symbols are the “tea leaves” of performance forecasting; they are not actual demonstrations of job skills. Consider this. Which method better predicts pilot skills? 1) watching an applicant “fly” a flight simulator, or 2) evaluating scores on a written pilot’s test? If you answered “2”, you should seriously consider a career in professional lawn care. If you answered “1,” you already know what expert studies have shown. Real-time performance in job samples is one of the best predictors of ability. In other words, if an applicant “trashes” a simulator, you can be pretty confident they won’t be able to fly a plane. The more abstract the test, the greater the “leap of faith” required to accurately predict performance. Misconception Three: Personality tests are good predictors of job performance. Fact: Nope. Personality tests generally fall into two groups: 1) those specifically designed to predict job performance, and 2) everything else. If your personality test does not fall into the first group, you are on your own. Personality tests only work when they measure traits important to performing specific job tasks. Personality tests designed for hiring are in a totally different league than tests that measure things like communication style or personality type. Job-related personality is highly job specific and tends to change with both task and job. In one of my past studies, I discovered that one popular generic personality test was correlated with stock sales, but not training program sales. Why? The test was pretty basic. It was developed to explain general personality, not job performance. By accident, two of its general factors aligned with stockbroker traits, but not with training program sales traits. Using personality tests as an effective hiring tool takes expertise. Misconception Four: Personality scores predict ability. Fact: Not even close. Personality tests are significantly different from ability tests. For example, only chance would allow an applicant to fake the answer to the following question: “Assume six sharp sticks cost fifty cents. Each sharp stick has a diameter of .125 inches and your ear canal has an opening of .75 square inches. You plan to stick as many sharp sticks as possible into your ear canal so you can be excused from taking the rest of this test. How much money will you spend on sharp sticks before having to call 911?” This kind of question takes some real brainpower to solve, and the answers are not easy to fake! However, consider the following personality test question: “You are sitting in a business meeting waiting for it to start. Do you: 1) ask your boss to polish your shoes, 2) after hearing that a new employee once wrestled for the WWF, state your opinion that pro wrestlers are closet homosexuals, 3) admit that you occasionally wear ‘a little something from Victoria’s Secret’ under your business suit, 4) quietly engage in polite conversation until the meeting starts?” If you get my drift, you already know that people tend to screen answers to personality tests (unless they are incredibly dull, in which case they get what they deserve). Studies show the relationship between personality tests and abilities range from a low of 2% (mental ability vs. mental preference scores) to a high of 8% (interpersonal ability vs. interpersonal preference scores). Any hiring manager who makes mental ability decisions based on personality data should not be surprised by the number of dull employees hired. Misconception Five: You can trust a test vendor to sell only validated tests and be willing to defend these tests if challenged. Fact: Santa Claus and the Tooth Fairy wish you “good luck.” Validation is the responsibility of the test user, not the test vendor. Validation is the user’s documented proof the test predicts performance. Employers always know more about the job than any test vendor. They need to use this knowledge to insist that the vendor conduct a study that shows a hiring test is an accurate predictor of performance. Misconception Six: Hiring tests are generally chosen based on thorough analysis of the job. Fact: Right, and Elvis is alive and living in Cleveland. In most organizations it works like this: Someone gets excited about a specific test. The next thing you know, the test becomes part of the hiring criteria. No job analysis, no study of effectiveness, nothing. Just someone’s “gut feeling” that the test measures something important. Using silly tests are what generally get organizations into trouble. This not only causes problems like legal challenges, it perpetuates the problem of hiring people with wide performance differences based on unvalidated data (you’ll recall my past article on what performance differences cost). Using an unvalidated test to hire people is like hiring people based on shirt size. Misconception Seven: You don’t have to worry about being sued. Few hiring cases get to court and employers almost always win. Fact: There have only been about 100 court cases in the last 40 years and employers have won most of them. Now let’s talk about settlements that never made it to court. Check out the following URL before you tell me you don’t have to worry about following the Uniform Guidelines. Nah! You don’t have to worry about losing in court, you can always “settle.” Recommendations In the last few months, I have been asked to look at the so-called validation data of several popular web tests. All I can say is, people who use these products will have more success winning stuffed teddy bears from a carnival booth. One vendor “validates” tests by “matching” scores to an average of high producers. They call this an “alternative” method of validation (most test professionals would call it something else). I don’t know what school that vendor attended, but “matching” is not validation. For one thing, job matching assumes that all low producers are different from high producers and that all high producers are identical. Anyone care to personally “check out” that assumption in his or her own organization? A good hiring test is supposed to predict both high and low performance. Any vendor suggesting a profile-match strategy doesn’t know the basics of hiring technology ó run away! Another vendor sells a test that seems to measure sociopathic behavior. Not a bad start, but sociopathy is only one factor among several that have been associated with job performance ó the vendor left out eight others that are equally important. This vendor’s validation manual is also a piece of creative fiction, using words such as “this proves…” It excludes statistical data to support claims, and often uses incorrect definitions for validity. Worse yet, the manual cites names and credentials of so-called “researchers” to bolster its credibility. This validation report represents the worst of all shams ó one cloaked in fake respectability! (By the way, screening people based on sociopathy will eventually attract attention under the Americans With Disabilities Act). Reputable test vendors will always suggest conducting validation studies for you. They will rate jobholder performance and compare it with test scores. The reports will be dull and tedious, use arcane terms, have lots of tables, and be filled with boring definitions. But they will report truthful data and keep you prepared for trouble. Reputable vendors will also suggest studying your job before “borrowing” validation data from another job. As a rough rule of thumb, the more “intriguing” the hiring test content, the more likely it will be baseless pop psychology. Remember, a good hiring system and a legal hiring system are the same thing. They just take a little more front-end work.