Advertisement
There’s More to a Test Than Questions
Get articles like this
in your inbox
The longest running and most trusted source of information serving talent acquisition professionals.
Advertisement
Related Articles
Once a week I get called by someone, usually an administrative assistant, asking if I have a personality test. When I try to get more information, she usually tells me her boss wants the test and she just wants the price.
I tell her, “Sorry. I sell solutions, not tests.”
She hangs up absolutely convinced I am a jerk.
Putting aside the fact she’s probably right, let’s examine her request.
When bosses want to buy a hiring test, it’s usually because they want to reduce hiring mistakes. With typical don’t-know-what-they-don’t-know aplomb, they send their hired help on a fishing expedition to find a test to “fix” what they don’t know. Then things get problematic.
Why is it so important to make a distinction between selling tests and selling solutions? Because selling tests benefits the seller; selling solutions benefits the buyer.
Testing falls along a continuum:
Buy-a-test people are like patients who walk into a doctor’s office expecting to be treated but not saying where, why, or how it hurts. It’s hard to recommend a solution when there’s no clear understanding of what needs fixing.
Best practices start with having a sound theory of performance behind the test. This is psycho-babble for: it’s always a good idea to test for something that makes a difference between high and low performance. Basically, this means if you use a test like the DISC or MBTI to make better hiring decisions, you better have hard data to support your decision (especially since reputable vendors of both tests recommend against using the DISC or MBTI for hiring).
Putting people in little style boxes may be fun stuff, but it takes homework to prove XYZ style actually predicts job performance (it usually doesn’t).
Are theories behind the DISC, MBTI, and MMPI based on job performance? No. Read their manuals. These tests describe personal differences, not job skills. Don’t make hiring decisions based on any test score unless you could produce data proving it predicts performance. And if the test authors advise against using their test as a hiring tool, then it’s probably a good idea to follow their recommendations.
What if the authors suggest only using the test to help make an informed hiring decision? Huh? What part of “testing” is unclear? If a score is used to “help” make a hiring decision, then they are being used as a test.
Ok, suppose the test is backed with a sound job performance theory?what’s next? Face validity, reliability, and internal construction.
Face validity means the test questions and format should resemble a legitimate test. That is, the test should never include invasive questions about the subject’s personal lifestyle choices (e.g., do you belong to a spouse-swap club?) nor questions that have little to do with job performance (such as asking the person to stand on their head and sing The Star-Spangled Banner in a public place). Not only is a lack of face validity bad test practice, it makes the organization look silly and unprofessional.
Reliability is more complex. A good test should produce similar results time after time, like a bathroom scale. If you weigh yourself on five consecutive days, you would expect your scale to produce similar results every day, not vary between 100 and 200 from one day to the next. Likewise, a hiring test should produce roughly the same results from one time to the next.
Reliability also applies to interviews. If three interviewers asked the same questions of an applicant and they heard dissimilar answers, then the interview can be considered unreliable. There are many popular tests that produce unreliable results. How do we know this? University researchers are paid to run investigations.
One sure sign of unreliability is short tests that produce long narrative reports, or long tests that try to measure too much, or tests that rely on opinions, or complicated scales.
If you’re certain the test is based on a solid theory of job performance and it’s reliable, then you are ready to determine whether it actually works. This means gathering numerical evidence that good scores actually predict satisfactory job performance and bad scores don’t.
This is no time for personal stories and anecdotes. Vendors and users alike have personal stories about what works and what does not; however, validity requires hard facts and abundant evidence the test works for you, in your job, in your organization. After all, that’s what you really want to know, isn’t it?
Validity is where the rubber meets the road. It is the point of contact between applicant skills and job skills. All the theory and reliability in the world is a waste of time if the test scores do not lead to better hires and promotions. Validity is the “solution” part. No validity, no solution.
In the beginning of this article, I mentioned the elusive search for a test to make better hiring decisions. Here’s the bottom line: Know first what you want to fix. It could be a turnover, training, performance, or a hiring problem. Within each of these areas, the problem could be traced to intelligence, planning, interpersonal, or motivational skills.
Each of these areas has different solutions. In fact, it could even be a bad manager, bad management, or bad environment problem. There is no way to fix a problem if you don’t know what’s broke.
Once you know what to fix, find a trustworthy, consistent way to measure it. This could be a job preview, one-on-one simulation, structured interview question, smart application blank (i.e., an application blank with specific questions developed based on a statistical study of turnover or productivity), biographical data, tests, case study, or demoting incompetent managers. Each method comes with a complete set of pros and cons. Nothing is perfect, but there are some methods that are considerably more accurate than others.
The last step is to make sure the measurement tool works as intended. All of the above work will be wasted unless you know for certain your test(s) deliver trustworthy, consistent, and accurate results.
Pick one: a test or a solution?