You can fool all the people some of the time, and some of the people all the time, but you cannot fool all the people all the time. — Abraham Lincoln
Honest Abe must have known about hiring tests. You see, the results of a foolish test will always show-up as soon as someone takes a closer look. Foolish tests make bad candidates look good on paper, but crash on the job; and, they make good candidates look bad on paper … but never get a job offer.
Like Abe said, you can’t fool everyone all the time.
I call the difference between good and bad hiring vendors “Black-Hearted” and “White-Knights.” A White-Knight vendor presents thorough research showing its test predicts job performance with a very high degree of accuracy. White-Knight vendors always report jobs, demographics, charts, tables, subjects, correlations, and probability of being wrong. They are very dull. A Black-Hearted vendor presents happy-user stories that are very light on real data and heavy on nonsense numbers.
You might ask why they do this. Well, I guess they might be reluctant to say anything negative; they don’t know how to develop a professional quality test; they don’t know any better; they don’t care; or, a little of all the above.
Right or Wrong, Never in Doubt
I’ve talked to dozens of junk-test vendors. They are all totally convinced their test is the best thing to happen since tax deductions. Their flashy websites often contain pseudo-scientific brain wave charts, claims their training tests are also useful for hiring, and ridiculous stories of client effectiveness. But — and, here is the hitch — a closer look at the miracle test only they were smart enough to develop shows it doesn’t begin to meet professional test standards. Heck. After a few minutes discussion, you find out they never even heard of them! One proud vendor, for example, told me they couldn’t go back and fix their test because the shareholders would never stand for it! I’ll guarantee any client using that test ends up with clones — and not necessarily skilled ones.
Cloning high producers is one thing. Cloning employees with the same personality and motivation can be a disaster.
What’s the Big Deal?
The big deal is no test is 100% perfect, but White-Knight tests are very good at separating good employee/managers from bad ones. In fact, their developers go to great lengths to prove that very thing. Black-Hearted vendors don’t really care about all that messy science stuff. They just want to make money, and client-users, not test vendors, are always responsible for test use. A Black-Heart test vendor can sell its garbage all the way to the bank. And, they have become more prevalent since the Web opened cheap doorways into every office computer.
At its best, a Black-Hearted test will just waste your money. At worst, it will bloat your payroll with unproductive clones, create unnecessary legal challenges, impair product quality, produce obnoxious service people, hire/promote unqualified managers, irritate clients, and make you work harder. And that’s for starters.
Why Follow Professional ‘Standards?
It’s simple, really. When vendors follow the 1999 Standards for Educational and Psychological Testing, users are assured the test has stable items, measures something productive, is consistent over time, and predicts job performance. Isn’t that a nice thing to know when using scores to decide whether to spend 70 thousand bucks on a professional knowledge worker? Just try to get approval to drop that kind of money on a piece of equipment without doing some exhaustive leg-work.
Always remember there is a big difference between a test that shows differences between people and one that shows differences in job performance.
How to Spot Black-Hearted Vendors
Here are some tell-tale characteristics of Black-Hearted Vendors.
They divide workers into two groups and calculate average profiles. This is a really nasty technique because it looks legit, but nothing could be further from the truth. For example, averages hide individual differences; the groups might be divided based on opinion instead of objective standards; difference in scores might have nothing to do with job skills; differences might have occurred by chance; and you have no assurance the test factors actually cause high or low performance.
They promote the “benefits” of comparing scores to national profiles. This is another example of junk science at its finest. As mentioned before, averages hide differences. Ask yourself, for example: do all people in XYZ job have the same profile? Are all people in the job-match profile fully skilled, highly productive producers? Do your own workers with the same profile perform differently? What is the individual range of scores in each job-profile? Do the profile factors actually cause job performance?
Their technical manual is filled with happy customer stories. Every test should come with a technical manual that contains an honest explanation of how it was developed, why its theory is important to job performance, results obtained from internal reliability data, demographic impact, and its ability to predict specific performance in various jobs. You need to know that stuff. A Black-Hearted test skimps on data, never reports the size of the studies, throws numbers around that look important (but aren’t), and fills in the blanks with stories from happy clients praising its virtues.
Another clue to a Black-Hearted test is an easy-to-read manual, few details, heavy self-praise, and referencing theories of long-dead people like Maslow, Jung, Freud, Aristotle, and Groucho Marx. These are all signs a Black-Hearted vendor has no idea what he/she is talking about. As Monty Python said in the search for the Holy Grail, “Run away!”
They proudly claim their test contains dozens and dozens and dozens of factors. This is truly amazing. You see, over the years, researchers have shown only about six factors are related to job fit and about five to seven are related to job performance. So a list of 30, 40, or 50 factors is utter nonsense. It’s a clear indication you are dealing with a Black-Hearted huckster. Furthermore, extensive research shows even the best list of personality factors is poorly correlated with job performance (i.e., only a few percentage points). Personality only adds useful accuracy when based on a thorough job analysis, and when used in combination with other validated tools.
Need Help?
Black-Hearted salespeople are very good. They know that clients are seldom test experts and won’t systematically compare their hiring test scores to job success. They base their decisions on personal trust. If every client was an expert, there would be few, if any Black-Hearted test vendors in the market.
What you can do to protect yourself, though, is to recognize the warning signs outlined above. Then, find someone who is an expert in psychometrics (i.e., the identification and measurement of human skills and factors) to review your tests and processes. There are not many of them around, but you can poke-around their professional website to find a few names. The right expert can help navigate the deep waters of testing. When you are done, they will have saved you a small fortune avoiding bad hires.
You might have been fooled once, but you don’t have to be fooled all the time.