Receive daily articles & headlines each day in your inbox with your free ERE Daily Subscription.

Not logged in. [log in or register]

Matching: the Newest Flavor of Assessment Tools

by
Dr. Charles Handler
Feb 16, 2012, 5:46 am ET

I continue to be impressed by the evolution of pre-employment assessment tools. This evolution is being driven by the continued growth of the value proposition assessment provides. As a result, an increasing number of new product include embedded assessments designed to help predict which applicants have the best chance of success.

This past year has brought a proliferation of firms that are using assessments to provide a new twist on matching online job applicants with job openings (and conversely allowing firms to match their job openings to candidate data residing in a database).

To understand the origin of Internet based matching, one has to turn the clock back about 15 years to the dawn of the job board. Job boards provided arguably the biggest overall change to the status quo for the way — both in the way people are hired because of the increased ability for individuals to find out about job opportunities, as well as for those hiring, to locate viable candidates.

Things have not changed much as even in the present day. The basic Internet job search equation involves a matching process in which each party provides information about who they are and what they are looking for. Behind-the-scenes algorithms living on servers evaluate the data provided by each party and calculate a match.

Although the job boards would argue otherwise, my basic description of the matching process used by most of them can be summed up with the phrase “garbage in, garbage out.”

I believe this is because most job boards do not really approach the matching process in a manner that uses what we I/O psychologists know about predictive science. The result is that while there are pockets of effectiveness, big job boards continue to be a source of noise for hiring personnel because they tend to deliver quantity over quality. This noise requires organizations to have strong capabilities for screening candidates at volume. Unfortunately, few companies have this capability, and even for those that do, the statistical chances of making a good hire go down each time an unqualified applicant enters into the hiring process.

I have ranted about the ineffectiveness of the matching process used by job boards again and again over the years and it seems that others are starting to agree with me. The difference between me and them is that they have begun to put their money where their (and my) mouths are by investing heavily in the creation of new businesses devoted to delivering candidate quality through matching that uses predictive assessments.

How Do These Sites Work?

An important difference between these companies and other assessment firms is that, for the most part, these companies provide matching that is part of the search process (as opposed to part of the application process). Their goal is to help companies stack the odds in their favor through a matching process that supplies candidates who have the best chance of being a good hire for a specific position.

There are a few variations on the basic theme, but these sites are all based on the concept of providing accurate matching using various pieces of information that help users to predict which persons are a match for a given situation, or vice versa.

One excellent analogy for the way these sites work is online dating sites. Back in 2005 I wrote about the parallel between online job matching and online dating and this analogy continues to work.

Both job matching sites and online dating sites work via the following process:

  • Users are looking for a new situation of some sort
  • Users define who they are (and what they are looking for in some cases) based on the information requested by the system. This is usually based on a profile creation process
  • Users search for matches to their needs via the comparison of their profiles to those of others stored in the database.
  • Results are returned electronically with some basic information about why the match was suggested (alignment in key areas indicated to be of a high level of importance).

In online dating all the matching process does is narrow down the database from millions to a few. Once this initial match is made, it is still up to the individual to communicate and evaluate the effectiveness of the match (usually by flirting and then going on some dates).

Hiring is no different. After the match is returned, the actual hiring process is used to evaluate mutual compatibility and to make important decisions about entering into a relationship.

The main way that online job matching sites are differentiated is by the information they use as inputs to their matching equations. Some use information that is provided by assessment tools that were developed using an accepted scientific process. Others do not, favoring the use of a variety of other types of information that may be relevant (past job experience, salary requirements, geographic location, favorite color, hobbies, etc.).

A thorough review of all the specific models being tried (some of which I’ve advised or consulted for) is beyond the scope of this article, as my focus is specifically on those models that use some form of assessment tool to help with the matching process. In these models, part of the profile creation process involves the individual completing assessment tools that provide a way to measure them on key attributes (personality, intelligence, skills, etc) to understand who they are and what they bring to the table. These sites may also use assessments to profile job seekers using a set of work-related values or preferences (autonomy, social responsibility, innovation, etc.) to determine what is most important to them in a job. Employers provide the other side of the matching data by indicating which attributes they feel are most important for performing a specific job or by profiling the values that the organization supposedly holds.

Are These Sites Really Effective?

I think that the jury is still out on the effectiveness of these sites. I am going to guess that as the data rolls in, some will be much more effective than others. Overall, many of these sites will eventually demonstrate a level of impact that is greater than that of sites using coarse keyword searches.

However, users of these sites should have realistic expectations. We have to take these sites for what they are: a relatively coarse matching tool that can be implemented quickly with a low cost. When it comes to assessments, you get what you pay for, and quick and easy matching is not going to tell you everything you need to know or ensure you make a good hire. Effective prediction requires precision; precision requires detailed effort. The more corners you cut, the less effective the results.

You still have to flirt and go on dates to figure out who you want to marry! The best value from these sites is allowing you to narrow down the pool from all the fish in the sea to those fish who have potential to thrive in your pond.

Here are a few factors that will impact the success of matches made using these sites:

  1. Quality of the assessment tool included: The better constructed the tool and the more research behind its effectiveness, the better the matching will be.
  2. Thoroughness of the definition process used by the company side: One potential shortcoming of this method is that the company-side profiles jobs based on the input of one or a few individuals. If these individuals are off base with their data, the accuracy of the whole process is compromised.
  3. Niche focus on one job or industry: Focusing on a niche allows for a deeper level of knowledge about what it takes to perform a specific job well and what predicts specific traits lead to success.
  4. Additional data collected: The quality, thoroughness, and relevance of the non-assessment related data used as part of the matching process will also impact the accuracy of matches.
  5. Ability to demonstrate that those hired using their system are “quality hires”? The more evidence that shows that matching impacts outcomes, the more effective the system at achieving its goals.

What You Need to Know

For job seekers: Go ahead and try. It can’t hurt. These sites all have some specific twist on the matching process, and there is value in interfacing with all of them as a learning experience. One of the big draws for job seekers is that these sites will give you information back to help compel you to make an investment in creating a profile. This information is usually in the form of a developmental report that is based on the assessment. This is valuable information for job seekers to help you understand themselves. It is unheard of for companies to share the results of assessments used in the hiring process with applicants, so as an applicant this may be your only shot at seeing your assessment results. Also, these sites can help you find jobs for which you are actually qualified — so it’s worth the time, but be ready for imperfect matches. Expect some error, albeit less than that of the big job boards. Don’t spend a dime to use these systems. They should be free.

Employers: Go ahead and try it. It can’t hurt. But remember that these sites will likely reduce your hiring error rate but they will not erase it. These sites are an excellent way to feed a more developed and focused assessment program that resides within your actual hiring process. Also be very aware of the importance of the accuracy of the profile parameters that you set. If this process is off base, the results will suffer. A few of matching companies help with this issue by using databases of information about the competencies that are critical for specific jobs based on information collected over the years by various sources. This removes the profile definition process from the employer, which can be helpful in some situations. If you are in a small- to medium-sized business, these sites can be an inexpensive way to find talent.

These sites will matter in the future, and make money, but it will take some time. When you boil it down, these sites create value in two ways. The first is in their candidate database. No companies will invest money to access an empty database. The big guys (Monster) can attest to the fact that the value of the business is in the size of the candidate database. The second way these firms generate value is via their impact on quality of hire. Unfortunately, most firms don’t measure quality of hire or trace it far enough back to understand the impact of their sourcing methods on it. So the real winners will be the ones who are able to figure out how to go viral so that they fill their database with candidate information, allowing them some bait to lure in companies who will pay for access. Unfortunately, quality of hire, while quite important, will likely always take a back seat to database size mostly because of how much harder it is to quantify.

Sit back and enjoy the coming onslaught of these companies. Look for one whose business model makes sense to you and give it a try. You have almost nothing to lose and some good employees to gain.

This article is provided for informational purposes only and is not intended to offer specific legal advice. You should consult your legal counsel regarding any threatened or pending litigation.

  1. Steve Balas

    Since 1997 I have used a pre-employment assessment tool. Quite frankly I think I would have been lost without one in the employment environment we have been in for years. Read Hire for Attitude by Mark Murphy. Another confirmaion based on research about the importance of knowing something more about a person than their education and skill.

  2. K.C. Donovan

    Great article Charles! How about just 2-3 examples of the “sites” you keep referring to in your post? Not asking for you to evaluate them – just want to see the types of assessment these “new” players are offering…

  3. Paul Basile

    Charles,
    I am a fan of your your work, always insightful and relevant – and this post is no exception.

    (Full disclosure: I run a matching site…)

    I agree especially with the importance of using accepted science, predictive assessments. Compromise in this destroys the huge potential of the approach. If we don’t validly predict performance, what are we doing?

    The second factor you mention, of the factors that will impact success, is creation of the job profile. This is very key: if the job profile is the product of one or a few individuals it risks compromising the whole process. In my observation, this is regrettably very common. To prevent this, we access a database of thousands of professionally-generated job profiles because, otherwise, we cannot accurately and scientifically match person and job.

    I might be a little more optimistic about the speed with which innovative-minded companies will embrace science-based job-person matching. Matching using serious science increases the likelihood of companies placing the right people in the right jobs and can be done fast and relatively easily, online and scalable, due to advances in analytics, technology and the science behind the assessments. It will save companies millions and improve productivity. Smart companies will go for that deal; we see signs of that already.

  4. Keith Halperin

    @ Paul: I look forward to (and am working toward) a more fact-based means of hiring (aka: Generally Accepted Recruiting Practices [GARP]). ISTM that large-scale acceptance of this fact-based approach requires that a tool or service be effective,easy to use/intuitive, and be very affordable.

    Cheers,
    Keith

  5. Culture Fit Is On Fire

    [...] international talent management organization  – has published articles on both measuring and hiring for culture fit, a Forbes article considered the ramifications of not “fitting in” at [...]

  6. Lauren Smith

    A few great examples of social recruiting platforms…

    1. BraveNew Talent (http://www.bravenewtalent.com/)
    2. Ascendify Social Recruiting (http://www.ascendify.com/)
    3. BranchOut (http://branchout.com/)

  7. Angus McDonald

    Charles, I like your article very much and agree that over the next few years we will be seeing very interesting developments in the way automated job matching changes the recruitment process. In many ways, I don’t see job matching in general as changing the status quo much in terms of the quality of offerings. There will be those who do it effectively (I hope to count myself amongst these being a part of Matchpoint Careers) and those who don’t, just like current recruitment practices as you suggest. However, job matching will offer the potential for SMEs to benefit from some of the tools and knowledge that has previously been used most heavily by the larger recruiters.

    With this in mind, I suggest that validity should be thought of more holistically. Maybe it’s not all about whether those organizations who go down the science-based recruitment route start hitting the expected 0.5 or 0.6 validities. Instead, what’s the average validity of all recruitment processes? (I’m not expecting an answer to this one!) If technology such as job matching makes at least ‘good’ recruitment more widely available, and raises the average predictive validity, then surely that’s a great thing for everyone.

    One very significant point, and one I have raised before on ERE discussions, is that what job matching sites have the potential to do is change very dramatically the shape of the recruitment pipeline. Technology allows us to put the assessments that science tells us are the best predictors of job performance right at the beginning of the sift process. Until recently, cost and practical issues (e.g. administering and scoring paper-based tests) have meant that tools such as psychometrics were used some way down the recruitment pipeline. In this process a large number of candidates would already have been rejected, typically through resume or application form sifts, in ways that we know are not effective. How many star performers might this process usually miss?

    The meta-analyses of validity evidence that are often cited (Schmidt and Hunter springs to mind) are now quite out-of-date, and mostly not based on the potential which job matching offers us. As you note, validation is essential, but if these approaches do allow large volumes of candidates to be profiled using the most effective talent-identification tools then matched against robust job profiles using evidence-based algorithms, the best of them might just start pushing upwards the currently accepted validity estimates.

  8. Extracting assessment wisdom is a human endeavor | Career Management and Workplace Culture Blog | TalentCulture.com

    [...] However, according to one of our #TChat Radio Show guests last night – Charles Handler, Ph.D., President/Founder of Rocket-Hire – Effective prediction requires precision; precision requires detailed effort. The more corners you cu…. [...]

  9. Extracting assessment wisdom is a human endeavor | Map Career Management Firm

    [...] However, according to one of our #TChat Radio Show guests last night – Charles Handler, Ph.D., President/Founder of Rocket-Hire – Effective prediction requires precision; precision requires detailed effort. The more corners you cu…. [...]

  10. The end of the journey for me (To new beginnings!) « Marc My Words

    [...] There has been enormous growth in understanding how assessments can help companies make better hiring decisions by considering work behaviors and cultural fit over skills and experience. You can read more about that in an article by Dr. Charles Handler, Matching: The Newest Flavor of Assessment Tools. [...]

  11. Extracting assessment wisdom is a human endeavor

    [...] However, according to one of our #TChat Radio Show guests last night – Charles Handler, Ph.D., President/Founder of Rocket-Hire – Effective prediction requires precision; precision requires detailed effort. The more corners you cu…. [...]

Post a comment

Please log in to post a comment.

Note: You need to sign up for an account on our new commenting system if you haven't already done so — even if you have an existing ERE account. Find out why »

Login Information