Here’s How to Avoid Picking a Shoddy Assessment Vendor

Selecting a pre-hire assessment vendor is not always easy, but doing it right can make a huge difference.

I’ve written about this subject before, as has my fellow I/O psychologist and ERE author Dr. Wendell Williams, and I’ll be doing a webcast with ERE on this very topic soon. I’m going to give you selection tips in a minute.

First, three major reasons why the vendor selection process can be so hard.

The Challenge of Differentiating Vendors

Even to the expert, it can be very hard to tell vendors apart because they may seem to all offer the same value proposition. There are so many different types of tools and the methods and technology for using them vary. It can be hard to tell who is peddling junk and who is actually providing a quality product. Even once the wheat is separated from the chaff, it can still be hard to determine which methodology is best suited for you. Differentiation begins with understanding the “DNA” of the assessment provider: who they are, where they come from, and how they approach the market. I break the market down into the following vendor types:

Matching companies — Work at the job search level, often use database of assessed candidates
Test publishers — Sell tests delivered via a simple system, do not alter content to match situation
Profilers — Use same set of content in all situations, benchmark vs. existing employees
Consulting firms — Work with firms to customize an approach, most often on a strategic level; often resell tests created by others
Solution providers — Use technology infrastructure to customize solutions for clients; often embed assessment within a suite of other products
Talent management firms — Offer assessment as an add on to other post-hire services. Usually offer assessments via a preferred reseller but may also have an assessment division within the firm.
Non-traditional vendors — Usually based on some proprietary or revolutionary idea; often deviate from what is acceptable on the science side

Vendors Often Overstate Their Value Propositions (and then offer no actual way to prove it)

Some of the worst strategies used to sell value include:

The 100% solution. Never trust a vendor that claims they can predict job success with levels of accuracy hovering anywhere near 100%. Predicting anything humans do with high levels of certainty is not an easy thing to do. Decades of research and practice by highly trained selection scientists cannot be ignored, and this experience generally shows that even 50% accuracy is a huge win. Many vendors will point to studies to show that they can predict with incredibly precise levels of accuracy. While this may be the case within their study, or within a very limited sample, the methodology for these studies is often not within the guidelines of what is accepted by professional I/Os (or the EEOC for that matter).
The “trust me.” Many vendors rely on a prestige factor to help them sell their product. In these cases there is usually a super genius founder or scientist who has developed his/her own proprietary methodology that is superior to anything else around. These vendors push this person’s brilliance and their superior science as the foundation for your success (often without including any actual evidence of real-world, bottom-line impact).
The star client roster. Many vendors rely on name dropping with no actual substance to support that any actual value was delivered for the clients they are listing. While many vendors are doing great things for great companies, be wary of letting the client names do the selling as opposed to the substance and value of the product.

Any vendor worth their salt will have a well-developed methodology for proving their value and will have documented believable results to show their ability to be a good business partner. These results will be easy to verify via technical reports and review of business outcomes as well as via the evaluation of their methodology by an expert.

The Decision-making Process Is Broken

Perhaps the biggest reason that getting it right when choosing an assessment vendor can be so hard is that the right people are not making the decision, the decision is not being based on the right information, and/or the wrong people are making the decisions for the wrong reasons.

It is rare that firms involve experts in the selection of pre-hire assessments. If a company has an internal I/O psychologist, they are generally in good shape because these folks can act as technical experts who tend to keep the process honest. Outside of this, it is rare that firms have any real technical expertise at the table to help them with a real evaluation of the methods and tools a vendor brings to the table.

Most firms do not select vendors via a formal competitive RFP process. Instead the selection process often involves some form of politically based decision that involves someone with status in the organization having used the vendor at previous employer. Relationships are often the direct antecedent to choosing a vendor over any real substance. Sometimes this will work out, but if I had a dime for every time I’ve seen the wrong vendor chosen based on a relationship or political pressure, I’d be writing this article from Tahiti.

Good companies that value their testing programs as a strategic asset do not make choices based on politics or relationships, but on cold, hard facts and an expert-driven process. Here are some tips for avoiding the major pitfalls when selecting a vendor.

Use a process: Be it formal or informal, good decisions are the result of a carefully developed game plan that defines what is needed, collects the required info, and supports objective decision making.

Assemble the right people (including experts): Get all the right stakeholders at the table and give them a voice. Having one person be the decision-making body can often invite danger.

Identify the value that must be demonstrated: Know very specifically what you expect to get from the program before you go shopping. Communicate this to vendors and use their responses to support decisions about their ability to deliver bottom-line impact. Define this value in terms that business leaders will understand and that will allow them to see the value of the assessment program.

Define key parameters before you begin, including:

What is to be measured
What implementation methods you can support
What technology and integration is needed
What additional things the implementation will touch
The budget and timeline

Collect data needed to differentiate: Use your process to collect real information that is verifiable. Look to your needs to discover key points of differentiation between vendors.Two of the biggest differentiators are relevant industry experience and ability to deliver actual business results. But, the differentiating data can be anything that speaks strongly to what you are looking for in a partner.

Take the assessments from the customer’s eyes: Both recruiters and candidates are customers of assessments, and I always insist my clients take the assessments being considered and think about what they are seeing from the eyes of the customer. This is often one of the most powerful steps in the evaluation process.

Working technology: The demo also provide a great chance to see how smoothly the vendor’s technology works for the user. I have experienced a vendor’s technology crashing during the sample assessment exercise. This generally does not bode well for them.

Evidence of service level: How well does the vendor listen during the evaluation process? Do they present a level of customer service and care that shows they really value getting and keeping your business?

Consider starting small when working with a new vendor and allowing them the chance to prove what they can do in a small proof of concept or pilot. Create a process that will allow the vendor to track their results directly back to the outcomes you defined in your assessment charter before you allow them the keys to the kingdom.