Demystifying AI in Assessments

Marketing hype machines are creating artificial intelligence (AI) FOMO — including in the talent-assessment space, with promises of exponential gains in efficiency and accuracy. With so many vendors and so many claims, things can consequently get murky trying to differentiate between assessment products. Then, too, it’s important to ask yourself: Are you really trying to improve prediction through AI, or are you experiencing FOMO about not being “AI hip”?

Assuming it’s the former, how do you separate fact from fiction and adopt a legit tool that is right for your needs? Here’s how to get started:

What AI Really Is — and Is Not

Know that AI is not a solution or a goal in itself and that it won’t magically fix all our problems. Rather, AI is a component that can be part of the solution. And of course, getting the right solution demands taking a step back and identifying the problem first. In doing so, you may find non-AI based solutions that are a better fit.

Simply put, AI is ”human intelligence or behavior mimicked by machines.” Forms of AI that you’ve probably heard of include:

Machine learning (ML). A machine’s capability to improve its performance as it is exposed to information without following specific directions, allowing the system to discover patterns and make interpretations automatically.
Natural language processing (NLP). Using machines to mimic how human beings understand language. NLP techniques are used to understand meaning in open text and speech.
Deep learning. The application of machine learning to a complex set of layers that seeks to mimic a human brain.

Many forms of AI are nothing new. In fact, some have been in use for years, even decades. That means that you sometimes have to take claims of AI innovation with a grain of salt. For instance, many techniques such as machine learning have been used to create assessment scoring algorithms since the 1950s — they are simply old wine in a new bottle.

AI is first and foremost technology that can automate lesser tasks, like finding certain information. Currently, it is nowhere close to human intelligence and does not really do a good job of thinking autonomously. It can only do what it is instructed to. AI does not work like the human brain, mostly because it does not understand causation — that is, it can see the relationship between things, but it cannot understand it. All of which means that AI is still incapable of making judgments the same way that people do.

The Baggage of AI

AI’s cognitive limitations open up a Pandora’s box of issues that directly impact confidence in its use. These include:

Bias. This is the No. 1 objection when it comes to AI. And justifiably so. The limitations of machine cognition make it easy for bias to creep in, contaminating its value proposition.
Garbage in, garbage out. Related to the above issue, AI is only as good as the data it is fed. If your data isn’t solid, your outcomes will be questionable.
Explainability. It is often not possible to show exactly what AI has done to achieve its outputs.
Overfitting. Models built using one sample may not hold when applied to another. This is especially problematic in assessments since they are often based on data from one specific sample.

Of course, these limitations are no reason to give up on the positive aspects of AI, such as the ability to help alleviate common hiring pain points.

Applying AI to Hiring

AI is omnipresent in hiring tools. The most common applications include:

Sourcing. AI tools provide the ability to scan for, target, and engage talent outside of the funnel. It is easy to see the benefits of having robots help fill your hopper with qualified candidates
Applicant volume. Hiring at scale has always been challenging — the human-processing limits of busy recruiters are easily surpassed when there is a tidal wave of applicants. AI can help manage large amounts of applicants.
Candidate experience. The ability of AI to engage candidates and help guide them through the hiring process can make a difference in their perceptions of an employer. AI also helps turn applicants into brand fans by making your process short and relevant.
Candidate quality. AI can help support decision-making on whom to hire and then provide concrete evidence that you have hired the right people for the job. This can further lead to outcomes such as increased tenure, lower attrition, and greater sales.

When it comes to talent assessments specifically, AI is definitely making a contribution to alleviating many of the common issues mentioned above.

Chatbots, NLP, and algorithmic matching. All provide assessments with the capability to quickly evaluate and recommend candidates based on an understanding of their qualifications, personality, and values.
Video interviews. They also use NLP and other aspects of AI to automate the evaluation of interviews, often providing competency and trait-based scores that can be fed directly into decision-making tools.
Tests. AI has found its way into many different testing tools, from writing test questions to auto-scoring simulations. AI is also very common as a way to evaluate the match between assessment results and key job requirements. Additionally, AI is also used to make assessments shorter and even create a situation where it evaluates information about applicants behind the scenes (e.g., social media profiles) so that applicants do not even know they are being assessed.

Note that just because assessment tools use AI does not mean that they use it well or in a manner that is compliant with government regulations. All the baggage of AI applies to assessments, and in many cases it is just simply not a sufficient substitute for traditional assessment methods such as the personality and cognitive tests that have been in use for decades.

Vetting the Vendors

The first step in your evaluation process is to turn on your BS detector and look closely at the vendor and its AI. Many vendors are selling pure snake oil. Others are not really AI companies at all. In some cases, AI is only a very small part of their core product, while other vendors are just hyping the same simple machine learning techniques that have been in use for decades. Knowing which questions to ask can help you see the truth — and sure enough, there are several quick ways to gauge a vendor’s legitimacy as a real assessment company.

Do they have an IO psychologist on staff?
Do they use job analysis to help study the job and choose the best tests?
Do they have real validation data and EEOC compliance technical reports?
Can they show evidence of the pedigree of the assessment (i.e., that it is based on quality research and sound psychological theory)?

Ultimately, at the intersection of technology and psychology lie both concern and opportunity. Moving forward without fear is actually pretty simple: Apply the tried-and-true methods that make any testing program effective and compliant:

Define the problem, make a plan, and be clear on how you will evaluate success
Conduct a job analysis to understand and document what job success looks like
Ensure that you are using valid and reliable measures that are free from bias
Ensure that those making hiring decisions are brought into the program and get the right data at the right time
Evaluate the impact of the assessment on KPIs

Yes, AI has a lot to offer talent assessments, but don’t believe the hype. When it comes to AI assessments, following the best practices that apply to all assessments will help ensure you get the best of both.