Guidelines for the Effective Use of Online Pre-Screening Tools

In our last article (Do Online Pre-Screening Tools Really Work?) we discussed the growing use of pre-screening tools and identified eight “half truths” associated with pre-screening tools. In this article, we provide some guidelines to help readers manage the risks associated with the use of pre-screening tools while maximizing the effectiveness of these tools. Following these guidelines won’t guarantee that pre-screening will achieve all the marketing claims made by pre-screening vendors, but it will help ensure that pre-screening provides maximum value to your staffing process. The guidelines discussed in this article are organized into four categories, each of which are detailed below. Category 1: Integrating Pre-Screening into Your Staffing Process It is important to understand that pre-screening be viewed as one step in a broader strategic process for sourcing, screening, and selecting candidates. Pre-screening is not a magic bullet that will solve all of your staffing problems. Rather, it is a critical component that will help you address some specific problems you may be facing (i.e., resume overload, lack of a systematic process for evaluating candidates, etc.). When deploying pre-screening it is important to clarify requirements and manage expectations about the purpose of pre-screening within this broader system. A failure to do so can result in unmet expectations, increased resistance, and other forms of organizational backlash. Paying attention to the following guidelines can help you avoid these problems.

- Emphasize the strengths of pre-screening. Pre-screening can efficiently screen out applicants who admit to lacking job relevant skills, experiences, or qualifications. This can reduce applicant pools by 50% or more, which is a tremendous value when dealing with large volumes of candidates. Pre-screening can also be effective for finding candidates who possess highly specific and unique qualifications (e.g. speaking Inuit or having experience using a tunneling microscope). These are the main reasons for using pre-screening. Mention them early and often.

Recognize the limitations of pre-screening. While pre-screening questions can identify applicants who lack job qualifications, they have limited value for actually predicting job performance. Just because a candidate says they have certain skills, that’s little guarantee they are truly skilled. If it was possible to identify top candidates by asking them a few simple questions about skills and experiences we would have been using tools like this for years. Simply placing these questions on the Internet does not give them some magical power to evaluate candidate potential. Companies that expect pre-screening questions to effectively identify the top 10% of candidates are probably setting themselves up for disappointment. The main purpose of pre-screening is to remove the bottom half of the applicant pool, not sort the top half.
Pre-screening is only as good as the questions. Writing good pre-screening questions takes a lot of work. Hiring managers or recruiters must be able to determine what requirements drive job success and be able to create questions to determine if applicants meet these requirements. This is not a trivial task. Make sure hiring managers know that they will have to do a fair bit of upfront work if they are to reap the benefits of pre-screening. We feel this is a weakness for many screening vendors. While they may provide excellent tools for asking questions, they often fail to do the post-sale work and analysis required to ensure the questions being asked are the right ones.
Emphasize pre-screening as one part of the staffing process. The purpose of pre-screening is to make high-level decisions about applicants’ general qualifications early in the staffing process. While pre-screening questions are an excellent first step for narrowing down the number of applicants, they are not suited for determining which applicants are the best fit for a particular job. Nor will they guarantee that the right applicants are applying in the first place. Communicate pre-screening as one step in a larger staffing process that includes using appropriate sourcing methods prior to pre-screening, and use other forms of assessment and interviewing techniques to evaluate candidates who pass pre-screening. Category 2: Selecting a Pre-Screening Vendor During the process of writing our review of assessment vendors, we identified over 20 companies who provide tools for pre-screening. There tends to be relatively few differences between these vendors in terms of the look and feel of their pre-screening questions. However, there are major differences when it comes to the technology used to support these questions, the emphasis placed on ensuring that pre-screening questions are valid and effective, and the ability of their systems to support other forms of assessment in addition to pre-screening. Price also varies widely, from less than $10,000 for simple, standalone systems to well over $100,000 for enterprise applicant tracking systems. Which vendor is the “best” will depend on the unique needs of your company. No matter what your situation, we feel that the following guidelines can help you to better evaluate potential pre-screen vendors:
Does the vendor focus on quantity or quality? The ultimate goal of staffing is to make good hires as efficiently as possible. It is not to make bad hires quickly. When evaluating pre-screen vendors, ask how they measure the value of their system. Note whether their response emphasizes measures of staffing volume (e.g., number of candidates processed) or measures of new hire performance (e.g., performance ratings, retention). Vendors who focus primarily on staffing volume may offer little help to ensure that the candidates you are screening in are truly the best performers.
Does the vendor effectively integrate assessment and information technology? Robust technology platforms are critical for effective use of pre-screening. Make sure you involve your IT department early in the vendor selection process. At the same time, remember that the ultimate goal is to hire better employees and not simply to get a system that is easy for your IT department to support. In other words, do not put the technological cart before the application horse.
Does the vendor support alternative forms of assessment? Pre-screening should be one of several assessment tools used to support staffing. A variety of other assessment tools such as knowledge tests and culture fit measures can also be used early in the staffing process to complement pre-screening. Look for vendors whose systems integrate and support these tools in addition to prescreening questions. Category 3: Writing Questions Perhaps the single biggest challenge to using pre-screening is developing effective pre-screen questions. Most pre-screening vendors leave it to the client to determine which questions to ask. Consequently, you need to clarify the process for developing questions prior to choosing a vendor. This requires determining who will be involved in writing or choosing the questions and what steps will be taken to ensure these questions are job relevant, legal, and effective. The following guidelines can help with this process:
Focus on specific, verifiable skills and experiences. As a rule, assume that candidates will always respond positively to a question if they feel they can justify or rationalize their answer. Most candidates would probably prefer to be eliminated during an interview than by a computer, and if this means stretching the truth to pass pre-screening than so be it. The best way to counter this is to ask very specific questions about what candidates clearly have or have not done. For example, rather than asking if someone is an “expert in Excel”, ask them if they have “written Excel macros.” Instead of asking if candidates have “managed people,” ask if they have been “responsible for making hiring and promotion decisions.”
Ensure all questions are job relevant. The #1 way to ensure the legal defensibility of any component of your hiring process is to establish clear links between hiring criteria and specific aspects of job performance. It is critical to demonstrate direct relationships between the content of pre-screening questions and job requirements. This is especially true if you are using questions to eliminate applicants from the candidate pool. Do not simply put in questions because people want to ask them. Challenge hiring managers to explain why the skills and experiences addressed by these questions are critical to job performance.
Ensure questions are not offensive or illegal. Make sure that questions do not violate any EEOC regulations and that they will not seem offensive to applicants. Be wary of letting hiring managers write their own questions unless they are well versed in EEOC hiring guidelines. It is easy to write apparently job relevant but potentially illegal questions. Avoid writing offensive questions by making sure that questions do not leave applicants wondering, “Why are they asking me this?”
Focus on “high impact” questions. The goal of pre-screening is to remove the maximum number of unqualified applicants with the minimum number of questions. Avoid questions that are answered the same way by the majority of candidates, as these provide little value for distinguishing between candidates. Also avoid questions that reflect “nice to have” concepts instead of critical skills and qualifications. For example, some pre-screening vendors recommend asking if candidates are interested in using certain skills. However, a wealth of personnel research indicates that being interested in a task has relatively little relationship with actual task performance. Furthermore, when we analyzed pre-screening responses we found that almost every candidate who indicated their skill level to be intermediate or higher also expressed a medium to high level of interest using the skill. What this suggests is that little useful information is gained by asking about interests. In general, limit the focus of pre-screen questions to specific experiences and requirements, and use other more in-depth assessments to get at soft skills like motives, interests and competencies.
Be consistent. It is critical that every applicant for a given position be asked the same pre-screen questions. Consistency is important for establishing legal defensibility. Using screening inconsistently will also reduce the effectiveness of your hiring process. Do not allow managers or recruiters to change pre-screening questions after a requisition has been posted. Category 4: Scoring and Interpreting Candidate Responses The best pre-screening questions can be rendered meaningless or even harmful if the responses to the questions are not appropriately interpreted. This requires establishing effective methods for evaluating and scoring candidate responses. The following are a few guidelines to consider in this area:
Assign scores carefully. Many pre-screening systems allow you to assign scores or points to different applicant responses. Most also let you create “knockout” questions where applicants who do not give the right response are disqualified from the candidate pool. Make sure that you carefully think through how these scores are used, particularly knockout questions. Test different scoring options by comparing scores received by candidates with other information you may have about them (e.g., resumes, interview results). Make sure that the scores accurately reflect key job demands and requirements and are not overly rewarding candidates who have job relevant but somewhat trivial experiences. If you are using pre-screening to place candidates into discrete groups based on qualifications, look at EEOC statistics to ensure you are not disproportionately eliminating candidates from certain protected groups. The advantage of scoring pre-screening is that it allows you to automatically sort through large numbers of candidates. The risk is that it lets you rapidly duplicate mistakes.
The highest scores are not always the best. Our research indicates that candidates that get the highest pre-screen scores are often overqualified, are more willing to “stretch the truth” in their responses, or have job relevant skills but non-job relevant experience (e.g., an IT professional applying for a administrative job on the basis of his/her knowledge of MS office software). Do not assume a one-to-one correspondence between pre-screening scores and performance potential.
Do not rely too heavily on pre-screening to evaluate candidates. Pre-screening is good for evaluating whether candidate meet the minimum requirements of a job. However, other forms of assessments such as interviews and personality or ability measures should be used to make more fine-tuned judgments about a candidate’s true performance potential. This is particularly true for assessing competencies associated with motivation, expertise, interpersonal style, and other “soft” skills.
Monitor the effectiveness of questions. Establish processes to track whether your prescreening questions are screening out appropriate numbers of applicants and to determine if the applicants being screened in are the “right” candidates. If large numbers of unqualified applicants are appearing at the later stages of your hiring process, then try to determine what your screening questions are missing. Conversely, make sure the questions are effectively identifying qualified candidates. Metrics that can be used to evaluate the effectiveness of pre-screening include interview-to-hire ratios, new hire performance and retention, relationships between question responses and resume quality, and question response frequencies. Note that the primary focus of these metrics is on candidate quality, not quantity. The above guidelines address the major risks and advantages of pre-screening we uncovered through our study of pre-screening. While using pre-screening and following these guidelines won’t guarantee great hires, it will greatly reduce the time you lose dealing with clearly unqualified applicants.