Part One in this series of three articles provided some basic information about the benefits of using online screening to quickly and efficiently evaluate the suitability of candidates applying for a job using the Internet. To quickly recap, the main idea of the article was that screening is a more effective tool than the resume for evaluating online job applicants, because it uses technology to presort and prequalify candidates and is based on standardized, objective criteria that are directly related to job performance. Part One also broke down screening into two major types: screening based on background information and simple qualifying questions (“non-scientific screening”) and screening based on data-driven assessments that measure critical job related competencies (“scientific screening”). Part One suggested that scientific screening offers an advantage over resumes and nonscientific screening, because it uses data to create a blueprint of job performance and to create filters used to select in candidates who match this blueprint. In short, these systems use a scientific process to predict which candidates are the best hires for a job. In the “no free lunch” department, Part One also suggests that effective scientific screening comes at a price, because it requires the collection and analysis of data and must conform to specific legal and professional requirements. The purpose of Part Two of this series is to take a closer look into scientific screening in order to help understand some of the reasons why effective, legal scientific screening requires extra effort. This will be accomplished by taking a closer look at the three major requirements for scientific screening that I outlined in Part One and explaining more about what occurs at each step. Requirement 1: Building a Blueprint of the Job Scientific screening systems must offer a way to create a blueprint of what “things” are critical for success at a job. These “things” are usually labeled as either Knowledges, Skills, Abilities, or Other aspects of the job (KSAOs). Just between us, KSAOs are basically the same thing as competencies, so I will refer to them as competencies from now on. Anyway, the identification of critical competencies is important because the ability of a screen to accurately predict will suffer if a poor or incorrect definition of job performance is used. There are two basic ways in which a blueprint suitable for use in scientific screening can be constructed.
- Job analysis. The first and most common way blueprints are created for jobs is by a process known as job analysis. A full blown, traditional job analysis is a study involving many labor intensive steps resulting in a highly detailed set of specifications for job performance. The steps involved include interviewing incumbents and their supervisors to develop a preliminary list of important competencies, and then gathering data using a series of questionnaires that are distributed to job incumbents and supervisors. The data from these surveys is then analyzed, and the results are used to create a data driven model of what is needed for job success. In many cases, this process is extremely detailed and can often take months to complete. While this information is useful and sometimes perceived as being required for legal reasons, for the most part, highly detailed studies are overkill and can represent a waste of time and resources. I think that most successful online screening models will either reduce the Job Analysis study down to a few key elements needed to develop a good solid blueprint, or will make use of Job Specification Databases to help automate the process of building a blueprint.
- Job Specification Databases. The second way to create a job blueprint has been made possible only by the advent of the Internet. The Internet has given us the tools to use information from many past job analysis studies to construct databases containing general information about the competencies required by almost every major job, job family, or type of job. Online screening systems can make use of these databases to anchor a system that allows users to build a blueprint for a job using drop down menus to build a blueprint of critical competencies.
There’s also a third way to create a blueprint: integration of the first two ways.
- Integration. Many purists in my field will cringe at the idea of not doing a full blown job analysis, but database driven systems do not have to completely replace job analysis studies. Rather, the two can be integrated to help make the process easier and more practical. For instance, additional quality control can be added to a menu driven process by requiring incumbents, supervisors, etc. to verify that a blueprint is accurate or even require them to complete short questionnaires and then generate blueprints based on the integration of survey data with stored information. The possibilities here are endless. My main point is that there is a better, faster way to build a good blueprint that does not require spending two months and lots of dollars killing flies with sledgehammers. Instead, the issue requires some new ways of thinking about the means needed to reach the desired end-hiring the right person quickly.
Requirement 2: Collecting Data From Applicants Comparing applicants to the blueprint required for job success involves asking them to answer questions that allow the measurement of the competencies that make up the blueprint. This process results in the data needed to predict which applicants are most likely to be successful. Creating relevant questions to reliably measure specific competencies requires quite a bit of testing, development, and statistical analysis. A full explanation of these is best saved for another article. For now, I will focus on providing some background information about the types of questions most commonly used in online scientific screening. The most common types of questions used for scientific screening include:
- Personality. Personality questions get at specific aspects of a person’s personality as they relate to success at a job. Personality questions seem to be the most common type of online screening questions currently in use, with the majority of tests focusing on the trait of conscientiousness (this trait is related to predicting if a person will come to work each day, can be relied on, etc). Personality is also very good for helping predict things such as teamwork behaviors, work styles, functioning in interpersonal relationships, performance under stress, etc. There is a lot of evidence supporting the usefulness of personality questions for predicting job performance.
- Biodata. These questions ask applicants about their life experiences and are based on the idea that what a person has done in the past is the best predictor of what they will do in the future. This type of question has been shown to be highly effective in predicting performance, but suffers from a few problems. First of all, it is often the case that no one can say WHY a particular biodata question is related to job performance, so it can be hard to link biodata questions tightly to competencies. Another problem is that some biodata items are developed based only on statistical relationships using a process that ignores the content of the question. This can be a problem, because it leads to items that ask things like, “How many jobs did you have in high school?” While the right answer to this question might have a great statistical relationship with job performance, these kind of questions often leave applicants asking “Why are they asking me this, what does this have to do with this job?” Questions like this can be very damaging, because they can leave applicants with bad impressions of the company doing the screening.
- Situational judgment. These questions present applicants with a work-related situation and ask them to indicate how they would react. This type of question can be highly effective because it measures how a person will react in future work related situations. Situational judgment items can be very effective for predicting performance on competencies such as interpersonal skills, business acumen, and decision making. One drawback is that there is a difference between saying you would do the right thing when you are trying to get hired and actually DOING the right thing on the job. Often these kinds of questions are best when integrated into structured situational interviews used later in the hiring process. Situational judgment questions are also an excellent basis for interactive business simulations and games, something that I am personally excited about seeing used more often in the near future.
- Cognitive ability. Cognitive ability questions measure how intelligent an applicant is. There are many different facets of intelligence and there are tests for all of them including: verbal ability, math ability, spatial perception, etc. The information from these tests is often said to be the most highly effective predictor of job performance. However, these questions can create legal issues because some minorities do not do as well on them as members of other groups. This means that much caution must go into their use. In terms of Internet screening, these types of questions can also present security issues since remote test taking offers all kinds of opportunities for cheating. Cognitive items are rarely offered in the initial online screening phase and are better suited for administration in supervised situations.
- Work Values/Interests. These type of questions are highly effective in measuring what is known as “Fit.” Fit is a concept that is defined in many different ways, so measuring it precisely is difficult. In fact, there is no one standard way to measure fit. It is most commonly measured using personality questions, but it can also include questions about a person’s work values and preferred work environment. Another reason that fit is difficult to measure is that it relies on the hiring organization to generate accurate profiles of their values for each area of the organization that is hiring. Such profiles are critical to good measurement of fit because they must serve as a filter against which applicants’ responses to fit questions can be compared. Despite these issues, fit questions are being used effectively and will become more popular as we gain experience using them.
- Simulations & Games. These incorporate all of the above types of questions plus more. It is too bad that there are not more Internet-based simulations currently in use because they are very powerful tools for predicting who will be successful at a job. This is because research and experience have shown that the closer a screening measure is to the actual job, the better it predicts performance. A classic example is the typing test. If you want to select the best typist, you can’t do much better than asking them to take a test where their typing skills are measured. Another benefit of simulations is that they can be made fun and interesting and can be easily delivered over the Internet. In the future, game-type assessments designed to entertain and attract applicants while also providing powerful screening will become common. Simulations are a win/win situation in my opinion and I believe that more and more firms will see their value. (If anyone is using game-type assessments please contact me, I am very interested in learning more about what you are doing).
All of the information about questions is made more complicated by the fact that the most effective way to measure the competency blueprint is to use several different types of questions that function as a system. This is because each type of question is best suited to measuring different competencies. In fact, in my book it is bad form to eliminate persons based on their score on questions related to just one competency. Think back to the typing test example. If you are screening typists it is certainly much better to screen for typing ability AND for personality traits that can tell you how reliable the applicant is. The best typist in the world is no use to you if they fail to show up when they are supposed to. This is why I recommend looking at questions using a systems approach that relies on a combination of questions that maximize predictive accuracy. Requirement 3: Providing an Index of Comparison The final step in the scientific screening process involves taking the completed screening questions, scoring them and making sense out of the results. The result of this process will be information that allows persons doing the hiring to compare applicants against the blueprint and against one another in order to help identify those most likely to be successful at the job. This process involves several steps, each of which relies upon data.
- Scoring. Scoring of questions is based upon a key that has been constructed during the test development process. Information regarding correct answers is stored on a server and when the data from questions is presented, it is filtered and values attached to each answer. The science behind test construction dictates that several items must be used to measure each competency. The items measuring each competency make up what is known as a “scale”. In scoring, individual items making up a scale are summed together to create a scale score that represents the sum of responses related to a specific competency. These scale scores are often combined to provide an overall score for each candidate.
- Determining Minimum Scores/Ranking. This step in the process involves comparing scale scores against a predetermined set of parameters in order to sort out and rank candidates based on how their scores compare to a minimum score needed to ensure job success. Most often the end result of this process involves assigning candidates to one of three groups: Fail, Proceed with Caution, or Pass. There are a variety of ways to decide what group a candidate is placed in, and making these determination can be a very complicated part of the process. The bottom line is that this part of the process allows you to weed out persons who are not acceptable and to rank those who pass based on their match to your blueprint.
- Reporting of Information. Once scoring and ranking has been completed the results need to be passed along to those who are making hiring decisions. There are many different ways that this can be done. Most commonly the screening provider provides a detailed report that shows each of the critical areas on the blueprint and how each candidate rates on this area with a final recommendation about the applicant’s suitability. Many of these reports can be very detailed and contain behavioral anchors that link a certain score to behaviors that can be expected on the job.
My two cents worth on reporting is that many systems are helping contribute to information overload. For screening to be really accepted by the mainstream, the results must be integrated right into the administrative tools supplied by ATS systems and the results of the entire screening process need to be summarized in one or two indices that can be easily understood by any hiring manager or recruiter. I do think full reports are valuable, but may best be used for personnel files, developing targeted interviews to obtain more information about weaknesses, or for providing baseline info for other HCM systems such as training once a candidate is hired. Why is it worth all this trouble? Screening that includes science in each of the three areas outlined in this article can help ensure that the most qualified applicant is chosen and can effectively address many issues such as turnover, fit, absenteeism, & shrinkage. The small investment made in using good science will pay for itself rapidly in return on investment made in human capital. Stay tuned: Part Three in this series will discuss the questions to ask when you are evaluating online screening vendors.