Validation is a best practice that can provide both critical information about the ROI of an assessment and the documentation required to support its legal defensibility. Unfortunately, proper validation is not the norm when it comes to the use of assessments. While many companies make use of assessments that have been validated in the past or that do satisfy some of the requirements for test validity, conducting the validation work required to fully satisfy best practices and gain an understanding or ROI is often not on even on the radar screen.

When it comes to validation, my experience shows that the biggest stumbling block is a lack of understanding of just what validation is and why it is so important. While the concept of validation definitely has its complexities, it can be boiled down to a few simple concepts which are discussed below.

What is Validation?

Webster’s online dictionary defines the word “validity” as:

….well-grounded or justifiable : being at once relevant and meaningful <a valid theory> b : logically correct <a valid argument> <valid inference>

….appropriate to the end in view : effective <every craft has its own valid methods>

These definitions definitely hold true when it comes to employment testing. Ask an I/O psychologist and he or she will tell you that validation simply means the act of establishing two key things: 1) That anything used to make employment decisions is job related, and 2) That the assessment actually measures what it is supposed to measure (i.e., that the test is “accurate”).

There are a variety of ways to document the job-relatedness and accuracy of a test as a decision-making tool; however, a working understanding of validation should focus on two general types of validation, content validation and criterion-related validation.

Content validation: Quite simply, content validation involves the documentation of the personal characteristics (i.e., experience, education, knowledges, skills, abilities, values, etc) required to perform the job.

At a minimum, claiming a selection measure is content “valid” requires an alignment of test content to job requirements so that support for the job-relatedness of the test can be documented. This means that the job or jobs in question must be carefully evaluated and that the input of subject matter experts (incumbents and supervisors) be used to create a full understanding of the various things that are required for successful job performance.

The process used to establish the job-relatedness of test content is known as “job analysis.” Once information about job performance and related characteristics has been documented via job analysis, selection measures can be mapped out to match job requirements. For instance, if job analysis shows that the job requires fast and accurate typing, then the use of a typing test to hire applicants for that job is assumed to be content-valid based on its relation to job performance requirements.

Content validation can satisfy EEOC requirements for claiming a test is valid (provided that proper procedures were followed). However, settling for only content validation is selling yourself way short. The real value proposition when it comes to validation lies in the evaluation of the ROI provided by a selection measure. This information can only be provided by criterion-related validation.

Criterion-related validation: Whenever possible, the statistical evaluation of the relationship between selection measures and valued business outcomes is desirable. This type of validation is known as “criterion-related validation” and it can provide concrete evidence of the accuracy of a test for predicting job performance. Criterion validation involves a statistical study that provides hard evidence of the relationship between scores on pre-employment assessments and valued business outcomes related to job performance. The statistical evidence resulting from this process provides a clear understanding of the ROI provided by the testing process and thus helps document the value provided. Criterion-related validation also provides support for the legal defensibility of an assessment because it clarifies the assessment’s accuracy as a decision-making tool.

While criterion-related validation may seem mysterious, it has much in common with two more well-known concepts that are used to help find value within business processes: six sigma and business intelligence. Both of these methods require that data be examined in order to help clarify relations between various process components. The resulting information can be used to help streamline business processes and uncover meaningful relationships between various streams of data. The creation of a feedback loop using criterion validation is really no different.

The 1-2 Punch

In an ideal world it is best to have both content and criterion validity evidence. Documenting content validity is a minimum requirement for any pre-employment selection measure; however, content validation alone can’t provide any evidence for the ROI associated with a test or selection measure. Adding statistical validation bolsters the legal defensibility of an assessment and provides insight into ROI. Unfortunately, most companies do not perform criterion-related validation.

There are a variety of reasons for this.

• Criterion study not possible: A legitimate reason for not conducting validation studies is that there are simply not enough people in the job to allow for a reliable study to be conducted. The greatest shortcoming of statistical validation is the fact that its results are not credible unless the sample size is well over 100. Therefore small companies and jobs with few incumbents will preclude the ability to gather criterion-related validation evidence.
• Lack of resources: Many companies feel that conducting criterion-related validation studies simply takes too many resources. They are unwilling to spend the money needed or take the time to collect the data required to perform such studies.
• Lack of understanding: Many companies do not understand the concept of criterion-related validation and are not aware of the value it can provide.
• Lack of guidance: Test providers often sell tests as being “valid” because they have been validated for jobs similar to the one in question. While this may be true, it is not always an “out” that allows one to avoid doing criterion-related validation. Many test vendors do not provide advice or services related to criterion validation, leaving the user on their own to figure out this part of the equation.

What does all this mean to staffing professionals?

• At a minimum all measures used to select employees should be content-valid. Failure to document content validity means the legality of your selection measures is compromised.
• Whenever possible, it pays to conduct criterion-related validation to add additional support for content validation efforts and to gain insight into the ROI of selection measures.
• When reviewing tests provided by vendors, make sure to understand that a claim that a test has been “validated” can mean different things. Don’t assume that just because the test has been validated in another setting, that it is automatically valid for your particular situation.
• Lack of proper validation for tests and selection measures can be costly! Citing lack of resources as a reason for not validating is like passing over dollars to pick up pennies.
• Both types of validation can add value. Content validation provides legal piece of mind and criterion validation bolsters this and provides the ROI evidence needed to build a business case for using assessment.
• Educate yourself about validation and the options available to you. For more information talk to an industrial psychologist about validation and legal requirements and consult resources such as the Uniform Guidelines for Employee Selection Procedures.

As with anything else, it may take a bit of extra time and resources to do things right, but the extra effort will provide value and piece of mind.

For the last seven years, Rocket-Hire has surveyed users of web-based pre-employment assessment tools, so we again asked members of the ERE community to tell us about their usage of typical pre-employment screening, testing, and assessment programs. As with years past, we zeroed in on the pulse of pre-employment assessment usage. And in an increasing climate of legal scrutiny for testing, and the hoopla surrounding the Ricci case, we decided to focus the content of today’s article on two issues that are inexorably linked: Implications of evaluating one’s assessment strategy, and attention to relevant legal issues.

Those interested in obtaining a copy of our full report can email us (chandler@rocket-hire.com) and we will be sure to send you a full copy once it has been completed.Or, check out an upcoming Journal of Corporate Recruiting Leadership, probably the October 2009 issue, where we’ll have an in-depth analysis.

A Word About our Methodology
This year, 148 recruitment and hiring professionals completed the Rocket-Hire Online Assessment Usage Survey. Respondents were evenly representative of recruiters, recruiting leaders, HR executives, business owners, and hiring managers, and featured a wide variety of organizations and hiring situations.

Use of Assessment Tools
Overall usage of assessment tools was generally about the same as in past years — roughly two-thirds of respondents. Of that two-thirds, 54% are deploying both paper-based and online assessment, and 30% are using exclusively online assessment. The remaining employ only paper-based tools. Most use a variety of different assessment methods, with the majority using between one and three different types. The table below reveals the proportion of firms using various common tools.

Usage rates of common assessment tools

Skills and knowledge certifications and personality assessments continue to be popular, with indicators of cultural “fit” and various cognitive ability measures also widely used. In addition, we observed a surge in the adoption of web-based background investigation strategies.

Interestingly, 2009 highlights a trend emerging over the last few years: It is now just as common to find organizations deploying assessment company-wide as it is to find it used for only one or two jobs. These findings reflect the fragmented use of assessment that we have experienced in our client work. To be sure, there are tons of ways assessment can be used, but there is no one major trend in how or how much it is implemented.

Effectiveness of Screening and Assessment Tools

This year, 80% of assessment users felt that assessment had a “positive impact” on their organization. In contrast, only 3% of assessment users felt these tools did not add value. However, only 64% of assessment users collected metrics to judge the quality of their interventions, and 79% judged ROI using one or many indicators. However, definitions of “ROI” vary widely, with some approaches considering a broad, cursory opinion from managers, whereas only a few use a true scientifically derived method such as a validation study.

Wide variation in the real-world evaluation of ROI clearly reveals that, while many companies do collect metrics, few of them are going about this in a way that reflects best practices. This is cause for concern given the fact that these same best practices create the foundation for demonstrating the legal defensibility of assessment measures.

Legal Issues

This year’s Supreme Court decision in favor of firefighters who challenged the City of New Haven, Connecticut, has once again focused attention on the legal aspects of pre-employment assessment. In anticipation of the High Court hearing this case, we wondered how much typical users paid attention to legal and regulatory issues related to using assessment to evaluate candidates.

Percent indicating legal issues are “central to this activity” or “very prominent”

These results express both positive and negative connotations. On the positive side, it seems that a good number of companies pay attention to legal issues related to testing and assessment. On the other hand, perhaps more firms should consider devoting resources to demonstrating and documenting legal defensibility. The lack of understanding of what is actually required for ensuring compliance and defensibility may create problems as both assessment usage and legal scrutiny increase.

The energy spent on legal considerations does not relate to the extent of assessment deployed, or size of the organization, nor number of hires. As with overall usage of assessments, legal concerns do not follow any pattern across survey respondent demographics or assessment strategies.

Legal defensibility is not just another set of meaningless hoops that must be jumped through and checked off. Proper evaluation of ROI via validation and legal defensibility are actually two sides of the same coin, the common goal being to document the job-relatedness of all tools and processes used to select employees.

Though the continued economic turmoil has put a damper on hiring, the adoption of in-depth assessment tools continues to grow, especially as more organizations devote resources to find out if testing and assessment is worth it. The bottom line: Quality hiring tools can improve performance and employee engagement (i.e., provide ROI) while also ensuring the legal defensibility of hiring practices, but it is essential to demonstrate how much they actually benefit your organization.

A complete discussion of the intricacies and technicalities of validation, discrimination, adverse impact, and differential prediction is beyond the scope of the words I am writing today. Suffice it to say that this case has placed competing priorities in the use of testing in the spotlight. These competing priorities are using fair testing while striving to eliminate discrimination in hiring. While title VII of the Civil Rights Act of 1964 has attempted to provide some guidance in relation to these competing goals, the Ricci case has laid bare some critical issues that in my opinion certainly call for the government to re-evaluate and modernize the standards it has set.

We are mandated to use valid tests. Valid tests can often lead to minorities being hired at lower rates than those of other races. This is seen as OK as long as the test has been validated, because in theory this means the test is job-related and job-relatedness is the standard by which the legality of testing is determined.

However, what are we to do when sticking to the use of validation — as we have been asked to do — creates a situation that actually inhibits the goal of ensuring diversity and fairness? This has been a thorny issue for those of us in my profession for a long time. There is no magic bullet. The dissenting opinion in this case led by Justice Ginsburg rallies around the idea that the spirit of diversity and fairness should be the highest standard to which we aspire in hiring. It is hard to argue with this point … except for the fact that there are technical issues which can stand in the way of our achievement of this goal.

So, what does all this mean for hiring in the corporate world? I offer my humble answer to this question as follows:

Don’t Panic -- Police and fire testing is the most highly scrutinized type of testing known to mankind. Don’t panic based on the results of this case. Do use this as a time to think about your use of testing and where it may leave you exposed.

Validate, validate, validate -- In this case the validity of the test was upheld. In my mind the validity of the test, while an issue, was not the main issue at hand. The only reason the city tried to throw out the test was because it ended up being counter to its goal of diversity. Despite this, I cannot stress enough the need to validate all testing that is used to make employment decisions. It is the cornerstone of best practices in testing and provides the documentation you will need should you find yourself in court. Without such documentation, you are toast! As an added bonus, validation is the process that provides awareness of issues such as adverse impact. You may not even know you have a problem unless you take the steps to validate. Remember, ignorance of the law is no excuse!

Look at the bigger picture -- I agree with Justice Ginsburg that the overall goal of eliminating discrimination is the highest standard to which we should be held. In the corporate world this becomes an issue of fairness in hiring practices across the board. One of the biggest ways to guard against problems while working to achieve diversity is to look at the demographics of your workforce vs. those of the available workforce in the area. If these do not look about the same, you have a problem. This problem can be rectified by actively recruiting for diversity. Diversity training programs are OK, and of course I support them, but the best thing to do is to put your money where your mouth is and be aware of your demographics and seek to hire diversity at all times.

Seek out testing that has been shown to reduce adverse impact -- The Uniform Guidelines on Employee Selection Procedures pretty much lay down the law when it comes to testing. A key part of this doctrine is that one should always seek out tests that are known to have less adverse impact. We know that cognitive tests have the most adverse impact while also providing the best predictive accuracy (i.e., validity). Resolving this conundrum remains the crux of the issue, with the Ricci case as firefighter tests are highly cognitively loaded. In the real world I feel this issue is best addressed via awareness of what is required for the job and by seeking out selection procedures that we know can test cognitive traits while displaying lower levels of adverse impact. If you guessed that I was going to recommend simulations as the best way to accomplish this goal, you are correct! The issues of this case are yet another piece of evidence that clearly demonstrates the value of simulations over more traditional types of testing.

I look forward to the discussion that my opinions generate and I am glad to see my corner of the hiring world getting its brief exposure in the national media spotlight. I certainly hope that the awareness generated should serve as a catalyst for change.

I was recently reading an article in which many of the members of the mission control team were interviewed about the mission and the various roles and tasks they performed. One of the most interesting things was a discussion about the fact that almost all of the mission control team members were very young.

The mission required extensive use of computers and the ability to use computers to do things that had never been done before. Computer science was such a young field, and the moon mission so unique, that there were no persons with any experience doing what needed to be done. In order to accomplish the mission, NASA hired a group of people who had the ability required to work with computers and experience working with computers, no matter what the application. The article did not say what methods were used to determine how hiring decisions were made, but clearly a non-traditional approach was required. We all know the result of the decisions NASA made and we all know that the combination of workers, equipment, and planning resulted in a resounding success — the kind of win any corporation would be proud of!

Even though the Apollo 11 mission happened 40 years ago, it’s ripe with lessons for those of us in the hiring profession. This is especially true when it comes to creating strategies to use assessment in the future of the hiring and the workforce. My thoughts:

• As has always been the case, the bottom line is that understanding the traits valued for getting the job done and including the scientifically based assessments required to measure them in your hiring process will help ensure you achieve successful performance of your mission.
• Creating the future may often require taking a leap of faith in someone’s abilities to make a contribution. Testing can help make this leap easier by helping to identify those individuals who possess the raw abilities and characteristics required. A caveat here is that — as with any employment testing situation — time must be taken to identify abilities and characteristics critical to desired outcomes as a first step in the development of the testing program.
• Training and experience are an excellent way to help mold the raw material that brought to the table. When seeking to do something that has not yet been done using new technologies, training is essential, as is the ability for hands-on learning in which team members can educate one another.
• Ideas are and will continue to be the currency required for successful progress. Identifying applicants who are creative and possess the proper thinking styles will become increasingly important.
• Bringing together persons with backgrounds in different areas and asking them to function together as a team will be essential to success. New innovations and progress are increasingly requiring input from those with vastly different backgrounds.
• Given the above point, it seems logical that an increasing amount of attention be paid to ensuring a harmonious cultural fit between the members of the team. It is often mismatches in work styles and values that cause problems within a team dynamic. There are an increasing number of applications which allow organizations to measure, model, and optimize fit when creating and aligning work teams.

Technology is continuing to make the above points easier for those folks with out a background in testing and assessment. As with almost every other area in our lives, technology is radically changing the landscape. When is the last time you sent a fax? Could you be as efficient without email? We are entering an age of unprecedented accelerated technological advancement. The time horizon for quantum leaps in technology has been shortened to milliseconds when one considers the grand scheme of things. The computing power and things needed to put a man on the moon 40 years ago can probably be duplicated by several laptops now.

The things we are going to see in our lifetime are going to blow our minds. This statement will be true for those of us in the general business of hiring, and those of us who specifically concentrate on assessment. Virtual simulations of entire jobs, human interactions, and interactions between humans and machines are going to be the future of hiring. I have just started to see some of the first steps in this direction. What I have seen and learned so far is that it is going to require a diverse set of perspectives to create the hiring tools of the future. Folks in the testing and recruitment industries are going to need to team up with persons in other, seemingly unrelated industries in order to make it happen.

I am currently working on putting together a consortium that represents key stakeholders from the assessment and gaming industries as well as persons from other areas yet to be defined. I am just at the concepting stages at this point, so I encourage interested parties to post here if you want to talk about this idea.

One of the most exciting things for me at this year’s conference was the launch of SIOP’s new blog/interactive community site, the SIOP Exchange.

I was part of a team that created this blog in order to help promote I-O psychology and build an increased sense of community amongst SIOP members and other interested parties. I encourage those folks in the ERE community who are interested in the viewpoint of I-Os on topics related to our work to check it out. The Exchange offers RSS feeds that will help keep you aware of topics that may be of interest to you.

In addition to launching the blog, this year I participated in several panels in which assessment solution providers and the end users of assessments discussed important issues related to technology and testing. It is rare to such varied experience and expertise in the use of assessment in one place. I want to share some of the hot topics with ERE readers to help keep the ERE community updated on how testing and assessment experts are handling important issues that impact the use of technology based testing. Here is a quick rundown of some of the themes that were represented.

Technology is more accessible than ever: We have reached a point where the differences between the technology platforms of assessment providers have started to level off. Almost every company offering assessments now has a relatively sophisticated platform that can handle the basics of test delivery, scoring, and reporting. A good deal of providers also offer a nice candidate management system as part of their platform. One interesting facet of this movement is that I-O knowledge is starting to get “baked in” or embedded into the technology system. This trend is going to help make quality technology based testing available to small and mid-market companies. While I believe this to be a positive trend, we still need to be aware that there are trade-offs to be made. Several of these are discussed below.

Test security. As always, there was a good bit of discussion about the security of Internet testing. One of the biggest issues was the use of proctored vs. unproctored assessments. While there are some firms that currently do not allow their assessments to be used in an unproctored environment, the majority of providers will allow it. We are starting to see a variety of interesting methods to help mitigate cheating. My takeaway is that each situation dictates the need to decide between proctored or not. I believe that there are enough security strategies available that the negative impact of cheating is likely to be minimal.

Computer-adaptive testing. One of the most effective strategies to help thwart cheating is the use of tests that draw on large item banks in order to help ensure each test is different while also adapting the test content to the test taker’s ability level. CAT allows for shorter, more accurate tests. While it has been in use for years in the world of standardized testing, the leaders in the pre-employment testing community are starting to adopt this technology for their assessments. This marks a significant step forward in both security and usability.

Defining performance standards. We are still struggling with the line in the sand when it comes to thoroughly and accurately defining the performance dimensions to which an assessment will be linked. The rise of technology based hiring platforms has led to the streamlining of what has traditionally been known as job analysis. While this may be OK in some cases, we are still struggling to understand at what point we are taking liberties. My opinion is that thorough job analysis is always a good idea. Especially given increased activity by the OFCCP.

OFCCP audits. I had several conversations about increased activity in the area of OFCCP audits. This makes the use of best practices for assessment (job analysis, validation studies, documentation of adverse impact) even more important than ever. The cost of doing things right is likely to be much less than the fine you will receive if your audit does not go well.

Simulations (more to come in the Journal). The use of simulations is the cutting edge of our field right now. The offerings in this area are starting to increase in sophistication but are still mostly limited to call center and in-box type assessments as these translate quite well to a simulated environment. We have a long way to go in this area but I have seen notable progress in the right direction over the past year.

Technology and development. Assessment providers are continuing to link their pre-employment assessment products to onboarding and development products. This is a logical step when one uses competency models and understands that the pre-employment dialogue with a candidate offers useful data that serves to provide a baseline. End users of assessment have not fully caught on to the value of this viewpoint.

Every situation is different. One thing I gleaned from listening to assessment practitioners speak about their work is that every situation presents it own challenges. The contexts in which assessment is to be used vary quite a bit. Those seeking to use assessment correctly and ensure that it provides the level of ROI we know is possible need to ensure they enlist the help of an expert. There are lots of judgment calls to be made and it pays to have an expert to provide insight when important decisions are to be made.

Globalization. I-O psychology is more global than ever. The use of assessment has been rapidly spreading across Asia and Europe. It will be interesting to begin having access to data that can help us to understand the commonalities and differences across cultures and geographical locations.

A final overall impression is that we I-Os are still marginalized and underused. It was nice to sit in a room full of folks who know assessment works, know how to make it work, and can prove the value it adds, but frustrating to know that we are often not even given a seat at the table when important decisions are being made.

The purpose of this article is not to provide a detailed outline of the virtues of simulations (please refer to some of my earlier writings for this type of information). Beyond this, the crux of the issue is that simulations offer some really nice advantages over simple employment tests. These advantages include:

• A high degree of candidate engagement. Simulations are more fun and engaging then simply filling in radio buttons.
• A high degree of accuracy. Since simulations are miniature replicas of the job for which a person is applying, scores on simulations are likely to be strongly correlated with actual job performance.
• A realistic job preview. Simulations provide candidates with the opportunity to try out the job in question and allow applicants who do not feel the work is for them to remove themselves from consideration saving time and money.
• An employment branding tool. Fun and engaging hiring practices can really help reinforce an employment brand. Considering the trend in gaming and computer simulated environments, this may offer a competitive advantage when it comes to the coming generations of job seekers.
• Reduced bias. Simulations offer a way to help reduce bias and subjectivity in the hiring process due to their realism.

In order to better understand the future of job simulations for selecting employees, let’s take a quick look at the past and present state of affairs.

Simulations in the Past

Believe it or not, simulations for use in employment testing have been around for a really long time. The first simulations were used in industrial and manufacturing environments to help test an applicant’s ability to perform tasks that required strength and manual dexterity. Many organizations developed their own tests that were directly linked to the job in question. If a worker, for example, was required to thread a screw through a hole in a certain amount of time with a certain level of accuracy, then a mock-up of the task would be provided to potential employees. Pass the test and you are hired; fail, and it’s back to the ‘ol drawing board.

The fundamental idea behind these tests remains the same even to this day. This idea is that an employment test should have what is called “point-to-point correspondence” with the job. This simply means that activities found on the employment test are a miniature replica of activities that are required on the job.

Simulations for employment evolved beyond use in manufacturing to include exercises known as in-baskets. Here, applicants are placed in a situation where they are required to assume the role for which they are applying and handle the daily tasks associated with that job.

The drawbacks of each of the type of simulations used in the past are that the simulations themselves and the evaluation of the applicant’s performance on them were all analog. It was not possible to re-create many work environments and the low-fi nature of the simulations that did exist meant that humans were needed to score and evaluate applicant performance. This means that in the past, simulations were labor intensive and thus time-consuming and expensive.

Simulations in the Present

As with almost everything else in our lives, technology provided a serious upgrade to the evolution of employment simulations. While other areas of employment testing are still using much of the same content, simulations have benefited greatly from technological advances. For instance, it is now possible to recreate a great deal of work environments; performance on simulations can be evaluated in an automated manner; and simulations can be given remotely from anywhere in the world.

Here are some highlights of what is presently available in the world of employment simulations. I don’t usually use this column to mention specific vendors; however, in the present case I feel such mention is warranted given the value these products have to the evolution of simulation products.

Call centers: The bulk of simulations currently available for use in evaluating job applicants can be found in the call center space. Call centers are very amenable to simulations because the work environment (a series of computer programs and databases) is relatively easy to replicate and the tasks that make up job performance are easy to measure (data entry speed and accuracy, customer service, multitasking, etc). Some of the best call center simulations are those offered by FurstPerson, which has done an excellent job of creating technically sound and engaging simulation tools.

Manufacturing: Several companies are also currently offering excellent simulations for use in manufacturing environments. Today’s manufacturing jobs often place a higher premium on computer and logic skills than they do on things such as manual dexterity. Companies such as DDI and Select International have created excellent simulations for predicting a variety of valued business outcomes in the manufacturing sector.

Custom simulations: Simulations can be very expensive to create, especially when cutting-edge technology is involved. Most of off-the-shelf simulations currently available are off-the-shelf products that offer the ability for light customization. However, when it comes to accuracy and realism, nothing beats a custom simulation that has been created to directly correspond with the environment and requirements for a specific job. Shaker Consulting Group is a company that leads the way when it comes to building one-off custom simulations for hiring.

Managerial simulations and in-baskets: Managerial jobs are more complex than are entry-level jobs, and as such, creating simulations for these jobs has proven to be a difficult proposition. Modeling the job and the job environment can be difficult, and understanding behaviors that lead to success can be harder because successful performance is often a blend of a variety of traits. Censeo Corporation has recently come out with one of the first simulation tools that begins to offer a peak at the future of managerial hiring. This simulation uses scenarios and business problems to help evaluate key skills required for success in managerial positions. Beyond this product, a gap begins to be revealed in the ability to use technology to create simulations for advanced jobs. This gap highlights the drawbacks or limitations that currently mark the world of simulations.

While advanced managerial simulations (also known as assessment centers and in-basket exercises) are able to use technology to help create a realistic model of the environment and make administration of assessment content easier, these tools still require trained assessors to make evaluations. We are not yet at the point where we can evaluate complex behaviors in an automated fashion. We are facing a technological barrier in this area as translating choices and actions made by assessees into measurable human characteristics has not been successfully automated.

A Look at the Future

The best of simulations is yet to come. I firmly believe that as technology evolves, we are going to begin seeing a whole new breed of employment simulations.

I believe that the future lies in gaming technology. While this may not be true for every situation, most every job I can think of can benefit from game-type employment simulations. Take a quick look at the advantages I listed for simulations at the beginning of this article, and you can see that gaming magnifies all of them. Here are some of the companies that are working on the bleeding edge of gaming and employment-related simulations. The work these folks are doing offer a window to what the future may hold.

Brand Games: Specializes in creating video-based gaming to support employment branding initiatives. While it is not creating simulations to be used to evaluate applicants, the work it is doing is some of the most advanced out there when it comes to the combination of hiring and gaming.

The U.S. Military (profiled in the March Journal of Corporate Recruiting Leadership) has been creating some of the most advanced simulation tools out there. Many of these are specific to a particular role and are used for training, but they are also using gaming as part of employment branding exercises.

ETC-easy: It is currently working on the first customer-service gaming simulation ever created. I’ve seen the product (which is still in development) and I think it is a really great step. ETC is currently looking for validation partners for this product, so anyone interested in learning more should contact me directly to learn more.

Hudson Soft’s “Help Wanted/Job Island” video game: This product will be perhaps the most unique job-related game available when it arrives on shelves later this year. The game requires players to work in a variety of virtual jobs in order to discover what they like doing while engaging in video game play. I don’t think that the makers of this game are intending it to be used for any type of actual employment-related activities, but the mere fact that this game exists is very significant in my mind.

As advanced as the various products I have discussed may be, they are still far short of what the simulation of the future will look like. In the future we are going to see products that offer:

• Ability to translate actions on the screen into work-related performance dimensions. This is one of the biggest hurdles we currently face in terms of developing game-like simulations. Once we get over this hurdle, things will never be the same.
• Ability to score complex interactions that occur in an environment with few limitations (aka “Sandbox”). To make simulations realistic, they will need to offer a wide-open range of possibilities for the person engaging in the simulation. Providing the ability to attach scores to an almost limitless range of interactions is a significant hurdle.
• Potential for social networking, linkages, and communication. The movement toward web 2.0 and the ability for increased communication and flow of information will absolutely be a significant part of a good deal of future simulations.
• Continuum between selection and development. The connection between gaming and training is already one that is being cultivated and which clearly makes sense. There is no doubt that games used for selection can also offer opportunities for developing employees.
• Ability to create a data dossier that can follow one through the evolution of their career — say goodbye to the resume. In the future, individuals will be able to present their work history, skills, etc., within the framework of virtual interactions.
• Ability to ensure safeguards against cheating and ability to secure data. As with employment testing today, security and cheating are hot topics and justifiably so. The stakes are getting higher, especially in tough economic times such as those we are currently facing. The more open simulations and testing become, the more important this area will be to their success.

While these things may seem to be a long way,the development of all technology-based solutions is an evolutionary process. While progress may be slow, any movement forward no matter how small the steps is essential. Given the accelerated rate of technological change that is currently in effect, we need to be open to the fact that things we have not thought possible, will indeed be possible.

While its fun to speculate about the future, it is important to understand that simulations are not just a pie-in-the-sky pipe dream. There are things that are now available that can provide excellent ROI for companies looking to make smart, accurate hiring decisions. So, in the short term my advice for companies is to begin thinking about how simulations can be a benefit to hiring. Investment in simulation technology in the here and now can serve as a learning experience that can also have a strong impact on the bottom line. This represents a classic “win-win” situation.

As a relatively early adopter and a futurist, I have been doing a lot of thinking about how social networking may impact the world of pre-employment assessment. The first real application that we will see is the use of social networking to help provide accurate pictures of the culture (also called work values) within an organization. This type of matching is a big part of what we speak about when we discuss the “fit” between a person and an organization.

Matching job seekers’ cultural preferences to the culture of an organization is hardly new. However, the ability for individuals to provide data that can be used to create aggregate profiles is a game-changer when it comes to the concept of using culture matching to help facilitate fit. Most commonly, fit has been explored as a diad between one individual and a referent group within the organization. While these matches may be meaningful, they are somewhat static and one-dimensional.

One of the most exciting things about web 2.0 and social networking is the ability for large numbers of people to make evaluations or judgments about something in such a way that a relatively “true” aggregate picture of that referent emerges. This sort of thing has the flexibility to quickly account for changes in the aggregate as it is continually evolving. These aggregate profiles also keep things very honest and often provide a very good source of information for those looking into the relevance or value of something to them personally.

It is not hard to imagine how collective data about an organization’s characteristics, values, and culture could have major value for those who are thinking about working there. There are already a number of websites where one can get honest (albeit sometimes biased) narrative about what it is like to work at a company. Those of us who design tests know that there is way too much left to chance when using open-ended narratives as the foundation for decision-making.

But what if everyone involved filled out a standardized questionnaire that was designed to measure various aspects of culture?

This would allow users to speak a common language, to be calibrated to one another when they are making their judgments. There are two companies (www.Climber.com, and www.jiibe.com) that are providing the ability for users to complete a culture/work values questionnaire as part of the matching and searching process. Both of these companies are still exploring the possibilities here and I won’t go into detail about either one of their business models or current functionalities. Suffice it to say that they are both on the bleeding edge of using culture profiles to examine fit.

Both of these companies have two sides to what they do: the seeker side and the organizational side. Usually the organization is in control of the aggregate profile that is created to describe it or the various workgroups it contains, and the seeker is in charge of creating a profile that captures their values. But what if persons who currently did work, or had worked, for a company could use a standardized process to provide their input about what it is like to work at a company? And what if job seekers could match their profiles to these aggregate profiles as part of the job search process and be given feedback about their level of fit? All before even applying for a single job?

This would provide very valuable information. While the company profiles may not be “authorized,” that does not mean they don’t capture the truth. The truth is a very real force that companies can also leverage for their own internal purposes. The more data that is behind a profile, the more one can infer truth in what that profile says. Imagine a time when there are solid, data-based profiles about the values of many major employers, and job seekers can use this information to look for a good “fit” as one of the first steps in a job search. Perhaps they can even begin networking and conversing with others who are currently or have been employed at that company.

It will be very interesting to see what happens with the concepts I have discussed here. Certainly there are issues that will need to be worked out, as well as limitations. However, the concept I have discussed here is going to continue to evolve and will eventually provide a good deal of value to job seekers because it will provide them with information that they can use to make better, more informed choices. Internet job-searching has traditionally provided the job seeker with little control, information, or feedback about their job applications. This is going to be one of the biggest changes we are going to see as a result of the newest wave of web technologies.

The concept I have discussed here will also benefit those making hires because it will allow potential candidates who may not fit to think twice about applying, therefore pre-empting the need to deal with their applications or to hire a poor fit. I would also hope that organizations would take a good deal of interest in what their aggregate profiles on the open web look like and use that information as a diagnostic tool from which strategic initiatives may stem. There is presently tremendous opportunity for corporations to begin creating their own internal aggregate fit profiles for use in helping to understand themselves and make personnel decisions according to the fit between an individual and almost any aggregate or group.

Finally, I would be remiss if I did not offer my usual caveat. Employment tests of any sort do not tell the whole story. They merely offer relevant information to help facilitate informed decisions. Anyone who has ever had to make an important choice can tell you the value of relevant information to the achievement of the outcomes they seek.

This well-written and researched piece is somewhat groundbreaking in that it is the first article in a mainstream media outlet to provide evidence of cheating on employment tests. Those of us in the testing industry have always been concerned with the security of our tests and have taken a variety of precautions to defend against it, but this is the first time I have ever read actual evidence that documents the existing of cheating.

As a testing expert and someone who has a high degree of familiarity with Unicru/Kronos (the company whose tests are the subject of the piece), I figured it would make sense for me to weigh in on this important article. Here are some thoughts about the article and the issues it raises:

• Overall, the article should definitely serve as a wake up call to the testing industry that times are changing; but, I do not think that the problems it identifies should spell doom and gloom for employment testing. Security is an important issue for any industry that conducts business online (banking, shopping, etc). However, dealing with security issues has not served as a barrier to progress, but rather an evolutionary force encouraging adaptation and prevention.
• The case of Unicru/Kronos is somewhat unique in the testing industry because it tests such a huge volume of applicants each year. It is a safe bet to say that no other testing company does the volume that Unicru/Kronos does. Therefore I would be very surprised if the same systematic and premeditated cheating on personality-type tests was occurring with the tests of any other vendors. Differences between the way the Unicru/Kronos tests and those of other companies would make cross-test cheating a complete disaster for anyone who tried.
• Because the Unicru/Kronos test is used to provide a way to provide efficiency to high-volume hiring, its results are often used to make “go — no go” decisions. While it is not always advised to use a test as the only piece of information to eliminate an applicant from consideration, high-volume hiring often necessitates it. How else can one have any chance of evaluating 1,000 applications for 10 or 20 openings? Considering the current state of unemployment and the sheer number of folks involved, high-volume hiring starts to show some similarities to standardized testing. That is, lots of applicants and high stakes. While cheating on high stakes tests is not too common in the U.S., it is a significant problem in other countries such as China and India. I have had a good deal of experience implementing standardized testing in these countries and I can say with authority that we do have a variety of techniques that can be employed as preventative measures. As with any type of security, multiple layers provide the best defense. We as testing professionals have the ability to use item banks, alternate versions of items, parallel forms, and computer adaptive testing to minimize the chance that a reconstructed answer key will actually match a test.
• Consider the type of test in question when evaluating the potential for cheating and the impact it may have. The WSJ article claims the Unicru/Kronos test is a personality test. This is partially true, as it does have some strong ties to traditional personality tests. However, the most common personality tests do not actually have right or wrong answers for the individual questions. Rather, the scoring for the questions usually varies quite a bit relative to the specific job in which the test is being used (if the test has been properly configured). While the science behind Unicru/Kronos’ tests is beyond reproach and is actually very innovative, their tests are still a good bit different from those of the rest of the herd. Unicru/Kronos’ unique formula works for them and their tests are highly predictive when compared to many others out there. Why does this matter? I won’t go into the mechanics of it, but suffice it to say that it would be much harder to identify right or wrong answers on a more traditional personality test. Most traditional personality tests also have faking scales built in to identify those applicants who are answering in a socially desirable pattern. Figuring out the scoring algorithms that account for this makes cheating in any meaningful way even more difficult.
• Traditional personality tests by themselves really aren’t the best predictors of job performance. There is a ton of research to support the fact that cognitive ability tests are much better across the board at predicting job performance. These types of tests are much more susceptible to cheating, and test vendors are very aware of this fact. In many cases, these types of tests are administered only in proctored situations. If I were to really worry about cheating, it is cheating on cognitive tests that would be my major concern.
• Another interesting wrinkle in this whole thing is the fact that job applicants are actually resorting to cheating the companies that they wish to potentially work for. To me this is a clear indicator of the need for companies to take a good long look at not only the messaging around the test itself, but also at the manner in which the applicant is treated throughout the entire hiring process. Of course it is unrealistic to think that we will ever be able to keep all applicants from cheating, but I do believe that the more applicant-friendly the hiring process, the less likely applicants will be to cheat. Time and again I have seen the testing portion of the application process handled in a cold and unfriendly manner, with very little information about the reason for the test being provided to the candidate. This type of thing makes it much easier for a candidate to feel fine about breaking the rules. There should be messaging to candidates about the fact that testing is beneficial to them because finding a good fit between applicant and job is beneficial to all parties involved. No one likes to work in an environment where they do not fit in or where they do not have the competencies required to succeed.
• The bottom line here is that tests such as those offered by Unicru/Kronos do offer tremendous value. We I/O psychologists have collected more than enough evidence to demonstrate that testing, when done right, offers a huge advantage over more informal and unstructured hiring practices such as resume reviews and unstructured interviews. In the case of high volume hiring, the ability to provide an automated tool for screening out applicants is tremendously useful. Are these methods perfect? Absolutely not, but then again very little in this world is. Error and noise are accepted parts of the testing process. The goal of test providers and hiring professionals is to mitigate these to the best of their ability. In high-volume testing, cheaters make up a very small percentage of the total population of job applicants, making cheating just another source of noise that we must work on minimizing.

The issue of cheating is something that warrants a good deal of attention but one that should not stop our progress in the quest for better and more accurate hiring procedure. The issues discussed in this article serve to strengthen my stance that we need to be working on a new generation of hiring tools that are simulation-based. The more the testing part of the hiring process relies on scenario-based activities that mimic the work environment and the tasks that must be completed within it, the more realistic and engaging the application process will be for applicants and the harder it will be for someone to cheat. Of course we are a long way away from this goal at the present time, but we will get there eventually. Until that time, those using testing should be sure to make test security a critical part of the design and implementation of the testing process, while also bending over backwards to make sure the candidate is treated just as any valued customer of the organization would be.

The most significant change I expect to see in pre-employment assessments in 2009 is a slowing of uptake as some organizations slow their hiring to a trickle or cut things seen as non-essential (i.e., assessment) from their budgets. I am not going to miss the opportunity to suggest that cuts to the budget for assessment are unwarranted because a well-designed assessment program can provide ROI — no matter what the economic context may be. My thoughts about a slight slowdown in the use of assessment are just speculation; it will be interesting to see if hard data such as that provided by our 6th Annual Screening and Assessment Usage Survey supports this speculation. If you have not taken the survey yet, I encourage you to do so. It only takes about 10 minutes and the information you provide is very valuable.

Despite the possibility of a slowdown in the purchase of assessments, there are a number of trends, nine in all, that will help mark 2009.

Legal compliance will become an even bigger deal. This next year is going to be a monumental one for U.S. politics. No matter who you voted for, the fact that change is coming to our government is a reality. It will be interesting to see what the impact of the new regime will be on labor-related issues. My gut tells me that the reform agenda that will be prevalent in Washington may mean more aggressive enforcement of EEOC and OFCCP regulations related to fairness in the hiring process. The often-heard opinion that using assessment places one at greater legal risk is simply not true. An assessment program that has been properly implemented and evaluated can actually provide increased legal compliance. Score one for assessment in 2009!

The concept of “embedded assessment” continues to evolve. Even a diehard believer in the value of assessments such as myself should be willing to admit that assessment data is not the only piece of information that is useful when making hiring decisions. Good hiring is the result of an informed decision-making process. As technology that helps support this process approach continues to be developed, assessment will continue to be folded into the mix as a key ingredient. As time goes on, an increasing number of hiring products will include assessment in such a capacity that it will be relatively transparent, but will provide data points that will help to support decision-making. This also means that the traditional assessment firm will need to be ready to play well with others as their survival may depend on it.

More assessments available transactionally. One of the benefits that come with a decade of Internet testing is an unprecedented amount of data. This information is being used to help create off-the-shelf tests that are more accurate than ever. In the past, an off-the-shelf test would be extremely general and might have missed the mark when it came to measuring all of the things required on the job. The new generation of off-the-shelf tests leverages data to offer a much more complete measurement model. This trend will continue as an increasing number of companies offer tests that can be used transactionally, with little or no up-front work.

Hello, middle market. Trends 2 and 3 are definitely related to a continued push toward offering easy-to-use assessment-related products to organizations that previously may not have considered them. Productization and the creation of more accurate off-the-shelf tools will allow for a whole new type of product. There is definitely untapped revenue potential in this market segment.

Assessment content stays relatively stable. While the technology that is wrapped around assessments has changed significantly in the past decade, the actual content of the assessments themselves has not changed much. We have done a great job understanding which individual items work better than others, but assessments still require applicants to click radio buttons that correspond to personality-type items. This trend will continue into 2009. Radically different items or assessment modalities are still a ways off.

Simulations continue to simmer. Despite the stagnation in assessment content, I have seen some encouraging progress in the world of simulations. This progress will continue as new technologies become available. The future of technology and hiring lies in simulations. It is going to take some time until we have all the pieces in place to create what will be the first of a new generation of hiring tools. In the meantime the interest in simulations and the technology that is being explored are encouraging.

Continued links to development. The rise of interest in talent management platforms and the natural continuity between pre-employment assessment and performance management/development have continued to increase the links between pre-employment assessment programs and continued employee development. This movement will continue in 2009 as an increasing number of product offerings leverage pre-assessment data as an important input to the employee lifecycle.

Evaluation still not a priority. It seems like I bring this one up every year. Probably the one biggest point that continues to frustrate me is the fact that the majority of assessment users do not take the time to properly evaluate the impact of assessment on important business outcomes. I feel our current economic situation may serve to magnify this point, when sadly, it should do the exact opposite. Yes, assessment works, and the value it can provide is even more important during times when every dollar is scrutinized.

Increased self-awareness. Once we are looking back at these tough times in the rearview mirror, I think all of us are going to have learned some valuable lessons. Our entire industry is now on the hook to continue to add value, and we are all going to have to get resourceful, to adapt, and to learn about ourselves. The learnings we take away will hopefully help us form lasting habits that will help us all to emerge with a new mindset about what is valuable.

2009 promises to be an interesting time in the history of pre-employment assessment. I believe that the quality and variety of options will continue to grow and that technology will continue to allow us to do more with less. The timing for this trend could not be better and I hope that companies can begin to look at assessment as a profit center instead of viewing it as a costly drain on resources.

by Dr. Charles Handler and Mark C. Healy, M.A.

We I/O psychologist-types tend to be real data hounds. Much of the work we do for our employers/clients involves the use of data to investigate specific hypotheses in order to illuminate the underlying truth in a situation. The outcome of this work often has tremendous value to organizations because it provides them with hard data on which strategic decisions can be based. Additionally, the collection and analyses of data often helps us to identify new trends that we haven’t yet thought about.

Many of you who follow our articles know that we have a keen interest in the pre-employment assessment industry, and write quite a bit about its trends and happenings within. Our interest in data and trends has led to an annual online screening and assessment usage survey.

The idea for this survey was born back in 2002, when we became frustrated over the lack of available information about the usage of pre-employment screening and assessment tools. This lack of information has been a challenge because though everyone seems to be saying that screening is becoming a hot area, there’s little actual data available to confirm this statement or to tell us how hot it really is. This lack of information also makes it hard for those of us who follow this industry closely to provide factual information about how companies are using online screening and assessment tools, and what the results of this usage have been. This year we have made a few changes to the survey questions to help us be sure we are staying up to date with some of the major trends and issues that pertain to assessment and the manner in which it is integrated into the hiring process.

At the end of this article, you’ll find a link to this year’s survey. Take a few minutes to help other members of our community by providing information about your company’s screening and assessment practices. The more data that’s collected, the clearer existing and emerging trends will become. Last year, we had a record number of responses, a fact that seems to indicate the increased interest level in screening and assessment. Given the steady increase in interest and the lack of information about this industry, we feel the results will continue to have value for the ERE community. We look forward to reporting our findings right here on ERE sometime this coming Spring.

In order to provide some extra motivation, here’s a quick summary of the trends identified in last year’s results.

Summary of Findings

One hundred and forty one professionals completed the survey, representing a very small increase of over the previous year. Respondents represented a wide range of company sizes. The results confirmed what most of our previous surveys have found:

• Having an ATS installed is now nothing particularly innovative, especially if you work in a medium or large organization. For some, it’s hard to even imagine the old stacks of resumes and bulging file folders of applicant information.
• Prescreening and assessment are continuing their gradual penetration into the mainstream of recruitment and hiring.
• Qualifications screening, personality inventories, and skill and technical certifications continue to be the most popular online assessment tools. Assessments of cognitive abilities as well as fit with the company culture have expanded their footprint as well.
• Respondents report a lack of understanding, weak budgets, or a general lack of support for online tools as their primary obstacles to adoption or greater use of modern prescreening and assessment technology.
• Those organizations that formally evaluate their hiring practices tend to support the use of prescreening and assessment, but a large number of organizations are still failing to evaluate the effectiveness of their screening and assessment tools.

There clearly is a growing interest in scientifically derived hiring tools. This trend is encouraging but not particularly surprising; these numbers will continue to grow as more organizations understand the value in properly evaluating the impact of screening and assessment tools, seeing for themselves how quality hiring tools can improve a workforce.

This Year’s Survey

The goal of this year’s survey is to continue building on the information gathered from past surveys and to verify the fact that usage rates for online screening and assessment are increasing.

In order to help provide the ability to track changes in usage rates, this year’s survey is similar to last year’s, with a few minor changes to help account for trends that have developed since last year. Specifically, this year we have reduced the focus on screening systems and added additional questions about the impact of the economy on assessment programs, legal issues related to assessment, and budgeting/ROI for the use of screening and assessment tools.

While this is hardly a scientific survey, and its length limits the depth of the information that we are able to collect, there is tremendous value in the information your survey responses will provide. For instance:

• This survey will help to provide some reality to speculation regarding increases in the use of both online screening tools and online assessment tools.
• Your survey responses will provide information that is extremely useful for gaining a broader perspective on the evolution of the entire staffing process.
• The results of this survey will help provide companies with some basic benchmark information about how online screening and assessment are being deployed.
• The data from this survey will confirm our speculation that while many companies are using online screening and assessment tools, few are collecting the metrics needed to help them understand the true value they are providing. This is a problem that must be addressed, and we hope that verifying its existence will help to serve as a wake-up call.

All survey responses are confidential and individual responses will not be shared with anyone. By participating, you will help provide yourself and other members of the ERE community with up-to-date, accurate information about trends in the use of online screening and assessment tools.

We are interested in collecting information from anyone who is involved in the staffing process for his or her organization. This includes recruiters, hiring managers, staffing and HR executives, consultants, etc. We welcome participants from companies of all locations, sizes, and industries. We are not able to use information from individual vendors of screening and assessment tools or persons who may consult to multiple organizations.

Just click here and you’ll be taken to the survey page. This page has all the information you will need to complete the survey. Please feel free to forward the survey to anyone you feel might be interested in participating. If you have any questions, just contact chandler@rocket-hire.com.

The results of the survey will be featured in an ERE article sometime this coming Spring. We look forward to sharing our results with you.

High-Level Observations

• The rise of talent management. This seems to be the age of  “talent management” when it comes to the use of technology in HR. I saw a ton of companies offering “talent management systems.” These platforms use technology to cover a broad footprint of key HR areas/functions such as branding, recruiting, onboarding, learning, development, and communication. Talent management products are starting to provide HR practitioners with a technology based backbone that will allow them to integrate major HR functions. The integration of more functions into one platform is a trend that can have significant value given the traditional walls that tend to exist between the major areas of HR in larger organizations. There seems to be variation in the functions offered by the various talent management platforms as well as some gray area around what defines a talent management product. These gray areas are nothing new, and are indicative of the nature of an industry-wide trend that is causing vendors to jump on the bandwagon. While my overall thoughts about talent management products are definitely positive, one wonders how many vendors have just dubbed themselves as “talent management” providers to be trendy, and have not significantly changed their products.

• ATS Not Hip Anymore. Is it just me, or does there seem to be a significant decrease in companies selling products referred to by the term “ATS”? I could not help but think that much of the momentum in the use of hiring and technology seems to be in the idea of the broader, strategic, idea of talent management and less in the more tactical area of applicant tracking. The concept behind applicant tracking and its related functionalities are still of great importance; it just seems that the term itself is losing favor as traditional ATS functionalities are being baked into other products such as advanced screening/assessment management platforms and talent management systems.

• Goodbye paper resume. It seems that there is continued movement toward removing the resume from the hiring process. This is definitely true of the paper resume, but there is a broader movement afoot to deconstruct the resume and extract the types of data it usually provides (i.e., skills and experience). Key information about candidates is being collected via other opportunities in the search and application process such as the creation of detailed profiles that have fields that guide the entry of important candidate information in a standardized format. Parsing, another method of deconstructing resumes, seems to be continuing to evolve as a means of extracting important data from resumes. Resume deconstruction methods allow for much more efficient searching, and matching between candidate data and important job requirements. These methods are part of an overall trend that will see the integration of a variety of candidate information into a digital platform that will allow it to be standardized, categorized, and compared to key requirements for a job or career. While we have a long way to go toward the complete death of the resume, technology is helping us to continue the slow march towards the inevitable.

• Web 2.0 is hot. It almost goes without saying that the latest in HR technology is leveraging the benefits of easy access to information, communities, and data that are at the core of Web 2.0. Web 2.0 was everywhere at this show. It seemed to me that the applications of web 2.0 seemed to be concentrated more in the space of performance management products, but there is no doubt that it is having a large impact on the thinking of those creating recruitment-oriented products as well. I saw a heavy focus on the use of social networking tools such as Twitter and Facebook and on products that provide companies with platforms on which to build communities. The data-driven nature of web 2.0 provides a good deal of promise for the area of recruiting and hiring.

• We’ve come a long way. The products I saw at this show provide a sign of continued progress and evolution that seems significantly more advanced than the products available five years ago in many ways. While the core function of many products may remain the same, we are continuing to develop new ways to connect people and information. The products I looked at seem slicker, cleaner, and more usable than ever before. While it is often hard to cut through the smokescreens when one is looking at products on the tradeshow floor, the overall level of tech-savvy seems higher than ever. I was able to see firsthand the positive impact of technology on a variety of recruiting- and hiring-related products. Of course keeping up and separating faddish technologies from those that truly are game-changers will continue to be a challenge in years to come.

Assessment-related Trends

• Assessment is still an outsider. While there were a decent number of assessment vendors at the show, as an overall area of HR technology, assessment is still a very small piece real estate. This makes sense to me given the problems that organizations seem to have in understanding the value assessment can provide.

• Two directions of evolution for assessment. Technology is helping assessment to slowly evolve in two directions. The first of these involves assessment being integrated into the functions provided by bigger, broader systems and products such as job boards, ATS, and talent management. Embedding assessment into other products makes sense when one understands that the core value of assessment is to provide information that can be used in conjunction with other information to support informed decision-making. Pre-employment assessment actually may have the most value when it is an integrated part of a process-based approach into which its results are integrated. This trend has been very slow to develop, but we will continue to see a trend toward embedded assessments. The second direction in the evolution of assessment is toward the productization of assessments to be sold transactionally. While assessment has been sold transactionally for decades, the present state of evolution leverages millions of data points to help provide a new level of clarity about what content predicts certain traits, behaviors, and outcomes. This evolution allows for off-the-shelf products that are more accurate than ever before, while providing documentation to support the relevance of these products for specific jobs.

• Hello, talent management providers? Assessment seemed to be conspicuously absent from the slate of services provided by most talent management systems. While assessment is part of the deal for some of the big players in talent management, it does not seem to be a core part of the concept at this point. This is upsetting given the value assessment can have, not only during the hiring process, but throughout the entire employee life cycle. Anyone who considers themselves to be a player in the realm of talent management should strongly consider embedding quality assessment tools.

• Platforms evolve nicely. The software platforms that accompany pre-employment assessment products are evolving nicely. I refer to these as “candidate management systems.” Today’s candidate management systems offer many of the functionalities once seen only in ATS products. This evolution should help support the value proposition of assessment as products become easier to use and provide additional decision-making support.

• Simulations still lacking. While there has been some definite positive movement toward the creation of more advanced simulations, we still need to be pushing ahead in this area. While there are some nice simulation products currently available, the future of hiring will continue to move toward the increased use of simulations. Not enough steps are being taken toward the development of truly innovative and unique simulations. This is an artifact of a lack of buyer interest and the absence of “killer apps” that are needed to allow simulations technology to advance.

When it comes to HR technologies that support the recruiting and hiring process, we are moving in the right direction and have come a long way over the past five years. The technology level available for assessment both as a core process and as a supporting cast member is more advanced than ever. However, there is a lot more that we can be doing to integrate assessment into the products that seem to be marking the trends in HR and technology. Assessment still seems to remain the misunderstood stepchild of HR, which is a shame since there is so much evidence for the value it can have. I remain optimistic that assessment will continue to be brought more tightly into the fold in years ahead.

But many folks predict that it’ll be increasingly harder to find qualified applicants to fill job openings, especially when it comes to white collar, managerial, and professional-level jobs. We have all dealt with numerous situations where the age-old “mirror test” (hiring anyone who is breathing and thus has the ability to fog a mirror) is the only thing required to fill a position. In such cases, it’s harder to make an argument for using something that will actually serve to further reduce the number of applicants to be considered.

Assessment can have value even when one has few applicants to chose from or even when there is a shortage of qualified applicants. Below are five good reasons why it still makes sense to use assessment, no matter what the labor market looks like.

Reason 1: Assessment can be part of a good sourcing and branding strategy

Assessment has begun to occupy new places within the hiring process. Over the past five years we have seen it begin to be folded into the job searching and matching process. Assessment is a great way to help match job seekers to openings that are a good fit for their background, skills, and values. Even in a tight labor market, job seekers can still benefit from some direction to help them make important decisions about what jobs and organizations are right for them.

Compelling employment branding is the first part of the equation here. Much employment branding seems to be a regurgitation of the same old song and dance. What company doesn’t value diversity? What company doesn’t care about the environment? Assessment can help us to get past the fluffy, generic employment branding stuff and get right to the heart of the matter: matching people with the right employment situation.

Recently, assessment has become a key ingredient in “values matching,” a branding and sourcing strategy that can really make a difference. Values matching will continue to be a hot area because job applicants wants to ensure they “fit” with the culture and value of the organization. Assessment is an excellent way to provide a index of “fit” that can be used to help applicants to understand how well what they may fit with an organization, or even with a specific role, job, or workgroup within that organization.

Organizations that fill their hiring funnel with applicants who possess congruent values and have some degree of fit will find it easier to be sure they hire persons who have a good chance of sticking around longer and being more productive. Values matching can be used as part of an initial sourcing strategy by including it as a key element in the creation of job profiles used to match applicants with openings or as part of the search process on corporate web sites. Even if there is only one applicant for a position, if that applicant is a good fit, hiring them represents a good decision for all parties involved.

Reason 2: Assessment can still provide insight needed to support good decision-making

Suppose there are only two applicants for one open position. Whomever is making the hire still must make a decision between the two applicants. The decision-maker can still benefit from having some data to better understand each of the applicants from which they have to choose. Most employment decisions are made using resumes and unstructured interviews. These are the two most common tools available to hiring personnel. While both of these tools do have some value, they function best when used in conjunction with information that can help those doing the hiring to read between the lines. Assessment is a perfect tool to help those making hiring decisions to better understand each candidate. Informed decision-makers are more likely to make accurate decisions than those who must make decisions in the absence of good data.

The results of even a basic assessment can provide decision-makers with additional data points to help them make a more accurate decision. The number of applicants one has to choose from does not alter the idea that informed decision-making is the way to go. So, even in a tight labor market where one is lucky to have even a few applicants, it’s still valuable to provide decision-makers with tools to help them better interpret subjective information such as resumes and unstructured interviews.

Reason 3: Assessment can help you hire for potential

One common strategy in tight labor markets is to shift focus from hiring for a specific set of skills, abilities, knowledges, etc. needed to do a specific job toward hiring for potential. In such cases a few key abilities or characteristics are identified and applicants are evaluated based on their ability to bring this valued “raw material” to the table.

This change in focus allows the organization to hire those who may not have had the exact experience needed to do the job for which they are applying, but who have what it takes to learn the job. Such a strategy requires a strong focus on training and development. Still, such a program allows organizations to take a broader focus when looking to fill open positions. College recruitment and hiring programs are an excellent example of this. Hiring individuals straight out of college allows companies to fish in a bigger pond. Assessment is an excellent way to assess potential. Basic cognitive ability and problem-solving assessments provide an easy way to identify individuals who have a basic set of tools that will allow them to be an asset to the organization. Setting up such a program can often be less complicated than developing an assessment program that is tied to a specific job. There are tons of good quality, off-the-shelf assessments that have been created with the specific goal of measuring general constructs. These can be plugged right in, with less up front work than may be required to create a job specific test battery.

The idea of hire-for-potential, train-for-success is an attractive proposition in a tight labor market.

Reason 4: Assessment can support onboarding and development

Assessment is also a valuable tool because it can provide a good initial picture of an applicant’s developmental needs.

Most assessment providers have the ability to create an initial development report based on the results of an applicant’s pre-employment assessment. Even if there are few applicants for a specific job, using an assessment as part of the hiring process helps the new hire hit the ground running. Assessment data can help with onboarding by allowing the new hire’s manager to have a good idea of developmental needs on day one. This information can be used to create an initial development plan and to provide a baseline for future performance management and development activities. Such a strategy can also have a positive impact on things such as commitment, satisfaction, and turnover. Research has demonstrated that good experiences during the onboarding period can have a positive impact on each of these important outcomes.

Reason 5: Assessment can help the organization to better understand itself

You can’t expect strong results from the use of assessment without a clear understanding of what needs to be assessed, and why. Usually, this involves spending some time to take a look at both personal and organizational factors that must be addressed using assessment. More sophisticated implementations involve the ongoing use and development of an organizational competency model. Such a model allows the organization to define the key elements required for success across all jobs as well as for each specific job.

While this is not always easy, taking the time to break jobs down into the components required for success provides the organization with important insights about what is required for success. This information is valuable for hiring, but it also has value for other important processes such as development, training, and succession planning. Going through the effort required to anchor a good assessment program has value because it helps the organization learn about itself. This learning extends beyond the ability to identify key determinants of success. Using assessment properly requires a good bit of learning and experience. Why shy away from assessment during a time that can provide valuable hands-on experience? By choosing to use assessment in good times and bad, companies can gain experience that will benefit them in the long run.

No matter how many applicants there are for a given position, hiring the right one should be the result of an informed decision making process. The information used to support this process should be directly related to the various things required for success at the job and the organization. This is a universal truth. Those organizations that begin to cultivate this mindset will realize a significant long-term advantage that extends past the value of good hiring, into the realm of developing, managing, and retaining talent.

In this article, I praised the progress being made in understanding the value of quality assessment products in the modern hiring process.

I strongly believe that the words I wrote in 2006 are as relevant today as they ever were; in fact, they are more relevant now than ever. The days of the big job boards and their keyword matching strategies are numbered.

A lot has happened in the world of online recruitment in the last decade. The focus of most of the evolution has been on candidate quality and on increasing accuracy. This is in opposition to the focus a decade ago, which seemed to be on quickly delivering a large volume of candidates. What we have learned since then is that without a way to quickly evaluate candidate quality, we end up saddled with information overload. So, the objective for recruiting products is shifting to one that is focused on the alignment of personal characteristics with those that are required for performance on the job. This is a good thing. This movement has definitely been aided by the choice of several leading career portals to embed candidate quality into their bag of tricks.

A slice of the content of the original article from 2006 appears below. It has been updated to include a few examples of a few career portals that are now using assessment to help them better serve their clients (both job seekers and employers) by providing them with a way to quickly focus on candidates who have what it takes. Before we get started, I want to offer a few disclaimers here.

1. I am not saying that using assessment as part of the job search and match process provides all the data needed to make a good hiring decision. Rather, I am saying that it provides an excellent way to address the idea of “garbage in, garbage out.” By creating an applicant pool of individuals who are at least qualified and motivated, we are taking a very important first step towards ensuring systematic quality in the hiring process. I think of it as pre-screening that comes before pre-screening because it provides an applicant pool that will give one much better odds of making a successful hire.
2. In this article I am going against my policy of not ever naming the names of any vendors who provide the type of products discussed. Please note that none of the vendors discussed here have been included due to anything other than my personal opinion that they are good examples of the trend I am documenting. I was not compensated by them, nor does my inclusion of them in this article mean they are the perfect solution for your problems.
3. Although I am an expert in this space, I do not know everything, and there is definitely a chance that I failed to mention a company that provides the exact type of service I have described. I encourage someone from such companies (or a fan of one of these companies) to contact me to let me know what you are doing.

Below are some relevant highlights from my original article.

Folks are finally starting to get the idea that assessments are but one component of a broader process designed to help hiring professionals make systematic predictions that result in good hiring decisions. There’s much to be gained from this process-oriented approach, so it has been great to see assessments being used to help at various phases of the hiring process.

Despite the continued integration of assessment, there’s one area in which assessment has been underused. This area is the use of assessment tools during the job-searching process. Specifically, the integration of assessment tools into the searching/matching component of career portals.

Traditional methods available to users of career portals for locating jobs are extremely crude, consisting mostly of keyword searches or matching based on simplistic profile elements.

The fact that this basic process has endured as the standard for almost a decade now clearly reflects the quantity-over-quality focus that has reflected the marketing-centric attitude of many job boards since day one.

Blocking Out the Noise

This crude matching process results in excess noise. For applicants, noise means they’re presented with, and encouraged to apply for, a ton of jobs for which they’re not qualified. For hiring professionals, noise means no mechanism to assist them in making an initial high-level determination of applicant quality.

Noise makes it difficult to make good hiring decisions because it greatly increases the chance to make systematic errors.

The good news is that I am starting to see a shift in focus that will go a long way toward a reduction in noise and an increase in the ability to differentiate candidates based on various quality factors.

This shift involves the use of assessment tools as an integral part of the services provided by career portals. The use of assessment to help fill the hiring funnel with applicants who have the attributes required for success has numerous advantages, all of which are explained by simple probability theory.

Think about it: the more qualified the individuals in your applicant pool, the better chance you have of hiring someone who has what it takes. Over the past few years, I have learned of an increasing number of companies who are seeking to change things by integrating assessment into the matching process.

The basic aspects of this include the following:

• Seekers create profiles. This part of the matching process works as part of a registration process, before a job seeker has even expressed interest in a specific position. Along with other information collected during this process, the job seeker creates a profile based on a short assessment of basic qualities such as work attitudes, personality measures, etc.
• Corporate users define high-level requirements. This part of the matching process requires hiring personnel to create a template of the basic things it takes to do the job well. This often includes standard things such as skills and experience but also includes qualities often measured by assessments (personality factors, dimensions of fit, work values, etc).
• Noise is squelched. When a job seeker searches for a job using the profile he/she has created, results include jobs for which they are suited based on the compatibility between their profiles and those of available jobs.

There are many twists on this formula but the basic idea is the same: deliver an applicant pool that has less noise and makes the recruiter’s job easier. This is the start to ensuring quality hires because it’s sure hard to hire superstars if they aren’t even in your applicant pool to begin with.

Admittedly, my description of this process is overly simplistic. In job matching, just as with making hiring decisions, assessment data should serve as one of many data points that are used to identify the best candidate for the job. Truly effective matching systems will use parameters that include other key determinants of success such as experience, knowledge, and skills.

Weighing the Pros and Cons

Overall, the use of assessment in the matching process offers the following five advantages:

• Reduces noise. Allows filtering based on actual job-related parameters.
• Helps determine corporate fit. This process is a good way to find and hire applicants who fit a company culture.
• Increases quality. Allows insight into applicant’s ability to deliver what it takes for success.
• Educates job seekers. Allows applicants some insight about themselves and how suited they are for a particular job or career.
• Results in branding. It can provide a good applicant experience, leading to support for employment brands.

Of course, there are potential drawbacks to the use of assessment as part of the matching process. These include the following five issues:

• Learning curve. Users must learn to understand how to define jobs in terms of relative importance of the parameters used for matching.
• Accuracy. It is critical to match the templates against which applicants are matched to accurate reflections of the job.
• Uncertainty. Predicting performance is a difficult task, to say the least. Assessment-assisted matching is not a panacea and does not eliminate possibility that mistakes will happen.
• Quality control. To be effective, assessments must be created using the proper methodology; substandard assessment products will produce substandard results.
• Legality. While the use of assessments in matching is not illegal, it does require awareness of some potential legal ramifications (see original article for full coverage of legal issues related to the type of products discussed in this article).

So who is actually using some sort of assessment as part of a career portal or job board? I believe the companies below are all doing great things to move us forward toward an era of embedded assessments that can help reduce the noise associated with finding and applying for jobs online.

Climber: Climber.com uses some really cool technology to create a comprehensive applicant profile that includes an assessment of work values. The complete applicant profile is then compared to a profile created by employers making job postings. Part of this process includes an employer work-values profile that helps capture the values of the organization in order to help ensure that users (both applicants and employers) are matched based on a mutual “fit.” Climber goes beyond the norm for this type of exercise by using sophisticated data analysis techniques and AI in order to help optimize the search/match process for its members. It is truly on the cutting edge when it comes to delivering candidate quality via tight searching/matching through the use of sophisticated data analysis techniques and AI.

Jobfox: Provides applicants with the opportunity to take a short personality assessment once they have registered. While the results of this assessment are not currently used to help match job seekers and employers, Jobfox members are able to see the results of their assessment and use them for self-understanding and developmental purposes. Jobfox also has a very nice skills matching technology in which applicants take the time to create a skills profile that is then matched against the skills profile that is created for each position that is posted by organizations using Jobfox. The result is a match score that provides candidates with an idea of which of the jobs available on Jobfox are a good fit for their skills and experience.

Monster: Monster has partnered with DDI, a leading assessment firm, to provide its clients with the ability to add assessments to any job posting. This product is known as the Monster Performance Assessment or MPA. The ability to provide the MPA for most job postings is made possible by DDI’s extensive experience with assessment tools as well as the data it has collected over the past several decades. The MPA package represents a groundbreaking offering, as it is the first time assessments have been sold transactionally as part of the process of posting job openings. The MPA does not use assessment to help candidates identify jobs for which they are matched; rather, it provides recruiters with much richer data about a candidate’s qualifications for the position of interest. This allows recruiters to quickly shortlist candidates based on a set of standardized, job related data. While there are still some details being ironed out, Monster’s approach represents an interesting and potentially effective way to help deliver quality. Monster is also the only one of the mega-boards that has been bold enough to step out with an assessment-related solution.

eBullpen: eBullpen is a job board that uses a personality assessment as part of the process used to match job seekers with available openings. eBullpen has been working on its process for several years now and provides a legally sound, useful approach. This approach asks companies posting jobs to outline the personality traits, activities, and requirements for the job. Job seekers provide data on their personality traits, work preferences, and qualifications. Matching alogrythms are then used to help match job seekers with available openings.

Each of these four companies are stepping out into uncharted territory and is doing so with a slightly different approach. The specifics of these approaches aside, the big picture is what matters the most here.

This big picture is the idea that each of these folks provide a way for job posters to break down a job into a series of key data points which are then matched to a series of related data points in a profile or assessment snapshot created by job seekers.

Detractors have often lobbied against this kind of matching, saying that no candidate (or company for that matter) is interested in taking the time required to create the profiles needed to fuel this kind of process. I firmly disagree. Good hiring takes an investment on the part of both the job seeker and the employer, and good investment often takes a bit of effort. The old adage, “you get what you paid for” along with my personal favorite “garbage in, garbage out” both apply to this situation. I am confident that as we travel further down the road of embedded assessment, the data will bear out the fact that the small amount of time taken to fuel the matching process will prove to be time well spent

Despite this, within the next five years I expect to see career portals offering an increasing amount of value by beginning to change the focus from quantity to quality, with assessments as one of the core elements of this switch.

I think all stakeholders involved in the hiring process, including applicants, stand to benefit from this trend.

Article and research by Charles Handler and Mark C. Healy

Over the past decade, there has been a significant increase in the number of assessment tools available for use in both hiring and developing employees at all levels of the organization.

By far the most common use of such tools has been for selecting hourly and technical workers. This makes sense, considering that the bulk of many organizations’ employees reside at this level and because success at many of these jobs usually boils down to a few key attributes such as:

• How fast can the person learn the job?
• Is the person going to steal, ditch work, etc?
• Does this person “get” customer service and will they help customers acquire what they need?
• Can this person communicate effectively?

As organizations gain more experience with measuring applicant qualifications, it has become common to find online assessment within the hiring process for hourly and skilled technical positions. Broader sales and marketing efforts by vendors and consultants have led to dramatically increased sales of these products. But what about managers and executives?

As we found in our latest survey of online prescreening and assessment trends, having candidates for supervisory roles (from 1st level all the way to the C-suite) complete questionnaires, tests, and simulations online is common, but there still seems to be a lack of assessment at these levels, especially when compared to the use of assessment for hourly jobs. Nonetheless, adoption of these tools has grown over the last 10 years and is likely to continue to do so over the next decade.

While the basic model for success with predictive assessments still holds, there are some key differences between successful assessment for managerial and executive-level positions:

• Managerial jobs require a slate of skills that are not often required for success at simpler jobs. These include things like leadership, business acumen, managing performance, strategic decision-making, conflict resolution, etc.
• A mistake at this level of hiring can be much more costly than for an hourly hire, mostly because managers are responsible for potentially hundreds of people and millions in both costs and revenue acquisition.
• Managerial hiring has a tighter integration with ongoing developmental strategy and activities as organizations look to maximize their investment in an individual by understanding and developing potential from within.
• There is a broad range of job complexity once you hit supervisory levels. The selection process for a front-line leader can differ vastly from that used for executive roles. Executives often complete day-long assessment centers while first-level supervisors are more likely to be presented with a simple in-basket or situational judgment exercise, which is scored automatically.
• Feedback becomes a more integral part of the assessment process. Pre-employment assessment for hourly jobs almost never provides candidates with any feedback at all.
• When making promotional or hiring decisions about managers, assessment requires a “whole is greater than the sum of its parts” approach. Trained assessors (often both external and internal members of the organization) make ratings based on a variety of data.

The above differences are clearly reflected in the past, present, and future of managerial-level hiring and development tools. Let’s start with the recent past.

The Past

Other than interviews, technical knowledge, and previous job experience, best-practice managerial assessment over the past 50 years has typically involved mimicry of the most important tasks faced by supervisors of both people and process.

Typical tools include:

• In-basket simulations. Also known as an “in-tray” outside of the U.S., these often-intense exercises imitate the administrative work of a manager in a fictional company. The traditional version of this tool presents a candidate with a stack of memos, a calendar, process descriptions, employee files, and other paperwork. The task is to respond to issues contained in the memos, with attention to the supplemental information about the company and its employees.
• Role-plays. Traditionally live and in-person, the candidate plays the role of a manager dealing with a poor-performing employee, dissatisfied customer, or other interpersonal situation. The interaction is scored (usually according to competency ratings) live or from a video or audio recording.
• Situational judgment. Usually in the form of multiple-choice questions, these assessments present a real-life work situation and ask the candidate to choose among alternative actions. Sometimes, multiple responses are correct, but each response is worth a different number of points.
• Business case/problem analysis. Some assessment systems require the potential manager to read over an operational or strategic problem and render an opinion or plan. Some include mathematical calculations and necessitate clear writing. Others essentially mimic a work-related intelligence test.
• Formal presentations. Particularly in the case of hiring at the director-level and above, and especially useful for selecting sales executives, the candidate is required to make a presentation about a product, strategy, or current issue. Aspects of the talk, including organization, clarity, and technical knowledge, are rated by assessors.
• Testing. There are several standard assessments that seem to be commonly used to provide insight into the personality and cognitive ability of managerial candidates. In many cases, the activities above get at the same basic constructs, but we have found that the higher the job, the more common it is to find testing as part of the assessment-center experience.

For higher-level jobs, the above ingredients are often combined into a one-half to three-day offsite often called an “assessment center.” Assessment centers are very expensive due to the need to have trained, qualified personnel to execute and score the exercises.

So, as the level of job gets lower, there is a decrease in the complexity of the assessment process.

The key factor uniting all of these diverse tools into an assessment center methodology is that they attempt to simulate managerial tasks and decisions in a real work environment.

Essentially a “test,” these highly regarded instruments ask a candidate to think and behave like a competent boss, instead of simply demonstrating a general trait (e.g., conscientiousness), skill (e.g., Intermediate Java), or ability (e.g., logical reasoning).

Moreover, these tools force the hand of more reliable, standardized data in a selection or promotional process, removing a worthwhile amount of subjectivity from key personnel decisions. There is even scientific evidence of decreased subjective bias on the part of hiring managers when compared with unstructured, ad-hoc interviews.

Another factor that underlies the use of managerial simulations is that they require some budget, effort, and dedication on the part of the HR and line organization, as does any worthwhile human capital initiative. Traditionally, these are expensive, somewhat cumbersome tools that are brought into a company by individuals who are just learning how to use them.

But simulations that realistically judge a candidate’s managerial prowess soon prove themselves to be invaluable to leaders making placement decisions.

Traditional managerial assessments have also demonstrated greater hiring accuracy. They’re more appealing to job candidates than personality inventories and job applications. They have a real relationship with future performance on the job when compared to other hiring tools. They are usually in the form of a statistical correlation with a measure of performance as a boss.

Decades of research in industrial/organizational psychology stands firmly behind these tools.

The Present

So what is the state of managerial hiring tools today? We contacted a wide range of assessment vendors to chat with them about the tools they offer for the selection of managerial to executive-level jobs.

Today’s market for these tools provides a continuum of options ranging from the old school to some thought-provoking new ideas. Here is a summary:

• Most assessment vendors and consultants understand that using the same content for lower-level jobs is not an acceptable strategy. Testing and measuring simple competencies does not account for the differences in what it takes to be a good entry-level worker and what it takes to be a leader/manager. We have seen an increasing number of vendors offering new products that leverage traditional approaches to making selection decisions for more complex jobs (i.e., assessment centers, in-baskets, etc.).
• Competency models, once thought of as a temporary fad, are now well-entrenched and help support linkages between employee selection and development. While competencies are all the rage for employee selection of all types, the link between selection and development tools for managerial-level and executive jobs is much stronger than it is for hourly jobs. Most managerial-level selection tools can also be used for development (with scores indicating strengths and areas to work on), and almost every provider offers a version tuned for both. This makes sense, as this kind of assessment is commonly used for identifying and grooming high-potential employees and for making internal promotional decisions.
• The in-basket is alive and well and living on the Internet. Most vendors offer an online in-basket that provides “assesses” with a realistic representation of a range of daily tasks required of someone at the appropriate job level. A simulation of an email in-box, coupled with organizational charts, employee histories, and process descriptions is most typical here, replacing the giant stack of memos with an MS Outlook-style in-box full of action items and hyperlinks supplemental information.
• Technology is being used mostly for delivery and administrative functions. There are many folks out there who are using the same basic content and format as has been used for decades. The biggest difference is that technology has been used to make the delivery of the content easier and to help make the tasks of scoring and report writing simpler as well.
• Online roleplay simulations are slowly starting to appear. We did see a decent number of products that provide the ability to use avatars and present assessees with more complex data and problems as part of the assessment experience. However, these are still well in the minority. Some of the technologies that will provide the true next generation of simulations do not yet exist. Forward progress in this area will be slow over the next few years.
• The more complex the job level, the higher the level of touch. Most of the standardized products we saw for the lower echelons of management use automatic scoring built into the product, whereas products for more complex and higher-level jobs use trained assessors to do scoring and feedback and act as role players.
• There is very little ability to customize content unless you have deep pockets. Most of the tools we saw did not easily support high levels of customization, though some firms sell versions of tools that differ by setting (e.g., healthcare, manufacturing, general office). However, there are different types of assessments for different types of jobs, and the tasks and competencies we saw are general for most leadership situations. As with assessment for individual contributors, you should analyze the job and carefully match assessment content to the actual work performed.
• There has been very little work done investigating the ROI of the online versions of traditional assessments. We still believe that asking someone to perform key aspects of a job is still the best way to evaluate how well they will do in their new role. Anecdotally, there are thousands who will testify to the impact of a good assessment program for selection and development of managers and executives.

The Future

So what does the future of managerial-level assessments hold? Here are a few ideas:

• More realism via simulations, particularly using avatars and online representation of complex, real-work environments.
• More artificial intelligence will be used in scoring, with technology that partially analyzes a written or spoken response by a candidate and, at the very least, speeds up the scoring and rating of completed materials.
• More data to support the value of managerial assessments.
• More diverse product offerings available as uptake increases in this hot area.
• Tighter ties with competency models will be created so that assessment and development will be linked more closely.

Conclusion

Today, best-practice managerial assessment involves a combination of the tried-and-true approaches of the past and the innovative media of the present.

For the organization that’s serious about leadership, managerial assessment tools (especially in-baskets and role plays) represent the key to truly separating those who aren’t qualified to influence one’s workforce from those who will aid revenue, retention, and overall performance.

We expect that as technology evolves, so will the automation of administrative- and scoring-related tasks and the level of realism associated with these assessments.

Article and research by Charles Handler and Mark C. Healy

For the last five years, Rocket-Hire has surveyed talent-management professionals who use Web-based screening and assessment products to enhance and quantify their hiring processes. Our goal for this research is to document trends in the usage of online screening and assessment tools in order to help provide a clear understanding of the popularity of these tools and their perceived positive and negative attributes.

Late last year, we again asked recruiters, HR staff, entrepreneurs, and hiring managers to tell us about their use and opinions of typical hiring instruments. This article highlights our major findings.

Today’s report is just a brief summary of the results of this study. For more data and an in-depth analysis of these trends, check out your May issue of the print publication, the Journal of Corporate Recruiting Leadership.

Overall, our findings from this year, and the trends that emerge over the five-year life span of this survey, continue to be positive. We had the largest number of respondents yet this year (141 recruitment and hiring professionals completed the survey, just a tad above last year’s 136).

The increase in use of screening and assessment tools and the positive benefits of these efforts continue to notate a strong trend toward their popularity and value.

On the negative side, results clearly indicate a continued lack of proper evaluation for these tools, which is even more discouraging when one considers that lack of understanding of the benefits of screening and assessment is still a primary reason for the failure to adopt them. Thus, the need to demonstrate the value of assessment by including evaluation as a core part of implementation is crystal clear and, in our opinion, remains the No. 1 challenge related to the use of assessment.

Now on to a more detailed review of our results.

Use of Applicant Tracking Systems

As with previous surveys, a majority of respondents employ an applicant tracking system. Specifically, 79% already have or will soon have one deployed, or are considering installation. As can be seen in the table below, no matter what the hiring demand, typical ERE readers are familiar with using an ATS.

Table 1. Use of Applicant Tracking Systems by Hiring Demand

Use of Prescreening Tools

About 57% use online prescreening to sort through their job applicants. We defined prescreening tools as:

Tools that gather information about, or ask candidates to respond to questions about their experience, skills, and qualifications in order to identify if they meet minimum job requirements. These tools are typically used early on in the staffing process.

The table below captures the usage rates of each of the most prevalent forms of screening.

Table 2. Usage rates of common prescreening tools

Common tools in use include the popular qualifications screen, technical skill certifications, and analyses of “fit” with company culture. Every year, it seems that more organizations are using prescreening as part of their online recruiting strategy. But the breadth of this practice varies, with some companies using these tools for all jobs while some use it for one or two alone.

Table 3. Extent of Prescreening Deployment

Use of Assessment Tools

In contrast to screening tools, online assessment tools were defined as:

Scientifically based screening tools that look more deeply into a candidate’s abilities, interests, and skills. These tools include personality measures, cognitive tests (i.e., verbal and quantitative skills), situational judgment tests, job simulations, etc. These tools are typically used for a more in-depth evaluation later on in the staffing process.

Underscoring the growing increase in the popularity of these tools, most common forms of online assessment showed an increase in rate of adoption over previous years, as comparisons with 2002 reveal.

Table 4. Usage Rates of Common Assessment Tools (2007 vs. 2002)

Specific skills and knowledge certifications and personality inventories are common assessment strategies once the pool of one’s qualified candidates has been reduced. Indicators of cultural “fit” and various cognitive ability measures are also widely used.

For another angle on the extent of assessment implementation, we asked respondents to indicate the level of jobs for which assessment is used for evaluating candidates.

Table 5. Job Level of Assessment Implementation by Job Level

2007 appears to be the first year that our data show a general pattern of use across all levels of an organization. Nonetheless, executive-level openings tend to entail more personal and direct methods of selection and placement, and were less likely to involve the use of any online tools.

Effectiveness of Screening and Assessment Tools

This year, 65% of prescreen users and 77% of assessment users felt their tools added value to their organization. In contrast, a full 21% of prescreen users and 10% of assessment users felt these tools did not add value. But this is difficult to judge, especially if one does not take the necessary steps to adequately answer the question. In this case, only 27% of users of prescreening tools and 36% of assessment users collected metrics to judge the quality of their interventions. Moreover, users of metrics tend to report success with prescreening and assessment.

Table 6. Perceived Effectiveness of Prescreening Tools

Table 7. Perceived Effectiveness of Assessment

Clearly, collecting metrics helps to answer questions about effectiveness and, interestingly, provide a positive response to these questions.

The Future of Online Screening and Assessment

Of those companies not currently using screening or assessment instruments, 42% feel they will implement one or more in the future. For them, Table 8 summarizes the sorts of tools under consideration.

Table 8. Screening and Assessment Tools under Consideration

As with the tools current users are seeking to use more of (or expanding to include in their hiring systems), qualifications screening, assessment of cultural fit, skill/knowledge assessments, and personality inventories dominate interest. Still, the mystery surrounding the value provided by these tools, as well as a prevalent intimidation factor, keeps companies from jumping in.

Obstacles to the Use of Online Screening

As usual, we asked users of online hiring technology (as well as non-users), what they perceived to be the biggest obstacle to the adoption of prescreening and assessment in their organization. Again, we received nearly identical responses to previous surveys.

Table 9. Single Biggest Perceived Obstacle to Deployment of Online Tools.

While 18% of users feel there are no obstacles at this time, many hirers feel that one or more roadblocks prevent prescreening and assessment from being integrated into the staffing process. These reasons mostly reflect a lack of knowledge of the benefits of more in-depth hiring techniques. Moreover, there appears to be a general lack of knowledge of safeguards and metrics designed to detect problems when candidate evaluation systems are in place, even among regular users of assessment.

But do these worries represent valid concerns, or is it simply fear of the unknown? Arguably, when assessment tools are properly matched to the demands of the open position, the only relevant concern is applicant identity (e.g., the stereotypical “smart older sister” taking a math test for an applicant) and test content security. But candidate enhancement of qualifications is common on resumes, in interviews, and for some personality traits/scales. Since many of the same concerns exist for paper-based processes, this should not limit any one organization from deploying Web-based hiring technology. Overall, most companies may legitimately cite few or no substantive reasons to limit their use of online recruitment and hiring techniques.

Summary

This year’s results let us know what’s really going on, and it confirmed what most of our previous surveys have found:

• Having an ATS installed is now nothing particularly innovative, especially if you work in a medium or large organization. For some, it’s hard to even imagine the old stacks of resumes and bulging file folders of applicant information.
• Prescreening and assessment are continuing their gradual penetration into the mainstream of recruitment and hiring.
• Qualifications screening, personality inventories, and skill and technical certifications continue to be the most popular online assessment tools. Assessments of cognitive abilities as well as fit with the company culture have expanded their footprint as well.
• Respondents report a lack of understanding, weak budgets, or a general lack of support for online tools as their primary obstacles to adoption or greater use of modern prescreening and assessment technology.
• Those organizations that formally evaluate their hiring practices tend to support the use of prescreening and assessment, but a large number of organizations are still failing to evaluate the effectiveness of their screening and assessment tools.

There clearly is a growing interest in scientifically derived hiring tools. This trend is encouraging but not particularly surprising; these numbers will continue to grow as more organizations understand the value in properly evaluating the impact of screening and assessment tools, seeing for themselves how quality hiring tools can improve a workforce.

As both authors have seen over a combined 25 years in screening and assessment consulting, skepticism about the value of these tools remains strong. We will continue to champion the use of evaluation to “close the loop” and reveal how more in-depth evaluation of candidates may positively impact the bottom line.

A word about our methodology: Respondents were evenly representative of recruiters, recruiting leaders, HR executives, business owners, and hiring managers, and represented a wide variety of organizations and hiring situations. Specifically, 59% were employed by organizations with 500 employees or more, with 34% working for a company with 5,000 or more employees. In contrast, about 20% were involved with recruiting for companies with 50 or fewer employees. In terms of employee acquisition activities, 34% hired more than 500 workers in a given year, but a majority of respondents (66%) made 500 or fewer hires per year.

Every year, I take a few minutes to reflect on the things I have noticed while working with both producers and consumers of screening and assessment tools. Overall, I am very encouraged by what I have been seeing. The market for screening and assessment tools continues to grow. This makes me extremely happy because we I/O psychologists know the value that is to be had via the use of quality assessment tools.

The science geek in me is also very happy to see strong investment in innovation. I am really pleased to see the ways in which quality content is being combined with technology to collect the mountains of data that are required to uncover underlying truths about the relationship between human traits and job performance.

One of the most valuable means of insight into assessment trends over the past five years has been the annual ERE/Rocket-Hire Screening and Assessment Usage Survey. If you have not taken the time to complete this survey yet, please consider taking five minutes to help us out. Your input really does make a difference, and we greatly appreciate your time. A link to the survey is provided at the end of this article.

The major theme for 2008 will be increased integration of assessment into carefully designed products. Much of the rationale behind these products is the reduction of the work required to implement assessment via a combination of data and technology that results in a turnkey product that is both relevant and easy to use. This trend is going to continue to increase the accessibility of assessment and lower barriers to entry that have long been limiting the use of assessments. A quick look at the trends outlined below clearly demonstrates support for this point. In 2008, I believe we will see the following trends continue to unfold:

1. Continued Interest and Use in Assessment. The market for assessment tools will continue to increase as it has every year for the past decade. This will be the result of a number of factors, many of which are discussed in more detail below. In my mind, however, the number one reason for increased adoption is still the fact that properly used screening and assessment is one of the most important ways to increase the quality of hiring decisions.
2. More Vendors to Choose From. As interest increases, so shall the number of options available to consumers. For the most part, this is a good thing, as the bulk of new companies and offerings are being created based on solid best practices. As always, there will continue to be less sophisticated or technically-sound options available. With so many options to choose from, extracting the most value out of assessment tools will continue to require a solid game plan and due diligence as the foundation of the vendor selection process.
3. Increase in Acquisitions for Content. As companies continue to work toward developing products and increasing use of assessment tools, many are discovering that it is easier to purchase content that has a strong history behind it than it is to create their own. This has also been playing out as existing assessment companies continue to purchase smaller companies in order to increase their product range. Recruitment Process Outsourcing (RPO) is also a huge driver of this trend. RPO will continue to be a major force in the future of hiring, and assessment should end up as a key part of the better RPO offerings.
4. Increased Integration and Packaging of Assessment into Off-the-Shelf Products. Directly related to the trends mentioned above, there has been a continued push towards the development of product suites that provide a hiring process into which assessment is embedded. The goal is to create a more turnkey product that can be lightly customized and quickly implemented. The more quality assessment that is baked into a well thought-out process that can be implemented relatively easily, the more quickly assessment can begin to return value. This equation also lessens the need for professional services and makes the product a bit more “foolproof.” The more turnkey the product, the less assurance one has that it is “dialed in” to a particular position or job. The decision to use a more turnkey product as opposed to a more customized approach should be treated as a business decision driven by the contingencies of one’s particular situation.
5. Increased Attention to Products/Sales into Vertical Markets. Again, as the trends noted above unfold, we are seeing an effort to package assessment products that have been developed specifically for certain vertical markets. This is driven partly by the fact that vendors are finding that it makes sense to promote and sell products in a more targeted manner. Health care is one of the hottest examples of this trend, but I have also noticed increases in product creation and promotion in manufacturing and financial services as well.
6. More Products Designed for the Middle Market. The labor-intensive nature of assessment tools has not traditionally lent itself to use by smaller or mid-sized companies. An increasing number of vendors are creating turnkey products designed to overcome the need for professional services and large-scale data collection exercises. As interest in assessment continues to grow, these products will be well positioned to provide value for a wider range of companies.
7. Movement into Job-Search Process. An increasing number of job boards/career portals are discovering that adding assessment can provide much better search/match functions than do traditional methods, such as keyword search. I have been pushing this idea for the past five years, but this year will provide a noticeable increase in the use of assessment tools as part of the DNA of many job boards.

While the above represent what I feel to be very positive trends, I also see the perpetuation of a few trends that I am not so wild about. These things include:

8. Continued Fragmentation. Most users of assessment seem to still take a test-focused viewpoint in which the emphasis is on the use of one specific test rather than on the result of a coherent process. While some situations warrant this type of approach, it is best to build a process in which assessment data is tightly integrated into other decision-making data. The good news is that all of the trends outlined above represent movement in the direction of less fragmentation.
9. Tactical Focus. Assessment is still being used in a highly tactical manner. It is most often considered as a way to fight fires, as opposed to a preventative measure. As consumers experience more success with the use of assessment tools, a more strategic focus will result. It will be some time before we see this type of usage occur, but we are headed in the right direction.
10. Continued Lack of Proper Evaluation. My head hurts from repeatedly banging it against the wall over this issue. Yes, it is hard to create the buy-in needed to properly evaluate the impact of assessment tools. However, without proper evaluation, it is very difficult to clearly demonstrate ROI and build a solid business case for the use of assessment tools. While vendors are trying to help design products that can assist in this area, proper evaluation requires commitment from the consumer. The consumer is the one who needs to collect the data relating to the business impact of assessment. While this is not an easy task, if we can put a man on the moon, we should be able to collect relevant performance data.

This promises to be a great year for screening and assessment. Continued interest by consumers has created a buzz amongst vendors who are, in turn, creating products designed to eliminate the headaches often associated with using assessment. Of course, there are always tradeoffs. The more turnkey a product is, the less it will be optimized for a particular local situation. The degree of customization required is a business decision that should be made based on a thorough needs and cost/benefit analysis. Some things never change!

Want to help us learn more about what is going on with the use of screening and assessment? Please take five minutes to complete our survey. We will be sharing the results with the ERE community this spring.

We I/O psychologists spend a lot of time talking to people about what they do, and we try to identify the human traits and experiences it takes to perform their jobs. We then use this information to develop components of the hiring process that will allow organizations to make predictions about which applicants will make successful hires. A big part of a successful hire often centers around hiring someone who will not leave the job in a manner that will cause a negative economic impact on his or her employer.

Of course, the good news is that in many situations, even a slight increase in the accuracy of predictions made during the hiring process can lead to huge levels of ROI. In my world, one of the most common reasons for using assessment and best practices is to reduce turnover primarily for this very reason. It takes effort to find, evaluate, hire, and train someone. All this takes two precious commodities: time and money. So, helping predict who will stay in a job long enough to offset these costs (at a minimum) and provide even more value through delivering good performance of job tasks, etc., is totally worth the investment.

Separation from a job (aka turnover) is one of the most frustrating but fascinating parts of the work I do. At the end of the day, even when we take great care to follow best practices and use good science, there is absolutely no way we can predict things like performance and tenure with high levels of accuracy. This is because we are dealing with humans, and each of us is as unique as a snowflake. Not only that, but each of us also brings to the table links to other people (such as family and friends) and a context that is broader and often more important than that of the workplace. This makes predicting anything about what humans will do a difficult endeavor.

The stuff I work with ultimately takes place in, and impacts, the real world. Because of this, I try really hard to pay attention to my daily experiences as a way to gain insight into the reality of what people face in their jobs. For me, these experiences help me put the things I work on for my clients into perspective and provide some really interesting nuggets that help me feel more in tune with the reality of people and their jobs, especially when it comes to a subject as interesting as turnover.

To illustrate this, I want to share an exchange I overheard between two cashiers while shopping at a major retail chain the other day, and to use this story to make a few points about the many variables that can make predicting things like turnover so hard. (Please keep in mind that the context here is retail associate-level jobs and, as such, this article applies mostly directly to this stratum of the workforce. However, the general ideas apply to almost any situation.)

The exchange went something like this:

Cashier 1 (African-American woman who appeared to be in her mid-20s): “How long have you been working here?”

Cashier 2 (Caucasian male who looked to be in his early 40s): “Just about five days. I was selling real estate before this, but the market is so soft right now that I needed to take another job to help save some money. I want to go back to school to become an appraiser, so I took this job while I save up. What about you? How long have you been working here?”

Cashier 1: “Only a few weeks. I was a barista for five years before this. I got sick of it though, so I decided to try something else. I really want to use my degree in African-American studies, but I can’t really find any work that will allow me to use it how I want. I am just going to work here until I can find the right opportunity to use it.”

Cashier 2: “Oh, that’s really interesting. I have been doing some work with the local cultural museum, and I know they are always looking for people to help out over there. You should check it out and see.”

Cashier 1: “Cool, that’s a good idea. I’ll be sure to check it out. Maybe I can quit this job and actually put my degree to work for me.”

I made my exit at this point, but I couldn’t help but stop to reflect on the meaning of that brief exchange in the context of what I do. My reflections included:

• Even though job duties are standardized and some work environments rely on workers having a core set of traits to carry out these duties well, everyone is still very different and comes to the table with a totally different history and context. This context often has more of an impact on tenure than anything else because it represents a reality that is highly personal and often transcends things like money and time.
• There are a lot of folks out there who may be seen as overqualified for some jobs but whose life-reality places them in situations where they end up needing the job to make ends meet. This can have both positive and negative impacts depending on the person, the environment, or even things such as the person’s relationship with management.
• The staggering number of options for doing something with oneself and the reality of the opportunities provided to those of us living in the United States often compels people of all ages and races (not just Gen Y or whatever other “Gen”) to look at jobs as temporary.

The above represent both good and bad things for employers. Individuality and personal context can have a huge impact on organizational performance that can be either positive or negative depending on how they are understood and managed.

The exchange between the two cashiers led me to the following thoughts, specifically regarding the realities of our current workforce, predictive hiring, and turnover:

• In many cases, hiring people for their raw abilities rather than for their experience or technical skills can be a good strategy, especially if one is having trouble getting butts in seats. Someone who is passionate about customer service and can relate to people can probably do well at jobs that require these things, even if they have no experience. This is one of the biggest value propositions for personality and work-values assessments.
• It is important to be honest about opportunities and what they offer to a job applicant. The reverse also applies; applicants should be encouraged to be honest about their goals for the job. Anything less is going to accelerate things like turnover.
• Organizations need to understand the demographic from which they hire. It is important to understand trends in how job seekers view employers and to do one’s best to adapt hiring practices to account for these things. This also applies to sourcing and even employment branding.
• Understanding of context should drive creativity. If the cold, hard reality is that a worker isn’t expected to stay at the same job long enough to make the hire pay off, seek creative ways to change this equation. This may include offering flex time, telecommuting, or benefits that are of value to the employee.
• Hire and/or train management to be understanding of the context that each worker brings to the table so that they can relate to that person and understand the value that his or her unique circumstances bring to the table.
• And, most importantly, tracking the relationship between data collected in the hiring process and objective, measurable outcomes is absolutely critical for understanding the economic impact of the reality that is our workforce. From this information springs the power to justify processes or to understand the need to adapt and change them.

So, the next time you are bored in the checkout line, do a little eavesdropping and stop to contemplate the bigger picture.

No facet of assessment has experienced as much change over the past decade as has the reporting of candidates’ results. Those of us who worked with assessment back in the dark ages can attest to the frustration experienced with the reporting of assessment results.

In those days the following were the norm:

• One would simply overlay a scoring key, enter in numbers into a worksheet, make some calculations, and arrive at a score for each applicant and perhaps a percentile rank based on an ancient set of norms.
• In some cases one could fax off an answer sheet and receive back a narrative report summarizing an individual’s performance.
• Examining aggregate data for more than one applicant required creating an Excel file or similar table by hand. These often had embedded formulas to help provide the information needed to make decisions about an applicant relative to other applicants and/or some set of job standards.

The commonality in the above activities was that no matter how it was done, reporting assessment results was a time-consuming process that yielded little insight about an applicant beyond a set of simple parameters that required users to do some extrapolating.

If one wanted to compare applicants head to head, creating a system for doing so was relatively complex and often yielded results that did not always inspire confidence in one’s decisions. Collecting aggregate data that spanned multiple steps in the hiring process to facilitate decision-making was pretty much out of the question.

Then again, back then someone in Peoria couldn’t instantly access home movies shot by someone in Mongolia either. Luckily for all of us, times have changed.

Reporting has become one of the strongest points of the assessment process and now provides users of with a set of powerful tools that add a tremendous amount of value to everyone involved in the hiring process. How does reporting add value? A quick look at the standards for modern assessment reporting broken down into several key areas should provide a clear answer to this question.

Area 1: Candidate Management Functionality

Those readers familiar with using an applicant tracking system have already had experience with candidate management. By candidate management I mean the basic information required to manage data from applicants for multiple requisitions. This may include a range of things such as assigning applicants to a job opening, sending out links to allow candidates to take an assessment, tracking those applicants through the stages of the application process, communicating with applicants, and pushing applicant information out to others involved in the hiring process.

The level of candidate management functionality provided varies quite a bit across assessment vendors. As the ATS and assessment worlds have continued to converge, the level of candidate management functionality has increased. This is especially true in products that have been designed for the mid-market where consumers may purchase one system that is intended to cover all the bases.

Most of the more evolved assessment providers offer some level of candidate management, and even the smaller ones are now starting to catch up in this area. This functionality is not always needed however, as many enterprise users rely on their ATS for this kind of thing. In these cases it is more common for assessment providers to pipe a few key pieces of data into the ATS system or in some cases to simply use both the ATS and the candidate management system in parallel.

Area 2: Dashboard Reporting

Dashboard reporting is a concept that is much bigger than HR applications. The concept is pretty simple: use a Web-based interface to pull in all of the data that is relevant for accomplishing a certain set of tasks.

Dashboard reporting has been the most significant change in the manner in which assessment is used. To be sure, the line between this type of reporting and candidate management is often blurred as both share some of the same key functionalities. The major difference is that the dashboard provides the ability to access much richer data about the applicant than does the candidate management system.

One of the biggest benefits of the dashboard is the ability to present information in layers that allow the user to focus only on relevant information. Below are the layers that are most commonly provided as part of dashboard reporting:

Level 1: Stack Ranked Comparisons

This usually takes the form of a screen that provides a stack ranked list of all applicants for a given requisition. Within this screen, the following data is usually provided for each applicant:

• Basic info such as name, date applied, contact info.
• Scores on each component of the selection process along with a pass-fail indicator, usually red, yellow, or green.
• An overall pass-fail indicator.
• Basic indicators of score on competencies critical to the job.

Level 2: Individual Competency Feedback and High Level Narrative Feedback

When one clicks on a piece of information on the Level 1 display, they are able to access more detailed information about each candidate. For instance, one may be able to view graphic representations of a candidate’s score for each critical competency associated with the job. This information may also contain some narrative to help decision-makers understand the ramifications of these competency scores in terms of key job requirements.

The goal of this level of information is to provide decision makers with extra data that can help them read in-between the lines when making decisions. This type of data is often displayed graphically using a format that displays an individual’s level of a given trait relative to normative standards or colors that demonstrate where an applicant stands on a certain trait relative to what is required for the job.

Level 3: In-Depth Information

Information at this level is usually accessed from the Level 2 display; however, there are often pathways in Level 1 that allow direct access to this information. This level of information usually contains the following elements:

• Detailed narrative. This involves very detailed narrative information about what assessment results may mean in terms of specific work behaviors. For instance narrative for someone who has displayed high level of teamwork based on assessment results narrative may read “Joe Candidate can be relied upon to defer to the needs of the team over his own personal wishes.” This type of narrative can get pretty in-depth as there are many positive and negative behaviors and tendencies associated with each of the competencies measured by an assessment process. This information can often get somewhat tedious and is not always going to be correct, since each person is an individual who may act differently in different situations. Over-reliance on this type of information is a common problem for those making hiring decisions.

In my opinion, the other types of level 3 information are of greater utility to decision makers. These include:

• Coaching recommendations. Based on the same type of data as are detailed narratives, coaching recommendations take things one step further by suggesting ways to help deal with the negative side of certain competencies. These type of recommendations are best suited to helping use assessment results for onboarding. I believe this is a very important way to help gain extra value from the assessment process.
• Structured interview questions. The generation of structured interview questions that are based on areas where more information is needed about where a candidate stands on a certain trait has become a market standard for assessment reporting. This type of function is an excellent way to gather more data about areas where assessment results do not provide the level of clarity needed for decision-making.

Area 3: Aggregate Data Reports

This includes any number of different reports that can be generated based on the raw data collected during the application process. The most common example are reports required by the EEOC, OFCCP, and other government agencies.

While many assessment vendors do have the bases covered when it comes to these major, mandatory reports, many also work with clients to create customized reports. These include things such as applicant flow, source of hire, and any other key metrics that are related to the hiring process.

Applicant tracking systems often cover some of these things, but there is an increasing need to use data from all parts of the hiring process, including assessment, to feed business intelligence. In many cases, vendors work with clients to set up customized reporting that will allow them insight into any number of important aspects of the hiring process.

Area 4: Flat Paper Reports

Of course, not every online hiring system actually offers all of the hi-tech reporting summarized in this article. In many cases the high-tech systems offer the ability to print an old-fashioned paper report that contains all of the same data summarized in the system. In many cases this type of report is very useful and is consistent with what hiring personnel have been using for years.

The type of reporting needed depends on a variety of variables. When looking at using an assessment vendor, your reporting needs are a key thing to consider.

Understanding what your hiring personnel need to get the job done as well as the key junctures in your hiring process will help provide some understanding about exactly what type of reporting you will need. Reporting is a usability issue and as such, end users of the reports should have some input when it comes to making decisions about reporting needs.

We I-O psychologist-types tend to be real data hounds. Much of the work we do for our employers/clients involves the use of data to investigate specific hypotheses in order to illuminate the underlying truth in a situation. The outcome of this work often has tremendous value to organizations because it provides them with hard data on which strategic decisions can be based. Additionally, the collection and analyses of data often helps us to identify new trends that we haven’t yet thought about.

Many of you who follow my articles know that I have a keen interest in the pre-employment assessment industry, and write quite a bit about its trends and happenings within. My interest in data and trends has led to an annual online screening and assessment usage survey.

The idea for this survey was born back in 2002, when I became frustrated over the lack of available information about the usage of pre-employment screening and assessment tools. This lack of information has been a challenge because though everyone seems to be saying that screening is becoming a hot area, there’s little actual data available to confirm this statement or to tell us how hot it really is. This lack of information also makes it hard for those of us who follow this industry closely to provide factual information about how companies are using online screening and assessment tools, and what the results of this usage have been.

At the end of this article, you’ll find a link to this year’s survey. Take a few minutes to help other members of our community by providing information about your company’s screening and assessment practices. The more data that’s collected the clearer existing and emerging trends will become. Last year, we had a record number of responses, a fact that seems to indicate the increased interest level in screening and assessment. Given the steady increase in interest and the lack of information about this industry, we feel the results will continue to have value for the ERE community. We look forward to reporting our findings right here on ERE sometime this coming spring.

In order to provide some extra motivation, here’s a quick summary of the trends identified in last year’s results (for a more in-depth look, please see last year’s results).

One hundred and thirty-six professionals completed the survey, representing an increase of 51% over the previous year. Respondents represented a wide range of company sizes.

Summary of Findings

• Most medium-sized and large organizations have adopted or will adopt an applicant tracking system, but the use of an applicant tracking system does not relate to perceptions of hiring effectiveness.
• Some hiring tools, like qualifications screening, skills testing/certification, and personality tests, are becoming standard features of a strategic hiring process. The use of some methods (e.g., online interviewing) is rare.
• Few companies formally assess their prescreening tools or use metrics of any kind as a decision-making aid. Those who do tend to be aware of the effectiveness of their hiring systems and perceive greater value in them.
• People professionals report a lack of understanding, weak budgets, or a general lack of support for online tools as their primary obstacles to adoption or greater use of modern prescreening and assessment technology.

Overall, the results of our fourth annual survey are the most encouraging yet. These results clearly show a growing interest in the use of assessment tools. Valid, Web-based approaches to recruiting, screening, and placing job candidates are fast becoming the norm at leading-edge, people-focused organizations.

This trend is encouraging and, in our experience, not surprising. We feel these numbers will continue to grow as more organizations understand the value in properly evaluating the impact of screening and assessment tools.

Unfortunately, there is still a very widespread failure to adopt the business intelligence mindset required to evaluate the effectiveness of screening and assessment tools. Primary obstacles include a lack of understanding of the value of these tools, as well as budgeting issues.

As long as this deficiency exists, it is not surprising that skepticism about the value of these tools remains strong. We will continue to champion the use of evaluation to “close the loop” and help ensure an increase in the understanding of how properly implemented screening and assessment tools can impact the bottom line.

This Year’s Survey

The goal of this year’s survey is to continue building on the information gathered from past surveys and to verify the fact that usage rates for online screening and assessment are increasing.

In order to help provide the ability to track changes in usage rates, this year’s survey is essentially the same as last year’s, with a few minor changes to help account for trends that have developed since last year. Specifically, this year we have added additional questions on budgeting and ROI for the use of screening and assessment tools.

While this is hardly a scientific survey, and its length limits the depth of the information that we are able to collect, there is tremendous value in the information your survey responses will provide. For instance:

• This survey will help to provide some reality to speculation regarding increases in the use of both online screening tools and online assessment tools.
• Your survey responses will provide information that is extremely useful for gaining a broader perspective on the evolution of the entire staffing process.
• The results of this survey will help provide companies with some basic benchmark information about how online screening and assessment are being deployed.
• The data from this survey will confirm our speculation that while many companies are using online screening and assessment tools, few are collecting the metrics needed to help them understand the true value they are providing. This is a problem that must be addressed, and we hope that verifying its existence will help to serve as a wake-up call.

All survey responses are confidential and individual responses will not be shared with anyone. By participating, you will help provide yourself and other members of the ERE community with up-to-date, accurate information about trends in the use of online screening and assessment tools.

We are interested in collecting information from anyone who is involved in the staffing process for his or her organization. This includes recruiters, hiring managers, staffing and HR executives, consultants, etc. We welcome participants from companies of all locations, sizes, and industries. We are not able to use information from individual vendors of screening and assessment tools or persons who may consult to multiple organizations.

Just click on “take the survey,” and you’ll be taken to the survey page. This page has all the information you will need to complete the survey. Please feel free to forward the survey to anyone you feel might be interested in participating. If you have any questions, just contact me at info@rocket-hire.com.

The results of the survey will be featured in an ERE article sometime this coming spring. We look forward to sharing our results with you.

One of the most difficult aspects of using assessment lies in gaining an understanding of the various pricing models associated with it. This understanding is critical when it comes time to budget for an assessment program. Even if your organization has experience using assessment, budgeting can be a very difficult issue.

Budgeting for assessment can be hard because of the following:

• Lack of understanding with assessment and its value proposition.
• Purse strings that are often owned by different groups within HR.
• General inexperience with assessment pricing models.
• Lack of understanding around price differences for off-the-shelf vs. customized assessments.
• The increasing commoditization of assessment via integration into other staffing tools.
• Lack of good benchmarking data.
• Extreme variation in assessment pricing.

The remainder of this article provides some basic information that I hope will shed a bit of light on assessment-related budgetary concerns. What this article does not do is provide specific benchmarking information that will allow you to magically understand how much you should budget for using assessment.

Consumers of assessment understand some general rules about assessment and pricing:

• The more customized the assessment content, the higher the up-front cost. Once up and running with customized assessment, however, the actual costs of testing often will not differ. The cost is in the investment required to create the customized content. Customized content often provides incrementally higher levels of ROI, so the extra investment is often worth it.
• The technology level offered by a vendor does not necessarily impact price. Many companies now offer assessment delivery platforms that are robust and have a variety of candidate-management features built in. These systems can be configured very quickly and easily. However, the more custom-technology-related work required to implement the system (i.e., applicant-tracking-system integration, customized reporting) the higher the price. These jumps can be steep when one starts getting into ATS integrations and the like.
• Just as the assessment market is extremely fragmented, assessment pricing is all over the map and price is not a reliable index of assessment quality.
• Assessment is getting cheaper as it becomes more integrated into the bigger picture as a value-add to staffing processes and the systems used to back them.
• At the same time prices are dropping, quality is actually increasing in many cases, due to the accelerated wisdom that is being gained via the ability to collect and analyze previously unthinkable amounts of data.

With the above in mind, there are three key areas of best practices related to the use of assessment impact pricing models.

Area 1: Setting Up the System

There are two key factors related to set up: doing the work required to understand key job requirements and technology system configuration.

The work needed to verify that the assessment content is job-related and the set up of the technology system are often inexorably linked, such that there is one “set up fee” that includes all the work needed to get the assessment process up and running. For example, using an off-the-shelf test from a vendor with a modern, robust candidate management system is getting to be less and less expensive. Set-up fees for this type of package usually range from $1,500 to $5,000.

As the complexity of the implementation goes up, these fees can often rise sharply. First is the level of technology integration required, with deeper implementation requiring much deeper pockets. Second is the level of due diligence related to understanding key job requirements.

If a full job analysis is required, you may pay north of $10,000, depending on the scope of the initiative. Conversely, using an off-the-shelf tool can be extremely cheap. Just remember that you get what you pay for and that taking the time to do a more in-depth job analysis can have tremendous value because it contributes to legal defensibility and also serves as a critical foundation for a variety of other HR activities.

Area 2: Assessing Applicants

This area involves charging for the test content or for the use of a system that collects a variety of different types of predictive data from applicants. There are a variety of different models related to test/system usage. Understanding them is critical when working on setting assessment budget.

The most common models include:

• Traditional-”test-centric” model. This is a flat per-head charge for any assessment used. Prices almost always follow a volume model such that the more units used, the lower the price. There is a huge range in per-test pricing with the cheapest tests now running at about $5 per head all the way to over $150 per head. The main variables are the number of tests and job level for which the assessment will be used. For instance, a test for entry-level retail workers will be much less expensive than a test for marketing managers. There is not always a direct correlation between price of an assessment and its quality, so be careful when considering this type of pricing.
• “Per-recommended” model. Because companies with more evolved technology systems often offer a multiple hurdle approach that involves the collection of data at various stages in the hiring process, many customers are wary of paying per click when there is high applicant volume and many applicants are screened out at the first step in the process. This has led many companies to create a variation of the per-head model in which charges occur only when candidates clear the first hurdle or for those who are judged as acceptable.
• Per-seat-license model. Some companies price their assessment based on the number of employees. This model is most common among vendors that sell packaged software systems and is often calculated to provide a price point for yearly use of the system
• Flat-fee-license model. Some companies offer a flat-fee model, an agreed-upon price for unlimited applicant evaluation. In many cases this model includes a threshold value such that over usage causes some additional fees.
• Pay-for-performance model. Some companies will offer a variation of any of the above models in which there is a service level agreement put into place regarding the level at which the use of assessment will return a return on investment. This model is rare but it is an excellent way to hold vendors accountable. It also serves as a quality filter when choosing assessment vendors because it takes confidence in one’s product and process to offer this kind of model.
• Combo model. It is common to encounter some combination of the above models. This is especially true as assessment becomes more tightly integrated into different types of human capital management systems and processes.

Area 3: Evaluating Assessment Accuracy/ROI

This involves the work done to “close the loop” and evaluate the effectiveness of the assessment used. Unfortunately, this area is often not a concern for many organizations because they do not take the time to do this type of work.

This may be one of the biggest mistakes when it comes to budgeting, because evaluating assessment ROI is the only way to demonstrate that the use of assessment is paying for itself.

Such demonstrations are critical for understanding budgeting and for providing the ability to obtain more budget for the continued usage of assessment.

There are two types of charges that relate to evaluating assessment:

• Configuring reporting. These costs can vary quite a bit, but most technology platforms have accounted for most of the major types of reports that one would need. This means that reporting costs are often not a big deal.
• Validation. This usually requires professional services to set up data collection, analyze data, and report findings. One can expect to pay between $10,000 and $25,000 or more for this work, depending on the depth of evaluation work required. While this may seem expensive, remember that validation pulls double duty because it provides legal defensibility and proof of ROI. Further, the increasing integration of technology into hiring is leading to a business intelligence mindset that is causing feedback loops to become increasingly automated and thus allowing for savings when it comes to professional services.

Conclusion

In general, the trend in assessment is moving toward commoditization. Prices are dropping because assessment is getting easier to use off the shelf and is becoming more tightly integrated into other staffing-related products such as job boards and applicant tracking systems. This is good news for many folks because it also supports availability of quality assessment to the middle market, where it was not previously affordable.

In days of yore, assessment came in two basic flavors: direct off-the-shelf or highly customized. In both cases, the revenue was related to the professional services required to ensure the assessment was properly configured, created, and evaluated.

In most cases, each local assessment implementation was customized and was costly due to the sheer volume of work that was required to set up, maintain, and evaluate the assessment process. The good news is that technology has allowed us to move toward models that involve fewer professional services while allowing for increasingly higher quality assessment content.

While reduction in professional services can be a good thing, just remember that you get what you pay for and that using a system directly off the shelf is likely to yield less-accurate results than creating an optimized assessment.

One of the other new trends that helps to bridge the gap between off the shelf and customized assessment is what I call “light customization.” This involves the use of standard “blocks” of content that can be stacked to help optimize the relevance of assessment without requiring the development of fully customized content.

I feel that this offers the best compromise between optimal configuration and quick implementation. This type of system is usually much cheaper than the old model for assessment while often delivering almost equal results.

The final answer of how much one should budget for the use of assessment is, “it depends.” I wish I could provide some hard-and-fast guidelines, but costs for assessments depend on a multitude of factors including how much customization you need; the level of technology required; the level of job you are assessing for; the applicant volume you plan to experience; and the level of follow-up evaluation you plan to use.

Do not skip out on the proper evaluation of assessment, even if it does add extra cost. Evaluation is critical because this is a central part of understanding ROI and provides the data needed to build a business case for the use of assessment tools.

Once you have implemented and evaluated an assessment process, you will have a much better starting point for the development of next year’s budget. Live and learn.