Let’s begin with the assertion that having the right job skills is at the top of the food chain.

There is nothing more dangerous to the bottom line than employees not being able to perform a job. In fact, without the mental horsepower, organization skills, and interpersonal skills to perform a specific job, an employee is a potential train-wreck (although a happy one, if that counts for anything) … and, yes, managers are considered employees.

The Manager

Research shows the greatest source of job satisfaction or dissatisfaction is a person’s manager. I’d like to pretend I had no troubles with managers, but in my own career, I have gone overnight from being a valued employee to receiving a one-way train-ticket out of Dodge: same person, same skills, same organization, but different manager. I have also moved in the opposite direction. The minute my manager changed, my train ticket was canceled and I was welcomed back into the club. Again: same person, same skills, same organization, but different manager.

I have also worked in and with different corporate cultures characterized by innovation and continuous improvement; where loyalty was valued more than ability; that resembled an institution for the emotionally dysfunctional; where time stood still; and, where feelings of support people were valued more than the product of the professionals. Yes, manager fit and cultural fit are alive and well affecting human performance everywhere.

One Man’s Paradise is Another’s Hell

Ben Schneider, chair of the I/O program at the University of Maryland, has written extensively about what happens when personal culture clashes with organizational culture. He calls it ASA — an acronym for attraction, selection, and attrition.

It sounds more complicated than it is. Applicants are attracted (A) to organizations based on their cultural reputation; organizations select (S) employees who seem to “fit” their culture; and, employees who don’t fit leave through either voluntary or involuntary attrition (A). ASA forces are like a corporate iceberg. It has enormous inertia to resist any change.

There are a variety of recognizable cultures. For example, innovative vs. traditional; interdependent team vs. individual; cooperative vs. competitive; arrogant vs. self-effacing; autonomous vs. controlling; and, trusting vs. defensive, just to name a few. Of course, these cultures also come in all combinations and permutations.

Making ASA Work for You

As I mentioned earlier, culture is a powerful force that is slow to change. Sometimes a change in executive management will have an effect; otherwise, changing culture is often like putting a frog in a pan: too hot for comfort and the frog immediately jumps out; but, slowly raise the temperature, and the frog adapts. (No actual frogs were harmed in this example.)

So, how do we match employees to a culture?

First, we have to recognize the three major forces at work in organizational culture: 1) direct manager, 2) job-related, and, 3) organizational. Because ASA works in occupations as well as organizations, we can generally assume job-specific managers and current job-specific employees already fit the culture or they would have left through attrition (of course there are always a few exceptions). This means the first step is making sure applicant first fits the motivational (aka cultural) requirements of his or her job.

After we know the motivational expectations of the job, we have to “overlay” the culture of the organization. For the most part, this means identifying factors that touch all positions. Most commonly, these include a preference for innovation, working in teams, being competitive, and not being narcissistic. You might think of the first three factors as being bright-side (observable, positive) and the last as dark-side (hidden, dysfunctional).

What do I mean when I say dark side? Bob Hogan, a preeminent researcher in the personality/motivation field (and also a subject-matter expert on my long-ago dissertation committee), has shown that bright side factors lead to upward career mobility, but dark side factors tend to emerge when employees gain position-power in the organization. Dark-side narcissists are usually extremely charismatic; however, inside they harbor deep-seated feelings of superiority and entitlement. One only needs to think of the many public and political figures making the news by shamelessly taking advantage of other people to further their own egotistical objectives.

The Process

Starting with a known performance framework, a professional test developer interviews people from the organization using that information to build a survey that includes both occupationally specific and organizational-specific factors. The developer gives the survey to a few hundred people. Their answers allow the developer to determine things like inter-item reliability and construct validity (this data enables the developer to make deletions and edits to ensure a robust test). If all things go as planned, the next step includes either a concurrent or predictive validity study.

Validity studies confirm the test actually measures what it was designed to measure. They involve one group of people taking the survey and another group of people rating them. A concurrent study uses employees already on the job. It is quicker, but since employees are generally alike (i.e., survived the ASA thingy), it is harder to find differences between them. A predictive study gives better results because new employees are more diverse than seasoned ones; but, it takes more time.

Conclusion

The fit between organization, manager, and employee is more complicated than most people imagine. Broad-scope, one-size-fits-all surveys tend to ignore critical job-fit factors. Good fit involves understanding the momentum and inertia of attraction, selection, and attrition and separating them into factors that affect both the employee and the organization. Done right, this kind of survey ensures getting the right people into jobs they will enjoy. Caution should be taken, however, to remember that culture usually tells us very little about job skills. Maximum performance requires measuring both.

The article cites a lady who specializes in what she calls energy profiling. She claims she or one of her licensees can examine your photograph to determine with perfect accuracy (her words) your personality type. Amazing! And to think all those psychologists who worked their way through graduate school, suffered peer-reviewed research, and spent tons of money pursuing advanced degrees for the last 100 years could have just looked at your photograph! Go figure.

I searched, but aside from watching an engaging streaming video taken in front of some very picturesque mountains, I found little proof that she was qualified to produce legitimate hiring tools. Her PR firm did claim she revolutionized the fashion and beauty industries by sharing her simple beauty/fashion assessments with women around the world; helped women align their physical features in perfect harmony with their clothing, jewelry, hair color and style; and provided pioneering insights on weight, sex & intimacy/relationships, depression, self-esteem, parenting, finances, physical health, and spiritual health. Wow. After all that, I guess hiring was the only field left to master.

I don’t know about you, but I like to see a writer have professional certifications or special education that would convince me they actually knew what they were talking about. You know, the same way we would expect a medical correspondent to actually have practiced medicine, a legal expert to graduate from an accredited law school, or an engineer to a have a legitimate engineering degree. But that’s just me.

She presents, as proof of her work, a collection of streaming video segments and personal testimonials from people claiming her system changed their lives for the better. Sorry, folks, this kind of “proof” is nothing more than personal opinion. If you want to know whether something is fact, you have to produce facts to support your opinion. Unbridled enthusiasm unsupported with expert knowledge is a dangerous thing.

I’m sure she is sincere about what she does. No one would make such wild claims unless they were. Unfortunately, using a photograph system to type people and predict job skills is a shining example of pure nonsense.

Let’s list a few facts prepared by the DOL, published in 1978.

As I claimed before, if you want to see an example of a rare event when the government got it right, read through the 1978 Uniform Guidelines on Employee Selection Procedures.

Clearly Define What You Want to Measure

The Guidelines suggest it’s a good idea to conduct a professional job analysis before starting any hiring project. A professional job analysis includes talking with employees to learn what they do, talking with managers to learn what’s important, and talking with people who know if the job will change in the future. Since these folks cannot be expected to know about testing, the analyst converts their information into measurable competencies and verifies it with a wide range of job content experts.

You know this step has been missed when people in the hiring chain argue among themselves or complain the recruiter keeps sending them the wrong people. A good job analysis reduces job confusion. BTW… I’ve never yet seen a professional analyst break down a job into this lady’s purported energy types. We must have all missed that class.

Step Two

Now that we know what to measure, it’s time to hit the books to choose trustworthy measurement tools. In some cases, this will be structured interview questions, pencil and paper tests, job simulations, realistic job previews, case studies, planning exercises, technical knowledge tests, and so forth. The important thing to remember is that any process used to separate qualified from unqualified applicants, even if it is a yardstick, is a test. I cannot repeat this enough: interviews, resume screens, application blanks, and even specialized recruiting sources are tests!

It would be nice to know the tests you used were accurate.

The DOL says you cannot rely on validity claims made by vendors, marketing literature, third party statements, or any other source. These claims probably have nothing to do with your job. Can you use a test developed for bankers to hire your banker? Only if you can show the two jobs are highly similar. That’s a good thing. Why spend tens of thousands of dollars based on false assurances?

Conclusion

Read up on the Guidelines. I’m sure this won’t be the last time someone will naively try to expand market share. But making a hiring decision based on a person’s photograph is not only bad science; it is completely irresponsible behavior

I can just see the future. “Position open for individual with a well developed root chakra, median energy navel chakra, and a mature third-eye chakra. Candidates with an overactive sacral chakra or an undeveloped heart chakra need not apply. Mature crown chakra’s always welcome.”

Does HR need any more trouble with gaining credibility?

Validation is a best practice that can provide both critical information about the ROI of an assessment and the documentation required to support its legal defensibility. Unfortunately, proper validation is not the norm when it comes to the use of assessments. While many companies make use of assessments that have been validated in the past or that do satisfy some of the requirements for test validity, conducting the validation work required to fully satisfy best practices and gain an understanding or ROI is often not on even on the radar screen.

When it comes to validation, my experience shows that the biggest stumbling block is a lack of understanding of just what validation is and why it is so important. While the concept of validation definitely has its complexities, it can be boiled down to a few simple concepts which are discussed below.

What is Validation?

Webster’s online dictionary defines the word “validity” as:

….well-grounded or justifiable : being at once relevant and meaningful <a valid theory> b : logically correct <a valid argument> <valid inference>

….appropriate to the end in view : effective <every craft has its own valid methods>

These definitions definitely hold true when it comes to employment testing. Ask an I/O psychologist and he or she will tell you that validation simply means the act of establishing two key things: 1) That anything used to make employment decisions is job related, and 2) That the assessment actually measures what it is supposed to measure (i.e., that the test is “accurate”).

There are a variety of ways to document the job-relatedness and accuracy of a test as a decision-making tool; however, a working understanding of validation should focus on two general types of validation, content validation and criterion-related validation.

Content validation: Quite simply, content validation involves the documentation of the personal characteristics (i.e., experience, education, knowledges, skills, abilities, values, etc) required to perform the job.

At a minimum, claiming a selection measure is content “valid” requires an alignment of test content to job requirements so that support for the job-relatedness of the test can be documented. This means that the job or jobs in question must be carefully evaluated and that the input of subject matter experts (incumbents and supervisors) be used to create a full understanding of the various things that are required for successful job performance.

The process used to establish the job-relatedness of test content is known as “job analysis.” Once information about job performance and related characteristics has been documented via job analysis, selection measures can be mapped out to match job requirements. For instance, if job analysis shows that the job requires fast and accurate typing, then the use of a typing test to hire applicants for that job is assumed to be content-valid based on its relation to job performance requirements.

Content validation can satisfy EEOC requirements for claiming a test is valid (provided that proper procedures were followed). However, settling for only content validation is selling yourself way short. The real value proposition when it comes to validation lies in the evaluation of the ROI provided by a selection measure. This information can only be provided by criterion-related validation.

Criterion-related validation: Whenever possible, the statistical evaluation of the relationship between selection measures and valued business outcomes is desirable. This type of validation is known as “criterion-related validation” and it can provide concrete evidence of the accuracy of a test for predicting job performance. Criterion validation involves a statistical study that provides hard evidence of the relationship between scores on pre-employment assessments and valued business outcomes related to job performance. The statistical evidence resulting from this process provides a clear understanding of the ROI provided by the testing process and thus helps document the value provided. Criterion-related validation also provides support for the legal defensibility of an assessment because it clarifies the assessment’s accuracy as a decision-making tool.

While criterion-related validation may seem mysterious, it has much in common with two more well-known concepts that are used to help find value within business processes: six sigma and business intelligence. Both of these methods require that data be examined in order to help clarify relations between various process components. The resulting information can be used to help streamline business processes and uncover meaningful relationships between various streams of data. The creation of a feedback loop using criterion validation is really no different.

The 1-2 Punch

In an ideal world it is best to have both content and criterion validity evidence. Documenting content validity is a minimum requirement for any pre-employment selection measure; however, content validation alone can’t provide any evidence for the ROI associated with a test or selection measure. Adding statistical validation bolsters the legal defensibility of an assessment and provides insight into ROI. Unfortunately, most companies do not perform criterion-related validation.

There are a variety of reasons for this.

• Criterion study not possible: A legitimate reason for not conducting validation studies is that there are simply not enough people in the job to allow for a reliable study to be conducted. The greatest shortcoming of statistical validation is the fact that its results are not credible unless the sample size is well over 100. Therefore small companies and jobs with few incumbents will preclude the ability to gather criterion-related validation evidence.
• Lack of resources: Many companies feel that conducting criterion-related validation studies simply takes too many resources. They are unwilling to spend the money needed or take the time to collect the data required to perform such studies.
• Lack of understanding: Many companies do not understand the concept of criterion-related validation and are not aware of the value it can provide.
• Lack of guidance: Test providers often sell tests as being “valid” because they have been validated for jobs similar to the one in question. While this may be true, it is not always an “out” that allows one to avoid doing criterion-related validation. Many test vendors do not provide advice or services related to criterion validation, leaving the user on their own to figure out this part of the equation.

What does all this mean to staffing professionals?

• At a minimum all measures used to select employees should be content-valid. Failure to document content validity means the legality of your selection measures is compromised.
• Whenever possible, it pays to conduct criterion-related validation to add additional support for content validation efforts and to gain insight into the ROI of selection measures.
• When reviewing tests provided by vendors, make sure to understand that a claim that a test has been “validated” can mean different things. Don’t assume that just because the test has been validated in another setting, that it is automatically valid for your particular situation.
• Lack of proper validation for tests and selection measures can be costly! Citing lack of resources as a reason for not validating is like passing over dollars to pick up pennies.
• Both types of validation can add value. Content validation provides legal piece of mind and criterion validation bolsters this and provides the ROI evidence needed to build a business case for using assessment.
• Educate yourself about validation and the options available to you. For more information talk to an industrial psychologist about validation and legal requirements and consult resources such as the Uniform Guidelines for Employee Selection Procedures.

As with anything else, it may take a bit of extra time and resources to do things right, but the extra effort will provide value and piece of mind.

Then there are sites like UpMo.com that actually try to help a job seeker understand that a job is not a career. The subscription-based service launched earlier this year, but just this week added a job search engine that promises to filter the duff for its members. ResumeFit.com, meanwhile, serves the recruiter by incorporating a candidate assessment right into the resume. As a company partner told us, “this is great for triaging candidates on the front end.”

UpMo

Because this is a subscription-based service, there is just so far you can get before you must enter a credit card. From what we could see and from what its founder and CEO, Promise Phelon, says, UpMo is a career guidance resource that seems especially well-tuned to the young professional who will work for upward mobility (ergo, the UpMo name).

UpMo takes the pulse of your current employment readiness by assessing such things as the quality and breadth of your network. How often are you in touch with the people who can help you? It also compares you against a representative role model, charting your career against the hypothetical (or actual) individual whose career path you want to emulate. Like a good financial planning program, you can do “what-ifs” with your career to see how it changes the trajectory. Write a book; become a conference speaker; get an advanced degree are among the multiple choices. Of course, if you can’t write or speak, then those wouldn’t be good choices for you. UpMo is realistic about life, but it’s not going to tell you what you can or can’t do; only how actually doing it will make a difference.

Once you’ve got something you like — and can actually do — then UpMo creates a career plan so detailed it will break out the tasks into calendared increments and remind you to make that networking phone call or send a courtesy email. Ignoring it, of course, is optional. But the program is built on networking principles espoused by every career professional, so if you can keep to the schedule (which, by the way, you set) then that career timeline UpMo created for you can be achieved.

ResumeFit

You need an accountant who can handle all the financial details of your business, from cash accounting to preparing the tax returns for your growing, but still small business. So you post the job and get 200 resumes in the first 24 hours. As you work your way through the resumes, you quickly realize that it’s easy to weed out the obvious nos, but the resumes in the pile of possibles are all beginning to sound alike. Here is where ResumeFit fits in.

As Managing Partner Scott Runkle explains, “Our goals are simple near term: help a job seeker better differentiate themselves from their peers, and help the employers better understand how the job seeker may fit the role they are applying for.”

ResumeFit helps differentiate candidates by providing an assessment of their “soft skills,” those personality traits and work habits that help define “fit” and which can mean the difference between success and failure on the job. The assessment used by ResumeFit is called the WorkPlace Big Five Profile, which is based on the Five Factor Model of Personality. Job seekers take the assessment, then pay to make it available to employers.

It’s not a replacement for job-specific testing or those designed to measure how well a candidate fits within a very specific corporate culture, but for the majority of jobs and companies, a ResumeFit profile is a way to differentiate among candidates.

“For many small to midsize companies who may not even use assessments, the employer gains valuable insight into a candidate they may not otherwise gain from just the resume alone,” Runkle says.

ResumeFit offers a corporate screening service, in addition to the job seeker assessment. Role Fit Screener compares and ranks candidates against the profile of a company’s best workers.

BigDogHub

This is a site with a name to which we’re partial, even if it is misleading in that it has nothing at all to do with big dogs, literal or figurative. Instead, this is one of the new crop of profile sites, where resumes are enhanced with videos, pictures, personal data, and so forth. Ditto for employer sites, where the sample company profile is of founder John Hughes’ search and recruitment company.

The biggest differentiator here is that employers can post questions to the profiles. As the questions and answers grow, recruiters get a clearer picture of a candidate and their abilities and skills. Candidates get to do the same to employers.

Because of the nature of the site, job postings and profiles are only available to registered members.

Jobaphile$

Another “me too” job auction site, this one appealing to students, is built around the eBay model. Employers post a job; job seekers bid. Employers choose the bid they prefer based on price and the bidders qualifications.

What makes this different from the pack of other freelance and project sites in the world? Nothing that we can see. Yet we’re hesitant to dismiss it entirely, since Jobaphiles was nurtured by startup incubator Dreamit Ventures. It’s also gotten a bit of gee whiz publicity from Fox and an NBC affiliate.

The auction job genre keeps trying to gain a foothold outside the freelance world, but the concept has never taken hold. Instead, those college student jobs like tutor, nanny, research assistant, and so on gravitate to the “Gigs” section of Craigslist.

According to Gartner, Inc., the world’s leading information technology research and advisory company, more than 25 percent of the content that workers view each day will be dominated by pictures, video or audio by 2013.

As of this past February, emarketer.com ranked YouTube as the fifth-most popular website in the United States, eclipsed only by the likes of Google (who owns YouTube), Yahoo, and Microsoft.

Video, combined with the Internet, is a game-changer for recruiting. Used together they create a better candidate experience and raise the likelihood of a better hire. They also enrich recruiters by giving them a much deeper perspective on a candidate, in less time, than has ever been possible.

Video is particularly attractive to Gen Y — those young people between 20 and 29 who total about 70 million people. They are avid users of video and expect to be marketed to, taught, entertained, and recruited by video. Go to an Apple store and watch what young folks are doing: watching videos or movies or looking at pictures using the Internet. I rarely see any of them reading an article or an online newspaper.

They have been raised on television and those in the 25 to 34 age group watch more than 140 hours of it each quarter. The percentage of people watching videos and movies on the Internet has nearly doubled since 2006 and is now over 60% of all Internet users.

Some organizations are already leveraging the Internet and video to give them a competitive edge in reaching the millions of people who regularly use such sites as YouTube and Hulu.

Here is how they are doing it:

To showcase their company
They are creating career sites that are heavy with short videos featuring tours of the company, interviews with executives, candid chats with employees, and day-in-the-life scenarios of what people in particular positions do all day. They may include videos about the local area or videos that have been made by news agencies about the company. Examples of excellent career sites that contain video include those of KPMG, Deloitte, and Whirlpool. These have all won awards for excellence based on the success they have had in recruiting the talent they need using their career site. Companies such as RecruitTV and Thinktalk provide the expertise and service to help you produce these kinds of videos.

An interactive, video-based website is the core requirement for employment branding and may be the single best thing you can do to improve your success in attracting and hiring the people you want.

To post or distribute jobs
It is now possible to make a short video specifically describing a particular position, and then use that video instead of the usual written description. In London, three career magazines now provide this as an alternative to the written word. A Twitter-like application called 12Seconds allows you to make, yep, you got it, a 12-second video about a job and distribute it to a group of followers.

Monster Canada allows you to insert a streaming video into any job posting. And climber.com posts your video job description focused on Gen Y candidates to 45 different video sharing sites.

To hold career fairs
Virtual career fairs have been around for a while, mostly focused on college recruiting. CollegeGrad.com offers this type of virtual careers fair. For a broader audience CareerBuilder, Unisfair, and InXpo. A virtual career fair has much greater reach than a physical one and allows candidates to learn more about the positions you have and your organization. They are cost-effective ways to reach out to a broad geographical slice of people, quickly.

To do targeted marketing
Advanced and emerging uses of video include, for example, having your job video display when a person goes to a particular website or webpage. All clicks on your job display are tracked so that you can see who and how many show interest. This information will allow you to narrow down the sites where you display the ads, improve the content of the videos, and control costs.

Product marketers have used similar technology for a while and are now making it available for recruiting. As this technology matures, it will be possible to greatly reduce the number of unqualified applicants by limiting who actually sees a job ad.

To interview candidates
Interviewing candidates by streaming video is becoming more popular now that more than 60% of Americans have broadband access from their homes. With a simple webcam and a decent Internet connection using Skype, a recruiter or hiring manager or both together can interview a candidate from anywhere. This lowers costs and time to offer and provides a candidate and the hiring authorities an experience that is often as good as if not better than a face-to-face appearance.

Many companies offer video interviewing including Greenjobinterview.com, Clooks.com, and Hirevue.com.

For assessment and screening
A final way that video is being used is in candidate assessment. By creating scenarios and games that stimulate real-world experiences, recruiters can gain insight into how people would potentially react to them. These job simulations have been used by the U.S. Army and by retail stores intent on seeing how potential sales associates might respond to different customer problems. The U.S. State Department has recently started using a game to assess potential Foreign Service officers. It is called American Diplomat and recreates many of the scenes and issues a diplomat may encounter.

Another aspect of assessment is the self-assessment that candidates make when they actually see what it is like to do a particular job. Shaker Consulting does a good job of creating validated job previews that help candidates self-assess, as well as help recruiters and hiring managers.

Video is rapidly becoming core to recruiting success. Organizations that do not start to build video into every aspect of talent acquisition will find that they are at a competitive disadvantage, especially with college students and younger experienced hires. This is the age of video and we all need to learn to use it better.

For the last seven years, Rocket-Hire has surveyed users of web-based pre-employment assessment tools, so we again asked members of the ERE community to tell us about their usage of typical pre-employment screening, testing, and assessment programs. As with years past, we zeroed in on the pulse of pre-employment assessment usage. And in an increasing climate of legal scrutiny for testing, and the hoopla surrounding the Ricci case, we decided to focus the content of today’s article on two issues that are inexorably linked: Implications of evaluating one’s assessment strategy, and attention to relevant legal issues.

Those interested in obtaining a copy of our full report can email us (chandler@rocket-hire.com) and we will be sure to send you a full copy once it has been completed.Or, check out an upcoming Journal of Corporate Recruiting Leadership, probably the October 2009 issue, where we’ll have an in-depth analysis.

A Word About our Methodology
This year, 148 recruitment and hiring professionals completed the Rocket-Hire Online Assessment Usage Survey. Respondents were evenly representative of recruiters, recruiting leaders, HR executives, business owners, and hiring managers, and featured a wide variety of organizations and hiring situations.

Use of Assessment Tools
Overall usage of assessment tools was generally about the same as in past years — roughly two-thirds of respondents. Of that two-thirds, 54% are deploying both paper-based and online assessment, and 30% are using exclusively online assessment. The remaining employ only paper-based tools. Most use a variety of different assessment methods, with the majority using between one and three different types. The table below reveals the proportion of firms using various common tools.

Usage rates of common assessment tools

Skills and knowledge certifications and personality assessments continue to be popular, with indicators of cultural “fit” and various cognitive ability measures also widely used. In addition, we observed a surge in the adoption of web-based background investigation strategies.

Interestingly, 2009 highlights a trend emerging over the last few years: It is now just as common to find organizations deploying assessment company-wide as it is to find it used for only one or two jobs. These findings reflect the fragmented use of assessment that we have experienced in our client work. To be sure, there are tons of ways assessment can be used, but there is no one major trend in how or how much it is implemented.

Effectiveness of Screening and Assessment Tools

This year, 80% of assessment users felt that assessment had a “positive impact” on their organization. In contrast, only 3% of assessment users felt these tools did not add value. However, only 64% of assessment users collected metrics to judge the quality of their interventions, and 79% judged ROI using one or many indicators. However, definitions of “ROI” vary widely, with some approaches considering a broad, cursory opinion from managers, whereas only a few use a true scientifically derived method such as a validation study.

Wide variation in the real-world evaluation of ROI clearly reveals that, while many companies do collect metrics, few of them are going about this in a way that reflects best practices. This is cause for concern given the fact that these same best practices create the foundation for demonstrating the legal defensibility of assessment measures.

Legal Issues

This year’s Supreme Court decision in favor of firefighters who challenged the City of New Haven, Connecticut, has once again focused attention on the legal aspects of pre-employment assessment. In anticipation of the High Court hearing this case, we wondered how much typical users paid attention to legal and regulatory issues related to using assessment to evaluate candidates.

Percent indicating legal issues are “central to this activity” or “very prominent”

These results express both positive and negative connotations. On the positive side, it seems that a good number of companies pay attention to legal issues related to testing and assessment. On the other hand, perhaps more firms should consider devoting resources to demonstrating and documenting legal defensibility. The lack of understanding of what is actually required for ensuring compliance and defensibility may create problems as both assessment usage and legal scrutiny increase.

The energy spent on legal considerations does not relate to the extent of assessment deployed, or size of the organization, nor number of hires. As with overall usage of assessments, legal concerns do not follow any pattern across survey respondent demographics or assessment strategies.

Legal defensibility is not just another set of meaningless hoops that must be jumped through and checked off. Proper evaluation of ROI via validation and legal defensibility are actually two sides of the same coin, the common goal being to document the job-relatedness of all tools and processes used to select employees.

Though the continued economic turmoil has put a damper on hiring, the adoption of in-depth assessment tools continues to grow, especially as more organizations devote resources to find out if testing and assessment is worth it. The bottom line: Quality hiring tools can improve performance and employee engagement (i.e., provide ROI) while also ensuring the legal defensibility of hiring practices, but it is essential to demonstrate how much they actually benefit your organization.

I therefore thought that it might be helpful if I provided an overview of the concepts and logic behind this system. Whether or not you choose to actually apply the process, I certainly think it should be given consideration first.

Key Advantages of Competency-based Selection

• Improved objectivity, leading to truer talent acquisition
• Consistency in hiring decisions
• Legal protection for the employer
• Improved consensus on hires across multiple stakeholders

How it Works
The intention of a competency-based assessment is to move hiring decisions away from the classic intuition-led process (i.e. away from: “I feel that this person would be a great hire”). Instead, the system brings in a certain amount of objectivity, replacing much of the subjectivity of traditional assessments. Candidates are not primarily assessed or rated based upon their aspirations, opinions, or similar. Rather, the underlying premise is that we can anticipate how a candidate will behave in a role in the future through an assessment of their behavior in the past.

In competency-based interviews, questions are asked that require reference to specific events (e.g. “When was the last time you were overwhelmed by your workload?”). Candidates are discouraged from giving general answers and asked to focus on specific incidents. By examining how a candidate has actually approached real situations in the past, we can judge more accurately how they will act again in similar circumstances.

I state above that this is a “certain” amount of objectivity, rather than total objectivity, because no selection system can be expected to entirely remove opinion. However, what competency-based selection does is provide substantial justification for recruitment decisions. By drawing upon multiple sources and contributors, and aggregating the data drawn from these interviews, the outcome ceases to be an individual manager’s choice and instead becomes a consensus. This means that new hires will have the buy-in of all stakeholders, improving prospects for their successful induction into the company. It also results (through keeping the structured interview notes from each interviewer) in a substantial body of evidence to justify hiring decisions, protecting employers from litigation.

Setting the Target
In order to be able to make a judgment on a candidate’s capability, we need to have a frame of reference to set them against. This is something that is individual to every business, as no two organizations have the same expectations. Candidates need to be assessed against defined personality traits (competencies) appropriate to the role in question. Typically these competencies will be drawn from a large global framework, in combinations that are different for each business area (as different departments will need different profiles).

Competencies tend to measure behaviors such as Adaptability, Working under pressure, Customer management, and so on. A candidate’s interview performance against each competency is related to “Key Actions” and other criteria that the employer has defined as belonging to the trait in question.

Rating the Candidate
By asking a series of specific questions (at Red Hat we provide interviewers with the questions they need in a formal interview guide, customized for each business area) and rating the capability that the candidate has demonstrated in their answers against the chosen competencies, the interviewer has all that they need to move into the process of decision-making.

All interviewers gather together (virtually, where required) to discuss their findings. The different scores that they have each given to the candidate are compared and discussed, enabling agreement of an overall score for the candidate against each competency. It is these competency scores that are then used to decide whether to pursue a candidate further or not.

In the process of discussing their findings, there will be differences of opinion between interviewers. That is where the interviewer’s notes are essential, as all decisions on scoring need to be justified by findings from the interview. I’ve found that an interviewer knowing that they will have to justify their score encourages a high-level of self-discipline in producing these notes. Thus there is no room left for hunches and loosely-explained opinions, which as I stated at the start is one of the intentions and benefits of a competency-based system.

We’ve seen why a competency-based selection system can benefit businesses in a number of directions. By having a consistent and objective process in place, employers are not only being transparent and fair to candidates, but they are also greatly reducing corporate risk from litigation associated with decisions that can be challenged as unfair or discriminatory.

Perhaps most importantly, introducing this objectivity helps ensure that organizations are truly hiring the best talent, which is ultimately our goal as recruiters.

A complete discussion of the intricacies and technicalities of validation, discrimination, adverse impact, and differential prediction is beyond the scope of the words I am writing today. Suffice it to say that this case has placed competing priorities in the use of testing in the spotlight. These competing priorities are using fair testing while striving to eliminate discrimination in hiring. While title VII of the Civil Rights Act of 1964 has attempted to provide some guidance in relation to these competing goals, the Ricci case has laid bare some critical issues that in my opinion certainly call for the government to re-evaluate and modernize the standards it has set.

We are mandated to use valid tests. Valid tests can often lead to minorities being hired at lower rates than those of other races. This is seen as OK as long as the test has been validated, because in theory this means the test is job-related and job-relatedness is the standard by which the legality of testing is determined.

However, what are we to do when sticking to the use of validation — as we have been asked to do — creates a situation that actually inhibits the goal of ensuring diversity and fairness? This has been a thorny issue for those of us in my profession for a long time. There is no magic bullet. The dissenting opinion in this case led by Justice Ginsburg rallies around the idea that the spirit of diversity and fairness should be the highest standard to which we aspire in hiring. It is hard to argue with this point … except for the fact that there are technical issues which can stand in the way of our achievement of this goal.

So, what does all this mean for hiring in the corporate world? I offer my humble answer to this question as follows:

Don’t Panic -- Police and fire testing is the most highly scrutinized type of testing known to mankind. Don’t panic based on the results of this case. Do use this as a time to think about your use of testing and where it may leave you exposed.

Validate, validate, validate -- In this case the validity of the test was upheld. In my mind the validity of the test, while an issue, was not the main issue at hand. The only reason the city tried to throw out the test was because it ended up being counter to its goal of diversity. Despite this, I cannot stress enough the need to validate all testing that is used to make employment decisions. It is the cornerstone of best practices in testing and provides the documentation you will need should you find yourself in court. Without such documentation, you are toast! As an added bonus, validation is the process that provides awareness of issues such as adverse impact. You may not even know you have a problem unless you take the steps to validate. Remember, ignorance of the law is no excuse!

Look at the bigger picture -- I agree with Justice Ginsburg that the overall goal of eliminating discrimination is the highest standard to which we should be held. In the corporate world this becomes an issue of fairness in hiring practices across the board. One of the biggest ways to guard against problems while working to achieve diversity is to look at the demographics of your workforce vs. those of the available workforce in the area. If these do not look about the same, you have a problem. This problem can be rectified by actively recruiting for diversity. Diversity training programs are OK, and of course I support them, but the best thing to do is to put your money where your mouth is and be aware of your demographics and seek to hire diversity at all times.

Seek out testing that has been shown to reduce adverse impact -- The Uniform Guidelines on Employee Selection Procedures pretty much lay down the law when it comes to testing. A key part of this doctrine is that one should always seek out tests that are known to have less adverse impact. We know that cognitive tests have the most adverse impact while also providing the best predictive accuracy (i.e., validity). Resolving this conundrum remains the crux of the issue, with the Ricci case as firefighter tests are highly cognitively loaded. In the real world I feel this issue is best addressed via awareness of what is required for the job and by seeking out selection procedures that we know can test cognitive traits while displaying lower levels of adverse impact. If you guessed that I was going to recommend simulations as the best way to accomplish this goal, you are correct! The issues of this case are yet another piece of evidence that clearly demonstrates the value of simulations over more traditional types of testing.

I look forward to the discussion that my opinions generate and I am glad to see my corner of the hiring world getting its brief exposure in the national media spotlight. I certainly hope that the awareness generated should serve as a catalyst for change.

Ruling 5-4 in the case of Ricci v. DeStefano, the court’s majority said just because a disproportionate share of whites pass a test does not make the test discriminatory. Writing for the majority, Justice Anthony Kennedy said invalidating test results because of the statistical racial outcome, “… is impermissible under Title VII unless the employer can demonstrate a strong basis in evidence that, had it not taken the action, it would have been liable under the disparate-impact statute.”

Now, before an employer looking at the racial makeup of those who passed and failed a promotional exam and, almost certainly, other types of employment exams, can decide to throw out the results because it fears a discrimination lawsuit, it must have “a strong basis in evidence” to believe the test is discriminatory under Title VII of the Civil Rights Act of 1964 and its amendments.

Employment lawyers reacted with caution, saying the 92-page decision, including a dissent by Justice Ruth Bader Ginsburg, will take time to digest. Their initial impression, however, is that the court appears to have tempered if not invalidated the 80 percent rule of the Equal Employment Opportunity Commission. That rule states that the selection of a group at less than 80 percent of the group with the highest rate will be considered by the EEOC as evidence of discrimination.

However, the Supreme Court ruled statistics are not enough to show disparate impact. The decision says, “… a prima facie case of disparate-impact liability — essentially, a threshold showing of a significant statistical disparity …  and nothing more — is far from a strong basis in evidence that the City (of New Haven, Conn.) would have been liable under Title VII (for discrimination) had it certified the results.”

Merrily Archer, an employment lawyer in the Denver office of Fisher & Phillips who was previously with the U.S. Equal Employment Opportunity Commission, says the decision appears to be “completely at odds with the EEOC guidelines.”

Merrily Archer

“What does it mean to say an employer needs a strong basis in evidence?” she says. “My concern is an employer in the trenches. How is an employer going to apply this?”

“I disagree with the decision,” Archer adds.

A second attorney, who asked not to be identified because he had not completely read the opinion, suggested that the “court has muddied things up. The 80 percent rule was pretty straightforward for an employer: If you didn’t hit that percent, you had a problem.”

Today’s decision came in a case from New Haven, Conn. brought by a group of  white and Hispanic firefighters who scored high enough on a promotional exam to have been appointed to one of several captain and lieutenant openings in the department. No blacks were in the promotional group, although they made up about 30 percent of the department’s workforce in 2003 when the test was given.

Although New Haven spent tens of thousands of dollars hiring consultants to develop the promotional exam and validating it, the city’s Civil Service Board refused to accept the results, essentially denying promotions to the successful test takers. It heard testimony over five days that the test was deficient and that a less-discriminatory test existed. In the end, the city’s decision was based largely on the statistical results, which showed the promotional exam had a disparate impact on blacks.

Kennedy’s decision invites employers to engage stakeholders in the planning, design, compilation, and validation of these exams, but doesn’t require any particular process to be followed in the development and selection of a test. Once the test is given, though, the mere statistical results are not enough to invalidate it. Writes Kennedy:

“Nor do we question an employer’s affirmative efforts to ensure that all groups have a fair opportunity to apply for promotions and to participate in the process by which promotions will be made. But once that process has been established and employers have made clear their selection criteria, they may not then invalidate the test results, thus upsetting an employee’s legitimate expectation not to be judged on the basis of race. Doing so, absent a strong basis in evidence of an impermissible disparate impact, amounts to the sort of racial preference that Congress has disclaimed, §2000e-2(j), and is antithetical to the notion of a workplace where individuals are guaranteed equal opportunity regardless of race.”

The court’s decision is a victory for the firefighters who sued and a slap to Supreme Court nominee Sonia Sotomayor. She was one of three appeals court judges who issued a one-paragraph ruling upholding the city’s decision. The brevity of the decision as well as the notoriety of the case has become an issue in  her confirmation by the U.S. Senate.

Lead plaintiff, Frank Ricci, became a sympathetic figure after telling the Civil Service Board that he had dyslexia and had paid a neighbor to read onto tape the study materials.

I was recently reading an article in which many of the members of the mission control team were interviewed about the mission and the various roles and tasks they performed. One of the most interesting things was a discussion about the fact that almost all of the mission control team members were very young.

The mission required extensive use of computers and the ability to use computers to do things that had never been done before. Computer science was such a young field, and the moon mission so unique, that there were no persons with any experience doing what needed to be done. In order to accomplish the mission, NASA hired a group of people who had the ability required to work with computers and experience working with computers, no matter what the application. The article did not say what methods were used to determine how hiring decisions were made, but clearly a non-traditional approach was required. We all know the result of the decisions NASA made and we all know that the combination of workers, equipment, and planning resulted in a resounding success — the kind of win any corporation would be proud of!

Even though the Apollo 11 mission happened 40 years ago, it’s ripe with lessons for those of us in the hiring profession. This is especially true when it comes to creating strategies to use assessment in the future of the hiring and the workforce. My thoughts:

• As has always been the case, the bottom line is that understanding the traits valued for getting the job done and including the scientifically based assessments required to measure them in your hiring process will help ensure you achieve successful performance of your mission.
• Creating the future may often require taking a leap of faith in someone’s abilities to make a contribution. Testing can help make this leap easier by helping to identify those individuals who possess the raw abilities and characteristics required. A caveat here is that — as with any employment testing situation — time must be taken to identify abilities and characteristics critical to desired outcomes as a first step in the development of the testing program.
• Training and experience are an excellent way to help mold the raw material that brought to the table. When seeking to do something that has not yet been done using new technologies, training is essential, as is the ability for hands-on learning in which team members can educate one another.
• Ideas are and will continue to be the currency required for successful progress. Identifying applicants who are creative and possess the proper thinking styles will become increasingly important.
• Bringing together persons with backgrounds in different areas and asking them to function together as a team will be essential to success. New innovations and progress are increasingly requiring input from those with vastly different backgrounds.
• Given the above point, it seems logical that an increasing amount of attention be paid to ensuring a harmonious cultural fit between the members of the team. It is often mismatches in work styles and values that cause problems within a team dynamic. There are an increasing number of applications which allow organizations to measure, model, and optimize fit when creating and aligning work teams.

Technology is continuing to make the above points easier for those folks with out a background in testing and assessment. As with almost every other area in our lives, technology is radically changing the landscape. When is the last time you sent a fax? Could you be as efficient without email? We are entering an age of unprecedented accelerated technological advancement. The time horizon for quantum leaps in technology has been shortened to milliseconds when one considers the grand scheme of things. The computing power and things needed to put a man on the moon 40 years ago can probably be duplicated by several laptops now.

The things we are going to see in our lifetime are going to blow our minds. This statement will be true for those of us in the general business of hiring, and those of us who specifically concentrate on assessment. Virtual simulations of entire jobs, human interactions, and interactions between humans and machines are going to be the future of hiring. I have just started to see some of the first steps in this direction. What I have seen and learned so far is that it is going to require a diverse set of perspectives to create the hiring tools of the future. Folks in the testing and recruitment industries are going to need to team up with persons in other, seemingly unrelated industries in order to make it happen.

I am currently working on putting together a consortium that represents key stakeholders from the assessment and gaming industries as well as persons from other areas yet to be defined. I am just at the concepting stages at this point, so I encourage interested parties to post here if you want to talk about this idea.

For as long as I can remember, recruiters have focused on cost as the primary measure of their effectiveness and value to the organization. The most popular recruiting metric has been cost-per-hire, and recruiting functions justify their existence by showing how much less expensive they are than an outsourced solution.

This, however, has begun to change.

Even though we are in a recession, skilled talent is hard to find, and demographic projections indicate a long-term swing toward a candidate-driven market. If you are in Europe or recruit for European operations, the aging workforce and the lack of fresh, skilled talent has to be a major concern.

Positions are open longer and hiring managers ask for more resumes to review, not being satisfied with those recruiters provide. There may be hundreds of candidates on the job market at the moment, but managers are still frustrated at the inability of their internal recruiters to find what they consider quality talent. The most important metrics today are those of speed and quality. The best recruiters are measured on how quickly they present candidates and on the quality of those candidates.

In many organizations, outsourcing decisions are being made on these metrics, not on cost. Managers are finding that having a good employee when they need one is much more important than how much it costs to get him or her.

But one hurdle looms over all of this. That hurdle is to define what we mean when we say that one candidate is “better” than another. How do recruiters and hiring managers define quality? Who defines it? And how can it be tracked? These are the tough questions that need answers.

Quality can be defined and here are a few ideas on how to do it. The only caveat is that this process has to be dome for each type of position in your organization.

Number 1: Establish specific competencies or traits that equal quality in the minds of your hiring managers and use it to assess candidates

Most hiring managers do not have any definition of a “quality employee.” Some managers say that they know a quality employee when they have one, but they struggle with a hard definition.

A recruiter’s job is to help them create that definition. The place to start is to unravel the skills, competencies, and traits of the best performers. Unravel the ones that really differentiate average and superior employees and make a list of those. A list should be short and clear, with levels of accomplishment included. It might look something like the diagram you see with this article. I usually try very hard to keep the list to two key items per topic.

It may also be very useful to look at the worst performers and see what it is they don’t have. By listing the characteristics that are common to both the best and the worst employees in a function, you will begin to develop a profile that can eventually be used for selection, performance management, and development. These characteristics could be traits such as willingness to compromise, an open attitude toward new ideas, or frugality in business dealings. Or they could be competencies such as the ability to create spreadsheets in a certain time, or the capability of editing complex documents. And they can also include a level of knowledge such as expert-level knowledge of Unix or of a manufacturing process.

Most likely any definitions of quality would include elements from each of these categories. Notice that these are all output-based measures — in other words, measures that can be seen or demonstrated in the work an employee does. They are the opposite of input-based measures such as length of experience or level of education. These types of measures tell you very little about the quality of a person’s performance.

You may need to partner with your internal organizational development group or with your training department to do this. It does take time and it takes willing managers to partner with you in the process. The result, though, will be a much clearer understanding of what kinds of people need to be sourced and hired.

Number 2: Educate hiring managers

Very few hiring managers know much about selection or about what it takes to assess a candidate. Even though you may have put all the managers though some sort of interview training, I am sure they have forgotten most of it and have used it less. Most of us are not disciplined nor can we expect the typical manager to become expert with these techniques. One area where recruiters can add value is to pre-screen and evaluate candidates against the criteria that you developed above. These criteria, remember, should have been determined in partnership with the managers. Each of you can use lists of these and behavioral interview questions or a variety of tests can be developed and used to measure these traits, competencies, and knowledge. Managers can help you determine how to weight the criteria, and they should be well aware of the consequences of using the criteria.

You can spend small amounts of time over a few weeks presenting bits of this information and moving the managers to understanding and acceptance. If you can, you could also hold seminars and use case studies and examples from your own organization to help managers understand how important it is to select people with the right skills and the right organizational fit and attitude.

Number 3: Investigate and experiment with new tools for screening and selection

It is still a bit surprising to me that very few firms are taking advantage of the many online tools that are emerging to help screen candidates before investing a large amount of time in interviews. By using the Internet and your corporate website, you can ask candidates to engage in a dialogue and mutual assessment process. While you are looking at their skills and fit, they can be looking at your organization and can make decisions on whether or not they like what they see. Many people I have spoken with have seen one side of an organization while interviewing, and another less attractive one after they are hired. There is still value in letting candidates email other employees for information about the company and work-life. There is a need for job previews and better job descriptions that are based on reality, not what we wish were true.

By defining up front what constitutes a quality candidate, you can remove much of the present frustration candidates have over why they were not chosen for an interview, and you can also reduce the number of unqualified candidates who apply. Many do so because they do not know or understand your definition of quality.

By working with hiring managers, getting them to write down and define for you the competencies and traits of successful employees, and by putting those to use in your screening and interviewing processes, you can improve candidate quality in a measurable way.

So much has been reported and written about the case of Ricci v. DeStafano that it’s almost impossible to have missed the story of how 20 New Haven, Conn. firefighters were denied  promotions although they came out on top in civil service tests for lieutenant and captain. Eighteen of the top scorers were white; two Latino. None were black, although the city is 37 percent black and blacks made up 30 percent of the fire department in 2003, when the test was given.

When the city’s Civil Service Board got the results, it feared certifying the test would expose the city to a Civil Rights lawsuit on the basis that the test had a disparate impact on a protected minority. But not certifying the results meant an almost certain lawsuit from the successful candidates who might claim, as they later did, that they had been discriminated against based on their race. A part of Title VII of the Civil Rights Act of 1964 makes it illegal to “alter the results of, employment related tests on the basis of race, color, religion, sex, or national origin.“

The Morton’s Fork faced by the board was underscored by its 2-2 vote on certification, an outcome that meant the test results were not certified.

New Haven’s dilemma was neatly described by Justice David Souter during the Supreme Court hearing on the matter in April.

David Souter

“The problem I have with your argument,” he told the attorney for the firefighters, “is that it leaves a municipality or a governmental body like New Haven in a damned if you do, damned if you don’t situation… If they go forward with their hiring plan, they certify the results and go forward with it, they are inevitably facing a disparate impact lawsuit.

“If they stop and say, ‘Wait a minute, we’re starting down the road toward a disparate impact lawsuit and, indeed, there may be something wrong here,’ they are inevitably facing a disparate treatment suit. And whatever Congress wanted to attain, it couldn’t have wanted to attain that kind of a situation.”

Key questions

The key question the Supreme Court must decide is: Can a municipality — and potentially any employer — can reject the results of a test for racial reasons, even if a disparate impact is not proved? There are two other specific questions presented for the court, but they boil down to the same fundamentals.

Under court decisions and the Civil Rights Act of 1964 as amended by Congress in 1991, an employer can be found to have discriminated, and thus be in violation of the law and subject to civil penalties, if a “facially neutral employment practice… has an unjustified adverse impact on members of a protected class.”

But if the employer can show the test is job-related and there is a business necessity for its administration — say, testing the ability of a firefighter candidate to hoist a ladder while dressed in full gear — then the employer may be able to escape liability. However, there is a loophole here. The group claiming discrimination may still prevail if it can show that there are other, equally valid assessment methods meeting the employer’s objectives that don’t result in a disparate impact.

Job simulations

In the only published academic research comparing the results of pencil-and-paper tests (like the one administered to the New Haven firefighters) and interactive simulation testing, Amy Mills, of Aon Consulting, and Dr. Neal Schmitt of Michigan State University found little difference in the predictive value of the two on job performance. They did find that with the simulation, the performance of minority candidates was similar to that of white candidates. On the pencil-and-paper tests, minorities scored significantly lower.

“There’s less of an adverse impact in simulations than in the pencil-and-paper tests,” Schmitt told us for an article published in the May 2009 issue of the Journal of Corporate Recruiting Leadership.

In fact, in the New Haven test, the city weighted the written multiple-choice part, which is the subject of the lawsuit, at 60 percent of the overall score. An oral exam was weighted at 40 percent.

Though the disappointed firefighters presented evidence supporting the test, while the city offered statistical data to demonstrate a prima facie case of disparate impact, the validity of the test as a promotional tool was never actually at issue. Instead, the federal court ruled that the city had the right to throw out the results.

In ruling for the city, U.S. District Judge Janet Bond Arterton called the city’s decision “race neutral,” since “all the test results were discarded, no one was promoted, and firefighters of every race will have to participate in another selection process.”

The Second Circuit Court of Appeal upheld the decision in a single paragraph, that has now become a central part of the national judicial debate because Supreme Court nominee Sonia Sotomayor was one of the three judges to hear the firefighters appeal.

What will the court do?

Many observers think the court will rule in favor of the firefighters.

Chief Justice Roberts

Chief Justice John Roberts signaled his dissatisfaction when he asked during the oral arguments if the city was to “get do-overs until it comes out right?”

Justice Antonin Scalia challenged the idea that the city had been racially neutral in tossing the results. “It’s neutral because you throw it out for the losers as well as for the winners? That’s neutrality?”

If the court does overrule the lower courts, it could simply order the case back to the lower court to decide the city’s motives in tossing the test. The city would then be in the uncomfortable position of attacking the validity of a test it commissioned and approved, but Title VII would remain intact.

The possible, broader implications of a reversal could be the watering down of the “disparate impact” portion of the Civil Rights Act. The court could say the city had no right to refuse to certify the test simply because of the outcome.

At the extreme, the court could go so far as to rule sections of the Civil Rights Act unconstitutional, though almost no one expects that broad a decision.

One of the most exciting things for me at this year’s conference was the launch of SIOP’s new blog/interactive community site, the SIOP Exchange.

I was part of a team that created this blog in order to help promote I-O psychology and build an increased sense of community amongst SIOP members and other interested parties. I encourage those folks in the ERE community who are interested in the viewpoint of I-Os on topics related to our work to check it out. The Exchange offers RSS feeds that will help keep you aware of topics that may be of interest to you.

In addition to launching the blog, this year I participated in several panels in which assessment solution providers and the end users of assessments discussed important issues related to technology and testing. It is rare to such varied experience and expertise in the use of assessment in one place. I want to share some of the hot topics with ERE readers to help keep the ERE community updated on how testing and assessment experts are handling important issues that impact the use of technology based testing. Here is a quick rundown of some of the themes that were represented.

Technology is more accessible than ever: We have reached a point where the differences between the technology platforms of assessment providers have started to level off. Almost every company offering assessments now has a relatively sophisticated platform that can handle the basics of test delivery, scoring, and reporting. A good deal of providers also offer a nice candidate management system as part of their platform. One interesting facet of this movement is that I-O knowledge is starting to get “baked in” or embedded into the technology system. This trend is going to help make quality technology based testing available to small and mid-market companies. While I believe this to be a positive trend, we still need to be aware that there are trade-offs to be made. Several of these are discussed below.

Test security. As always, there was a good bit of discussion about the security of Internet testing. One of the biggest issues was the use of proctored vs. unproctored assessments. While there are some firms that currently do not allow their assessments to be used in an unproctored environment, the majority of providers will allow it. We are starting to see a variety of interesting methods to help mitigate cheating. My takeaway is that each situation dictates the need to decide between proctored or not. I believe that there are enough security strategies available that the negative impact of cheating is likely to be minimal.

Computer-adaptive testing. One of the most effective strategies to help thwart cheating is the use of tests that draw on large item banks in order to help ensure each test is different while also adapting the test content to the test taker’s ability level. CAT allows for shorter, more accurate tests. While it has been in use for years in the world of standardized testing, the leaders in the pre-employment testing community are starting to adopt this technology for their assessments. This marks a significant step forward in both security and usability.

Defining performance standards. We are still struggling with the line in the sand when it comes to thoroughly and accurately defining the performance dimensions to which an assessment will be linked. The rise of technology based hiring platforms has led to the streamlining of what has traditionally been known as job analysis. While this may be OK in some cases, we are still struggling to understand at what point we are taking liberties. My opinion is that thorough job analysis is always a good idea. Especially given increased activity by the OFCCP.

OFCCP audits. I had several conversations about increased activity in the area of OFCCP audits. This makes the use of best practices for assessment (job analysis, validation studies, documentation of adverse impact) even more important than ever. The cost of doing things right is likely to be much less than the fine you will receive if your audit does not go well.

Simulations (more to come in the Journal). The use of simulations is the cutting edge of our field right now. The offerings in this area are starting to increase in sophistication but are still mostly limited to call center and in-box type assessments as these translate quite well to a simulated environment. We have a long way to go in this area but I have seen notable progress in the right direction over the past year.

Technology and development. Assessment providers are continuing to link their pre-employment assessment products to onboarding and development products. This is a logical step when one uses competency models and understands that the pre-employment dialogue with a candidate offers useful data that serves to provide a baseline. End users of assessment have not fully caught on to the value of this viewpoint.

Every situation is different. One thing I gleaned from listening to assessment practitioners speak about their work is that every situation presents it own challenges. The contexts in which assessment is to be used vary quite a bit. Those seeking to use assessment correctly and ensure that it provides the level of ROI we know is possible need to ensure they enlist the help of an expert. There are lots of judgment calls to be made and it pays to have an expert to provide insight when important decisions are to be made.

Globalization. I-O psychology is more global than ever. The use of assessment has been rapidly spreading across Asia and Europe. It will be interesting to begin having access to data that can help us to understand the commonalities and differences across cultures and geographical locations.

A final overall impression is that we I-Os are still marginalized and underused. It was nice to sit in a room full of folks who know assessment works, know how to make it work, and can prove the value it adds, but frustrating to know that we are often not even given a seat at the table when important decisions are being made.

“So, how much revenue can you drive for us this year?”

I thought it was a joke, but he wasn’t joking. Maybe it was a trick question — no, it wasn’t. So, I said, “Before I answer, may I ask you a few questions?” He acquiesced…

How many salespeople can I hire?
What is the marketing budget?
What is the travel budget?
What is the budget for cost of sales?

To all of these fundamental business questions, the answer was, “I haven’t decided yet.” Very quickly what I initially thought was a joke became an interview nightmare. Red flags were waving in front of my face telling me to run from this opportunity as fast as I could.

After hearing his responses to my questions, I responded, “Revenue is a function of the investment made in both sales and marketing. How can someone give you a number that you can believe in without having answers to those questions?”

He leaned back in his chair and said, “Well, the other candidate gave me a number.” I told him that I could certainly provide him with a revenue number, but not to come looking for it. I attempted to explain further, but the CEO’s thought process was one-dimensional. He was interviewing a Vice President of Sales candidate as if the role was sales representative. (Mind you, I don’t recommend that question for that role either.) Following my instincts, I told him that it was best that he proceed with hiring the other candidate and I wrapped up the interview. He was dumbfounded to say the least.

If you are hiring a Vice President of Sales, there are five critical areas to explore of your candidate, but a revenue growth commitment is not one of them.

Sales force Recruitment. If you have decided to hire a Vice President of Sales, you are expecting them to build a sales team. Hiring salespeople is both risky and costly. The ideal candidate for this role should have a methodology that mitigates the risk and quickly gets the new hire up to speed. Bad hires kill the bottom line, but so do unproductive salespeople. Every day that a new salesperson is on the bench, not yet ready to generate sales, they sit on your books as a liability. Thus, a key skill that the ideal candidate will possess is development of a process to screen and onboard new sales team members.

Sales Process Development. One of the goals of having a sales organization is to establish consistent performance. This can only happen if a defined process has been established for the salespeople to follow.

Many companies hang their hat on the performance of a single rainmaker. One person generating 75% of the revenue means that you have one highly profitable team member and a bunch of unprofitable salespeople on the team. What happens if the rainmaker leaves for greener pastures? Having a well-defined sales process in place reduces the amount of time for new hires to get up to speed, as well as provides continued direction and focus for the tenured salespeople. This translates into another key benefit: scalability. Your company’s ability to experience significant growth resides on this leader’s acumen at building a process that leads the entire team to perform.

Compensation Plan Formulation. In many companies, one of the responsibilities of the Vice President of Sales is the formulation of a sales compensation plan. Sales compensation plans should be designed to reinforce the sales process that has been developed. Salespeople do not need a job description to tell them their job. The compensation plan tells them where to focus their time. The wrong plan can tank the company; the right plan can lead to explosive results. To learn more about developing the right compensation plan, read my article titled, “The Equilateral Triangle Model for Developing Sales Compensation Plans.”

Metric Management. In many sales environments, today’s sale is not necessarily an indication of a salesperson doing the right things now. Thus, you are paying commissions for what they did right one month, three months, or maybe a year ago. This makes it critical that other metrics are measured beyond revenue achievement. There is an old expression: what gets measured, gets done. In essence, the process that they create has multiple measurement points that allows for the creation for a dashboard. The metrics on this dashboard show the performance of the sales team and allow for intervention when performance is not meeting expectations. Thus the key is to understand how the candidate uses metrics to develop, manage, and grow their sales team.

Performance Management. The world would be a wonderful place if every salesperson hired performs like a rock star, but that doesn’t happen. You will have both over-achievers and subpar performers, and each requires a different management approach. Top performers need nurturing, appreciation, and growth opportunities, while subpar performers need support, guidance, and intervention. Handle the top performers wrong and they leave. Handle the subpar performers incorrectly and they can suck the profits from the company. Thus, in the interview process, it is important to understand the candidate’s management approach for different situations.

Not sure what questions to ask of your Vice President of Sales candidate? Send me an email and I’ll send you my favorite 24 questions when interviewing these candidates.

The Vice President of Sales is a key member at the executive table. As a business owner, when screening these candidates, focus on the skills that lead to the creation of your sales architecture® which means you are selecting a candidate that creates a sales organization based that delivers consistency, stability, and profitability.

Going back to the earlier story, that company did hire the other candidate … and fired him six months later after he did not deliver on the expectation he set in the interview for growing the business.

The purpose of this article is not to provide a detailed outline of the virtues of simulations (please refer to some of my earlier writings for this type of information). Beyond this, the crux of the issue is that simulations offer some really nice advantages over simple employment tests. These advantages include:

• A high degree of candidate engagement. Simulations are more fun and engaging then simply filling in radio buttons.
• A high degree of accuracy. Since simulations are miniature replicas of the job for which a person is applying, scores on simulations are likely to be strongly correlated with actual job performance.
• A realistic job preview. Simulations provide candidates with the opportunity to try out the job in question and allow applicants who do not feel the work is for them to remove themselves from consideration saving time and money.
• An employment branding tool. Fun and engaging hiring practices can really help reinforce an employment brand. Considering the trend in gaming and computer simulated environments, this may offer a competitive advantage when it comes to the coming generations of job seekers.
• Reduced bias. Simulations offer a way to help reduce bias and subjectivity in the hiring process due to their realism.

In order to better understand the future of job simulations for selecting employees, let’s take a quick look at the past and present state of affairs.

Simulations in the Past

Believe it or not, simulations for use in employment testing have been around for a really long time. The first simulations were used in industrial and manufacturing environments to help test an applicant’s ability to perform tasks that required strength and manual dexterity. Many organizations developed their own tests that were directly linked to the job in question. If a worker, for example, was required to thread a screw through a hole in a certain amount of time with a certain level of accuracy, then a mock-up of the task would be provided to potential employees. Pass the test and you are hired; fail, and it’s back to the ‘ol drawing board.

The fundamental idea behind these tests remains the same even to this day. This idea is that an employment test should have what is called “point-to-point correspondence” with the job. This simply means that activities found on the employment test are a miniature replica of activities that are required on the job.

Simulations for employment evolved beyond use in manufacturing to include exercises known as in-baskets. Here, applicants are placed in a situation where they are required to assume the role for which they are applying and handle the daily tasks associated with that job.

The drawbacks of each of the type of simulations used in the past are that the simulations themselves and the evaluation of the applicant’s performance on them were all analog. It was not possible to re-create many work environments and the low-fi nature of the simulations that did exist meant that humans were needed to score and evaluate applicant performance. This means that in the past, simulations were labor intensive and thus time-consuming and expensive.

Simulations in the Present

As with almost everything else in our lives, technology provided a serious upgrade to the evolution of employment simulations. While other areas of employment testing are still using much of the same content, simulations have benefited greatly from technological advances. For instance, it is now possible to recreate a great deal of work environments; performance on simulations can be evaluated in an automated manner; and simulations can be given remotely from anywhere in the world.

Here are some highlights of what is presently available in the world of employment simulations. I don’t usually use this column to mention specific vendors; however, in the present case I feel such mention is warranted given the value these products have to the evolution of simulation products.

Call centers: The bulk of simulations currently available for use in evaluating job applicants can be found in the call center space. Call centers are very amenable to simulations because the work environment (a series of computer programs and databases) is relatively easy to replicate and the tasks that make up job performance are easy to measure (data entry speed and accuracy, customer service, multitasking, etc). Some of the best call center simulations are those offered by FurstPerson, which has done an excellent job of creating technically sound and engaging simulation tools.

Manufacturing: Several companies are also currently offering excellent simulations for use in manufacturing environments. Today’s manufacturing jobs often place a higher premium on computer and logic skills than they do on things such as manual dexterity. Companies such as DDI and Select International have created excellent simulations for predicting a variety of valued business outcomes in the manufacturing sector.

Custom simulations: Simulations can be very expensive to create, especially when cutting-edge technology is involved. Most of off-the-shelf simulations currently available are off-the-shelf products that offer the ability for light customization. However, when it comes to accuracy and realism, nothing beats a custom simulation that has been created to directly correspond with the environment and requirements for a specific job. Shaker Consulting Group is a company that leads the way when it comes to building one-off custom simulations for hiring.

Managerial simulations and in-baskets: Managerial jobs are more complex than are entry-level jobs, and as such, creating simulations for these jobs has proven to be a difficult proposition. Modeling the job and the job environment can be difficult, and understanding behaviors that lead to success can be harder because successful performance is often a blend of a variety of traits. Censeo Corporation has recently come out with one of the first simulation tools that begins to offer a peak at the future of managerial hiring. This simulation uses scenarios and business problems to help evaluate key skills required for success in managerial positions. Beyond this product, a gap begins to be revealed in the ability to use technology to create simulations for advanced jobs. This gap highlights the drawbacks or limitations that currently mark the world of simulations.

While advanced managerial simulations (also known as assessment centers and in-basket exercises) are able to use technology to help create a realistic model of the environment and make administration of assessment content easier, these tools still require trained assessors to make evaluations. We are not yet at the point where we can evaluate complex behaviors in an automated fashion. We are facing a technological barrier in this area as translating choices and actions made by assessees into measurable human characteristics has not been successfully automated.

A Look at the Future

The best of simulations is yet to come. I firmly believe that as technology evolves, we are going to begin seeing a whole new breed of employment simulations.

I believe that the future lies in gaming technology. While this may not be true for every situation, most every job I can think of can benefit from game-type employment simulations. Take a quick look at the advantages I listed for simulations at the beginning of this article, and you can see that gaming magnifies all of them. Here are some of the companies that are working on the bleeding edge of gaming and employment-related simulations. The work these folks are doing offer a window to what the future may hold.

Brand Games: Specializes in creating video-based gaming to support employment branding initiatives. While it is not creating simulations to be used to evaluate applicants, the work it is doing is some of the most advanced out there when it comes to the combination of hiring and gaming.

The U.S. Military (profiled in the March Journal of Corporate Recruiting Leadership) has been creating some of the most advanced simulation tools out there. Many of these are specific to a particular role and are used for training, but they are also using gaming as part of employment branding exercises.

ETC-easy: It is currently working on the first customer-service gaming simulation ever created. I’ve seen the product (which is still in development) and I think it is a really great step. ETC is currently looking for validation partners for this product, so anyone interested in learning more should contact me directly to learn more.

Hudson Soft’s “Help Wanted/Job Island” video game: This product will be perhaps the most unique job-related game available when it arrives on shelves later this year. The game requires players to work in a variety of virtual jobs in order to discover what they like doing while engaging in video game play. I don’t think that the makers of this game are intending it to be used for any type of actual employment-related activities, but the mere fact that this game exists is very significant in my mind.

As advanced as the various products I have discussed may be, they are still far short of what the simulation of the future will look like. In the future we are going to see products that offer:

• Ability to translate actions on the screen into work-related performance dimensions. This is one of the biggest hurdles we currently face in terms of developing game-like simulations. Once we get over this hurdle, things will never be the same.
• Ability to score complex interactions that occur in an environment with few limitations (aka “Sandbox”). To make simulations realistic, they will need to offer a wide-open range of possibilities for the person engaging in the simulation. Providing the ability to attach scores to an almost limitless range of interactions is a significant hurdle.
• Potential for social networking, linkages, and communication. The movement toward web 2.0 and the ability for increased communication and flow of information will absolutely be a significant part of a good deal of future simulations.
• Continuum between selection and development. The connection between gaming and training is already one that is being cultivated and which clearly makes sense. There is no doubt that games used for selection can also offer opportunities for developing employees.
• Ability to create a data dossier that can follow one through the evolution of their career — say goodbye to the resume. In the future, individuals will be able to present their work history, skills, etc., within the framework of virtual interactions.
• Ability to ensure safeguards against cheating and ability to secure data. As with employment testing today, security and cheating are hot topics and justifiably so. The stakes are getting higher, especially in tough economic times such as those we are currently facing. The more open simulations and testing become, the more important this area will be to their success.

While these things may seem to be a long way,the development of all technology-based solutions is an evolutionary process. While progress may be slow, any movement forward no matter how small the steps is essential. Given the accelerated rate of technological change that is currently in effect, we need to be open to the fact that things we have not thought possible, will indeed be possible.

While its fun to speculate about the future, it is important to understand that simulations are not just a pie-in-the-sky pipe dream. There are things that are now available that can provide excellent ROI for companies looking to make smart, accurate hiring decisions. So, in the short term my advice for companies is to begin thinking about how simulations can be a benefit to hiring. Investment in simulation technology in the here and now can serve as a learning experience that can also have a strong impact on the bottom line. This represents a classic “win-win” situation.

This is a fine strategy for making friends, but absolute nonsense for a recruiter (I had another word in mind, but it would have been politically incorrect.)

After some initial chit-chat, the only interview questions a recruiter or hiring manager should ask are ones that provide trustworthy and reliable data about whether the candidate has the skills for the job.

Canned Questions and Pickled Answers

Why do so many people miss the obvious? Anyone involved in hiring knows there is a big difference between acing an interview and acing a job. Furthermore, as we all know, working in a job we neither like nor have the skills for is a painful experience. I can understand this kind of clueless interview recommendation coming from an inexperienced hiring manager; but, should we accept this advice from someone who either passes himself off as an expert or who recruits for a living? After all, screening applicants based on job qualifications is the recruiter’s job, isn’t it?

What Do You Know and When Did You First Know It?

Asking the right interview questions requires knowing first what to look for. And, if recruiters have to resort to qualification questions like, “What is your greatest accomplishment?” or “How would you describe yourself?” or “What is your greatest strength or weakness?” it’s a sure sign they neither have a clue how to identify specific job skills nor how to measure them. Any determined applicant will rehearse answers to questions like these. Any experienced recruiter knows, at best, they serve as knock-outs.

I once worked with a “professional” recruiter who claimed he had a better way to ask interview questions. He drew close, looked around to see no one was listening, and whispered, “How would your best friend describe you?” (With great difficulty, I choked back a description I really wanted to use to describe him).)

I can just envision the flood of nasty-grams I am about to receive from recruiters in angry disagreement; but, I did not invent this stuff any more than Newton invented gravity. Best-practice interview techniques are supported by thousands of peer-reviewed investigations conducted by hundreds of experts. So, if anyone wants to argue, here is a list of roughly 200 universities. I’m sure they would love to hear your opinions!

Knowing What You Need

If knowing what to look for in an applicant seems so simple, why do so many people get it wrong? For one thing, it’s not as simple as knowing the results you want to achieve. Results do not tell you how a job was done — or even who did it — they are the scores at the end of the game. They do not tell you what the player did, when or why the player did it. Just knowing results leads to assumptions about the skills used to achieve them. You need more; otherwise your assumptions will lead to hiring mistakes. Let’s use Tiger Woods as an example.

Woods’ objective is to use the least amount of strokes to put a little white ball into 18 little holes. These holes are inconveniently located amid trees, sandy pits, hills, ponds, and grassy patches. The total number of strokes is the desired result; but, Woods is only partially in control. Between his first whack and last plop, Woods has to confront temperature, humidity, wind, clubs, lawn maintenance, equipment, other players, onlookers, physical conditions, and a host of other factors out of his control — any of which can affect his score. The same is true of job-holders.

Although we treat other people as if they are in total control of their performance, we reserve the right to make excuses for our own behavior. Psychologists call this fundamental attribution error. That is, you are totally responsible for whatever happens to you … but I am entitled to blame others for whatever happens to me. Attribution error interferes with hiring decisions every time we hear a candidate tell us he was unsuccessful. Fundamental attribution error addresses only one part of the human condition; halo is another.

Humans tend to use snippets of information to make sweeping assumptions about other abilities. This is called the halo/horns effect.

What? You misspelled the word disenfranchise? You must be a complete doddering idiot who needs help tying his shoes!

What? You reduced the overall consumption of paper clips in your last job? You are obviously qualified for our presidential suite!

How often have you heard someone suggest the first two minutes of an interview make or break a candidate? Do you honestly believe someone’s entire career-skill set can be measured in two minutes? The halo/horns effect causes us to make errors both for and against every candidate.

To summarize, there are many insidious forces actively at work whenever applicant, recruiter, or hiring manager meet: silly interview questions; fundamental attribution error; halo and horns; unclear expectations; and, assuming results and skills are related. It’s a mystery why more hiring decisions aren’t disasters!

Systems and Solutions

Think of job performance this way. Every employee has to confront certain kinds of situations. Generically, these situations require a combination of one or more of the following abilities: cognitive ability (e.g., mental horsepower); planning/organization; interpersonal skills; special skills/abilities; and, specific motivational components. It gets confusing when you try to evaluate more factors than these.

For example, if you have been trained in behavioral interviewing, you probably noticed that after four or five questions, you start getting the same answers. Or, if you asked a candidate separate questions about solving a difficult problem, making a tough decision or analyzing data, you begin to hear the same story. That’s because problem solving, decision making, learning, and analysis are often so entwined that it’s difficult if not impossible to separate them. Being impossible to separate means it is almost impossible to measure them individually. Instead, it’s better to look at them as a package called cognitive ability.

Ever hired a psychologist to administer tests to an applicant only to find the report awash in personality factors and character evaluations? Well, unless your psychologist has been trained in how to evaluate job skills, he or she can only do what they were trained to do: provide mental-health evaluations. Evaluating applicants’ mental health takes you right straight into conflict with the Americans with Disabilities Act. All you really want to know is whether the person has the skills for the job.

And another thing: avoid fuzzy concepts like business savvy, budgeting, tough mindedness, or drive to achieve. Fuzzy terms and hiring mistakes go together. One rule of thumb is if you cannot measure a job skill in a few minutes, then it probably is so complex that it cannot be accurately measured until the person is on the job a few months. Take leadership. Has anyone ever seen a “leadership”?

Leadership is the ability to bring a collection of individual skills together at the right place, the right time, and under the right conditions. More often than not, the skills vary with the situation. Sometimes they might require interpersonal ability, sometimes they might require analysis and correct decision making, and sometimes they might require planning. Leadership is not something you can see in a few minutes. It is a result of many things happening over time. Even the traditional leaderless group discussions that so many assessors are so fond of suffer from halo (e.g., extraverted people tend to perform better than introverts).

Interviews as Tests

It helps to understand that every problem has three components: 1) a stimulus; 2) an employee response; and, 3) a result. If you have tracked this article so far, you should understand that learning all three components are important to knowing whether the applicant has the job skills you need.

Vendors who sell behavioral interviewing programs often train participants on how to ask for background information, to probe specifically for what the candidate did or said, and to verify the results. These activities go by many acronyms (BEI, BBC, STAR, ABC, and so forth); however, regardless of the term used, the most important goal in behavior interviews is gathering sufficient information about all three components so applicant faking is minimized and specific applicant skills are clarified. Accuracy leads to better hiring decisions.

Simulations, pencil and paper tests, case studies, planning exercises, and the like, follow the same stimulus-response-result pattern. The main difference is you control the stimulus and know the result you expect. That improves accuracy.

Conclusion

The recruiting field is awash in nonsense and bad advice. This leads organizations to hire too many wrong people and reject too many right ones. Experts estimate this cost ranges anywhere between 20% to 50% of base salary. Being passionate about a hiring methodology and knowing it is valid and reliable are not the same thing. If a product or report seems off-target, ask to see studies proving scores actually predict job performance, look at the vendor’s professional credentials to see if they belong to the right associations (SIOP), or simply ask if the product was specifically developed to predict job performance. A vendor making claims that sound too good to be true are no different than the emails announcing your lottery winnings. A little common sense and education makes a world of difference.

Lesson #1 -“Minor colleges” produce some of the best players on Super Bowl teams.

It’s clear from examining the player rosters that most Super Bowl players don’t come from powerhouse football colleges. Some examples include:

• The Star Players. The four most notable star players in the game all come from non-football powers. For Arizona, star quarterback Kurt Warner came from Northern Iowa – Burlington and star wide receiver Larry Fitzgerald came from the University of Pittsburgh. For Pittsburgh, starting quarterback Ben Rothlisberger came from Miami of Ohio and running back Willie Parker came from North Carolina. Not a single one of these players former university teams made it into the Associated Press Top 25 rankings this year.
• The Remaining Players. Of the 112 active players on the final rosters of the combined teams, only four came from this year’s top college teams that are perennial powerhouses (i.e. Florida, Oklahoma, and USC).
• The Arizona Cardinals roster successfully recruited players from non-powerhouse teams like Louisiana-Lafayette, Kansas State, Richmond, Northern Iowa, Hawaii, Brown, Delaware, Fresno State, Tennessee State, Trinity University, UC Davis, Northern Iowa, and of course, Clarion!
• The world champion Pittsburgh Steelers roster includes players from such non-powerhouse university teams like Hofstra, Clemson, Kent State, Marshall, Tulane, Southern Mississippi, TCU, Northern Colorado, Syracuse, Rutgers, Indiana (of Pa.), La.-Lafayette, and perennial powerhouse…Tiffin.

Lesson #2 –“Experience” isn’t required to become a Super Bowl head coach.

At least this year, previous experience as a successful head coach isn’t a requirement for getting your team to the Super Bowl.

The best head coaches aren’t always the most experienced. You can’t say the coaches of either team this year are experienced, veteran head coaches. Neither has been a head coach at a Super Bowl before. Both would have to be considered as “inexperienced” head coaches (both are only in their second year of being a head coach anywhere in the NFL), and both are relatively young.

Mike Tomlin of Pittsburgh, the NFL Coach of the year, is only 36 years old (the youngest Super Bowl coach ever) and Arizona’s Ken Whisenhunt is only 44.

Lesson #3 – “Recruiting/draft systems” still need continuous improvement.

No organization puts more resources into recruiting than an NFL team, yet even their vaunted efforts can produce some major errors and omissions. There are numerous stories of how number one overall draft picks failed to excel in the NFL but there are also major omissions in the recruiting process where top performers went “undrafted.”

Even the best recruiting and screening systems can be improved upon in order to find a “non-obvious” talent.

For example, Pittsburgh’s star running back Willie Parker, a two-time Pro Bowl selection and his teammate James Harrison, the 2008 NFL defensive player of the year, were both undrafted. On Arizona’s team, Pro bowl quarterback Kurt Warner and his center, Lyle Sendlein, were both undrafted.

Lesson #4 – It takes organization-wide excellence to make the Super Bowl.

The teams with the most star players don’t automatically make it all the way to the Super Bowl.

It takes more than star players to win championships. The team with the most star players (those selected for the Pro Bowl), the New York Jets, led all teams with seven selections but they didn’t even make the playoffs.

Three teams (the Giants, the Vikings, and the Titans) each had six star players selected to the Pro Bowl. Having six stars certainly helped each team to get into the playoffs. But you won’t see them in the Super Bowl, because it takes excellence throughout their entire organization and great coaching, in addition to great players, to win it all in your conference.

Lesson #5 – Don’t stereotype top performers.

It’s easy to reject candidates “out of hand” because they don’t fit our mental stereotypes of what top performer should look like. Always use “on-the-job” performance to select top talent instead of broad stereotypes.

Be careful who you write off as being “over the hill.” Even though everyone knows pro football is a young man’s game, there are certainly enough exceptions to that rule to make you think twice before generalizing about age limits on star talent.

For example, Kurt Warner is both a Super Bowl and a Pro bowl starting quarterback and he accomplished both feats at the ripe old age of 37. Incidentally, he clearly beat out Arizona’s other well-known quarterback Matt Leinart, the much younger third-year phonon from USC.

Other Pro Bowl “senior citizen stars” selected this year include 39-year-old Jets quarterback Brett Favre, 44-year-old John Carney (the oldest Pro Bowler ever), and 42-year-old Jeff Feagles both from the Giants.

Clearly, top talent comes in a variety of ages.

Lesson #6 – Consider corporate alumni for rehiring.

It’s easy to overlook talent that we’ve previously rejected.

Keep in touch with your former employees. Pittsburgh Steelers long-snapper Jared Retkofsky was fired (cut from the team) by the Pittsburgh organization three different times. Recently, he was working as a furniture mover before being rehired by the team to fill a sudden need. This success story demonstrates that organizations should keep an eye on their former employees and then to consider them for re-hiring as a “boomerang.”

Lesson #7 – Prioritize your positions and business units.

NFL teams excel at prioritizing both individual positions and units.

It has been proven statistically that “defense wins super Bowls,” so most teams that are serious about getting to the Super Bowl prioritize their recruiting to ensure that they have the strongest defense. This year’s likely Super Bowl winner, Pittsburgh, has both the #1 ranked rushing and passing defense. Likewise, businesses also need to learn that not all functions, departments, and business units have the same impact on winning and profits.

Not a single corporation that I’m aware of prioritizes positions as effectively as NFL teams. You can clearly tell what positions are “mission-critical” on an NFL team by identifying the positions that they draft first, the average amount that they pay for that position, and the number of backups for that position that they carry on their roster.

Most teams focused their recruiting and development efforts on four key positions, quarterbacks, pass rushers, receivers, and running backs. Other positions are rated as low impact.

For example, an average punter might make $200,000 a year, they would never be drafted in the first round and the team would only carry one punter (with no backup).

In direct contrast, the high-priority position of quarterback will be paid millions, the position would be a top draft target, and there would be two or even three backups kept on the roster to fill in for injuries and to allow time for development.

Corporations need to also prioritize their recruitment and development efforts and resources on the 20% of the positions that are mission-critical and thus they make the most impact on innovation, competitive advantage, revenue, and profit.

Lesson #8 – “The ability to handle pressure” is critical to make it to and win a Super Bowl.

Everyone knows that it takes a combination of both “A” and “B” players in order to win championships, but not everyone accurately defines what an “A” player really looks like.

Define excellence to include the ability to handle pressure. The NFL provides numerous great examples of how players that perform well during the regular season suddenly “unravel” under the pressure of the playoffs and the Super Bowl. Teams that fail to win the Super Bowl one year proactively go out and recruit individual players who not only can compete head-to-head against last year’s Super Bowl winners but who have also demonstrated that they can handle the “pressure of “big games.”

For example, running back Efrain James for Arizona was a mediocre player during the regular season but in the playoffs he was given the starting assignment. He has excelled during the playoffs as a result of his extensive playoff experience. Other players like Kurt Warner and Larry Fitzgerald have clearly demonstrated their ability to “step up” and move to the “next level” under playoff pressure.

Corporations need to also develop mechanisms for recruiting and assessing their employees to see which ones best handle the pressure and thus can perform at the “next level” when the corporation needs extraordinary performance.

Lesson # 9 – It takes a competitive analysis to win championships.

Many in HR and recruiting look at their firms’ needs in relative isolation. In the NFL, however, recruiting and game plans are developed after analyzing the strengths and weaknesses of the opposing team.

Recruit and organize to develop a competitive advantage. During the regular season, but especially during the playoffs, team managers conduct “side-by-side” analysis in order to identify and develop matchups, which provide their team with a winning edge.

Recruiting and player selection in key positions is done in such a manner as to provide “an edge” over  competitors. The “plays” of competitors are also analyzed to a painful degree in order to identify weaknesses that your team can exploit.

Every team develops and modifies its plays in order to take advantage of the head-to-head player matchups where you have a higher probability of coming out with a big play. Teams consciously avoid trading or losing key players to teams that they frequently compete against because that could lessen their chances of winning against that competition.

Corporate recruiting managers need to develop the same competitive advantage mentality and periodically “chart” areas where recruiting can give your firm a significant competitive advantage over your direct competitors.

Lesson #10 – It takes a performance culture to win consistently.

Some sports teams win consistently over the years, even though their players change at a rate of 20% or more per year. For example, the LA Lakers have dominated their opponents for years, while the LA Clippers, who play in the same city and building and recruit from the same sources, are perennial losers.

Build a performance culture. The best teams in sports are laser focused on becoming and staying number one. In order to do that, they develop a “performance culture” that makes winning everything.

As Vince Lombardi once said, “Winning isn’t everything…it’s the only thing!”

A performance culture in any environment focuses on recruiting, retaining, developing, differentiating rewards, and making assignments based on performance data and results. Because everyone throughout the organization has the expectation of winning, performance permeates every management aspect of the firm.

In the NFL, teams like the Steelers, the Cowboys, the Colts, the Giants, and Patriots consistently excel during the regular season and later win championships because they have all developed performance cultures.

The players leave their current team at nearly 20% per year, the coaches change also but the performance culture approach is a constant. Unfortunately, many HR and recruiting departments are more focused on equal treatment and avoiding criticism.

Establishing and maintaining a performance culture is hard work and it certainly gathers criticism. As a result, few in HR are willing to take the heat and the risk to their own job security that comes with transparency and basing everything on performance.

Final Thoughts

By the time you read this, the Super Bowl will be over and its importance will begin to wane.

However, if you’re smart, you’ll avoid the criticism that invariably comes from HR people (i.e., “I hate sports analogies.”)

Instead, study the most competitive of all endeavors outside of war (the Super Bowl) in order to learn many valuable lessons about people management. It is my contention that the recent catastrophic failures of mortgage firms, Wall Street financial institutions, and other corporations could never have occurred under the glaring criticism, transparency, metrics, and the performance cultures that are necessary in order to get teams to the Super Bowl. The NFL model is a good one to study and copy.

This well-written and researched piece is somewhat groundbreaking in that it is the first article in a mainstream media outlet to provide evidence of cheating on employment tests. Those of us in the testing industry have always been concerned with the security of our tests and have taken a variety of precautions to defend against it, but this is the first time I have ever read actual evidence that documents the existing of cheating.

As a testing expert and someone who has a high degree of familiarity with Unicru/Kronos (the company whose tests are the subject of the piece), I figured it would make sense for me to weigh in on this important article. Here are some thoughts about the article and the issues it raises:

• Overall, the article should definitely serve as a wake up call to the testing industry that times are changing; but, I do not think that the problems it identifies should spell doom and gloom for employment testing. Security is an important issue for any industry that conducts business online (banking, shopping, etc). However, dealing with security issues has not served as a barrier to progress, but rather an evolutionary force encouraging adaptation and prevention.
• The case of Unicru/Kronos is somewhat unique in the testing industry because it tests such a huge volume of applicants each year. It is a safe bet to say that no other testing company does the volume that Unicru/Kronos does. Therefore I would be very surprised if the same systematic and premeditated cheating on personality-type tests was occurring with the tests of any other vendors. Differences between the way the Unicru/Kronos tests and those of other companies would make cross-test cheating a complete disaster for anyone who tried.
• Because the Unicru/Kronos test is used to provide a way to provide efficiency to high-volume hiring, its results are often used to make “go — no go” decisions. While it is not always advised to use a test as the only piece of information to eliminate an applicant from consideration, high-volume hiring often necessitates it. How else can one have any chance of evaluating 1,000 applications for 10 or 20 openings? Considering the current state of unemployment and the sheer number of folks involved, high-volume hiring starts to show some similarities to standardized testing. That is, lots of applicants and high stakes. While cheating on high stakes tests is not too common in the U.S., it is a significant problem in other countries such as China and India. I have had a good deal of experience implementing standardized testing in these countries and I can say with authority that we do have a variety of techniques that can be employed as preventative measures. As with any type of security, multiple layers provide the best defense. We as testing professionals have the ability to use item banks, alternate versions of items, parallel forms, and computer adaptive testing to minimize the chance that a reconstructed answer key will actually match a test.
• Consider the type of test in question when evaluating the potential for cheating and the impact it may have. The WSJ article claims the Unicru/Kronos test is a personality test. This is partially true, as it does have some strong ties to traditional personality tests. However, the most common personality tests do not actually have right or wrong answers for the individual questions. Rather, the scoring for the questions usually varies quite a bit relative to the specific job in which the test is being used (if the test has been properly configured). While the science behind Unicru/Kronos’ tests is beyond reproach and is actually very innovative, their tests are still a good bit different from those of the rest of the herd. Unicru/Kronos’ unique formula works for them and their tests are highly predictive when compared to many others out there. Why does this matter? I won’t go into the mechanics of it, but suffice it to say that it would be much harder to identify right or wrong answers on a more traditional personality test. Most traditional personality tests also have faking scales built in to identify those applicants who are answering in a socially desirable pattern. Figuring out the scoring algorithms that account for this makes cheating in any meaningful way even more difficult.
• Traditional personality tests by themselves really aren’t the best predictors of job performance. There is a ton of research to support the fact that cognitive ability tests are much better across the board at predicting job performance. These types of tests are much more susceptible to cheating, and test vendors are very aware of this fact. In many cases, these types of tests are administered only in proctored situations. If I were to really worry about cheating, it is cheating on cognitive tests that would be my major concern.
• Another interesting wrinkle in this whole thing is the fact that job applicants are actually resorting to cheating the companies that they wish to potentially work for. To me this is a clear indicator of the need for companies to take a good long look at not only the messaging around the test itself, but also at the manner in which the applicant is treated throughout the entire hiring process. Of course it is unrealistic to think that we will ever be able to keep all applicants from cheating, but I do believe that the more applicant-friendly the hiring process, the less likely applicants will be to cheat. Time and again I have seen the testing portion of the application process handled in a cold and unfriendly manner, with very little information about the reason for the test being provided to the candidate. This type of thing makes it much easier for a candidate to feel fine about breaking the rules. There should be messaging to candidates about the fact that testing is beneficial to them because finding a good fit between applicant and job is beneficial to all parties involved. No one likes to work in an environment where they do not fit in or where they do not have the competencies required to succeed.
• The bottom line here is that tests such as those offered by Unicru/Kronos do offer tremendous value. We I/O psychologists have collected more than enough evidence to demonstrate that testing, when done right, offers a huge advantage over more informal and unstructured hiring practices such as resume reviews and unstructured interviews. In the case of high volume hiring, the ability to provide an automated tool for screening out applicants is tremendously useful. Are these methods perfect? Absolutely not, but then again very little in this world is. Error and noise are accepted parts of the testing process. The goal of test providers and hiring professionals is to mitigate these to the best of their ability. In high-volume testing, cheaters make up a very small percentage of the total population of job applicants, making cheating just another source of noise that we must work on minimizing.

The issue of cheating is something that warrants a good deal of attention but one that should not stop our progress in the quest for better and more accurate hiring procedure. The issues discussed in this article serve to strengthen my stance that we need to be working on a new generation of hiring tools that are simulation-based. The more the testing part of the hiring process relies on scenario-based activities that mimic the work environment and the tasks that must be completed within it, the more realistic and engaging the application process will be for applicants and the harder it will be for someone to cheat. Of course we are a long way away from this goal at the present time, but we will get there eventually. Until that time, those using testing should be sure to make test security a critical part of the design and implementation of the testing process, while also bending over backwards to make sure the candidate is treated just as any valued customer of the organization would be.

The most significant change I expect to see in pre-employment assessments in 2009 is a slowing of uptake as some organizations slow their hiring to a trickle or cut things seen as non-essential (i.e., assessment) from their budgets. I am not going to miss the opportunity to suggest that cuts to the budget for assessment are unwarranted because a well-designed assessment program can provide ROI — no matter what the economic context may be. My thoughts about a slight slowdown in the use of assessment are just speculation; it will be interesting to see if hard data such as that provided by our 6th Annual Screening and Assessment Usage Survey supports this speculation. If you have not taken the survey yet, I encourage you to do so. It only takes about 10 minutes and the information you provide is very valuable.

Despite the possibility of a slowdown in the purchase of assessments, there are a number of trends, nine in all, that will help mark 2009.

Legal compliance will become an even bigger deal. This next year is going to be a monumental one for U.S. politics. No matter who you voted for, the fact that change is coming to our government is a reality. It will be interesting to see what the impact of the new regime will be on labor-related issues. My gut tells me that the reform agenda that will be prevalent in Washington may mean more aggressive enforcement of EEOC and OFCCP regulations related to fairness in the hiring process. The often-heard opinion that using assessment places one at greater legal risk is simply not true. An assessment program that has been properly implemented and evaluated can actually provide increased legal compliance. Score one for assessment in 2009!

The concept of “embedded assessment” continues to evolve. Even a diehard believer in the value of assessments such as myself should be willing to admit that assessment data is not the only piece of information that is useful when making hiring decisions. Good hiring is the result of an informed decision-making process. As technology that helps support this process approach continues to be developed, assessment will continue to be folded into the mix as a key ingredient. As time goes on, an increasing number of hiring products will include assessment in such a capacity that it will be relatively transparent, but will provide data points that will help to support decision-making. This also means that the traditional assessment firm will need to be ready to play well with others as their survival may depend on it.

More assessments available transactionally. One of the benefits that come with a decade of Internet testing is an unprecedented amount of data. This information is being used to help create off-the-shelf tests that are more accurate than ever. In the past, an off-the-shelf test would be extremely general and might have missed the mark when it came to measuring all of the things required on the job. The new generation of off-the-shelf tests leverages data to offer a much more complete measurement model. This trend will continue as an increasing number of companies offer tests that can be used transactionally, with little or no up-front work.

Hello, middle market. Trends 2 and 3 are definitely related to a continued push toward offering easy-to-use assessment-related products to organizations that previously may not have considered them. Productization and the creation of more accurate off-the-shelf tools will allow for a whole new type of product. There is definitely untapped revenue potential in this market segment.

Assessment content stays relatively stable. While the technology that is wrapped around assessments has changed significantly in the past decade, the actual content of the assessments themselves has not changed much. We have done a great job understanding which individual items work better than others, but assessments still require applicants to click radio buttons that correspond to personality-type items. This trend will continue into 2009. Radically different items or assessment modalities are still a ways off.

Simulations continue to simmer. Despite the stagnation in assessment content, I have seen some encouraging progress in the world of simulations. This progress will continue as new technologies become available. The future of technology and hiring lies in simulations. It is going to take some time until we have all the pieces in place to create what will be the first of a new generation of hiring tools. In the meantime the interest in simulations and the technology that is being explored are encouraging.

Continued links to development. The rise of interest in talent management platforms and the natural continuity between pre-employment assessment and performance management/development have continued to increase the links between pre-employment assessment programs and continued employee development. This movement will continue in 2009 as an increasing number of product offerings leverage pre-assessment data as an important input to the employee lifecycle.

Evaluation still not a priority. It seems like I bring this one up every year. Probably the one biggest point that continues to frustrate me is the fact that the majority of assessment users do not take the time to properly evaluate the impact of assessment on important business outcomes. I feel our current economic situation may serve to magnify this point, when sadly, it should do the exact opposite. Yes, assessment works, and the value it can provide is even more important during times when every dollar is scrutinized.

Increased self-awareness. Once we are looking back at these tough times in the rearview mirror, I think all of us are going to have learned some valuable lessons. Our entire industry is now on the hook to continue to add value, and we are all going to have to get resourceful, to adapt, and to learn about ourselves. The learnings we take away will hopefully help us form lasting habits that will help us all to emerge with a new mindset about what is valuable.

2009 promises to be an interesting time in the history of pre-employment assessment. I believe that the quality and variety of options will continue to grow and that technology will continue to allow us to do more with less. The timing for this trend could not be better and I hope that companies can begin to look at assessment as a profit center instead of viewing it as a costly drain on resources.