Managerial-Level Assessments: Past, Present, and Future

Article and research by Charles Handler and Mark C. Healy

Over the past decade, there has been a significant increase in the number of assessment tools available for use in both hiring and developing employees at all levels of the organization.

By far the most common use of such tools has been for selecting hourly and technical workers. This makes sense, considering that the bulk of many organizations’ employees reside at this level and because success at many of these jobs usually boils down to a few key attributes such as:

How fast can the person learn the job?
Is the person going to steal, ditch work, etc?
Does this person “get” customer service and will they help customers acquire what they need?
Can this person communicate effectively?

As organizations gain more experience with measuring applicant qualifications, it has become common to find online assessment within the hiring process for hourly and skilled technical positions. Broader sales and marketing efforts by vendors and consultants have led to dramatically increased sales of these products. But what about managers and executives?

As we found in our latest survey of online prescreening and assessment trends, having candidates for supervisory roles (from 1st level all the way to the C-suite) complete questionnaires, tests, and simulations online is common, but there still seems to be a lack of assessment at these levels, especially when compared to the use of assessment for hourly jobs. Nonetheless, adoption of these tools has grown over the last 10 years and is likely to continue to do so over the next decade.

While the basic model for success with predictive assessments still holds, there are some key differences between successful assessment for managerial and executive-level positions:

Managerial jobs require a slate of skills that are not often required for success at simpler jobs. These include things like leadership, business acumen, managing performance, strategic decision-making, conflict resolution, etc.
A mistake at this level of hiring can be much more costly than for an hourly hire, mostly because managers are responsible for potentially hundreds of people and millions in both costs and revenue acquisition.
Managerial hiring has a tighter integration with ongoing developmental strategy and activities as organizations look to maximize their investment in an individual by understanding and developing potential from within.
There is a broad range of job complexity once you hit supervisory levels. The selection process for a front-line leader can differ vastly from that used for executive roles. Executives often complete day-long assessment centers while first-level supervisors are more likely to be presented with a simple in-basket or situational judgment exercise, which is scored automatically.
Feedback becomes a more integral part of the assessment process. Pre-employment assessment for hourly jobs almost never provides candidates with any feedback at all.
When making promotional or hiring decisions about managers, assessment requires a “whole is greater than the sum of its parts” approach. Trained assessors (often both external and internal members of the organization) make ratings based on a variety of data.

The above differences are clearly reflected in the past, present, and future of managerial-level hiring and development tools. Let’s start with the recent past.

The Past

Other than interviews, technical knowledge, and previous job experience, best-practice managerial assessment over the past 50 years has typically involved mimicry of the most important tasks faced by supervisors of both people and process.

Typical tools include:

In-basket simulations. Also known as an “in-tray” outside of the U.S., these often-intense exercises imitate the administrative work of a manager in a fictional company. The traditional version of this tool presents a candidate with a stack of memos, a calendar, process descriptions, employee files, and other paperwork. The task is to respond to issues contained in the memos, with attention to the supplemental information about the company and its employees.
Role-plays. Traditionally live and in-person, the candidate plays the role of a manager dealing with a poor-performing employee, dissatisfied customer, or other interpersonal situation. The interaction is scored (usually according to competency ratings) live or from a video or audio recording.
Situational judgment. Usually in the form of multiple-choice questions, these assessments present a real-life work situation and ask the candidate to choose among alternative actions. Sometimes, multiple responses are correct, but each response is worth a different number of points.
Business case/problem analysis. Some assessment systems require the potential manager to read over an operational or strategic problem and render an opinion or plan. Some include mathematical calculations and necessitate clear writing. Others essentially mimic a work-related intelligence test.
Formal presentations. Particularly in the case of hiring at the director-level and above, and especially useful for selecting sales executives, the candidate is required to make a presentation about a product, strategy, or current issue. Aspects of the talk, including organization, clarity, and technical knowledge, are rated by assessors.
Testing. There are several standard assessments that seem to be commonly used to provide insight into the personality and cognitive ability of managerial candidates. In many cases, the activities above get at the same basic constructs, but we have found that the higher the job, the more common it is to find testing as part of the assessment-center experience.

For higher-level jobs, the above ingredients are often combined into a one-half to three-day offsite often called an “assessment center.” Assessment centers are very expensive due to the need to have trained, qualified personnel to execute and score the exercises.

So, as the level of job gets lower, there is a decrease in the complexity of the assessment process.

The key factor uniting all of these diverse tools into an assessment center methodology is that they attempt to simulate managerial tasks and decisions in a real work environment.

Essentially a “test,” these highly regarded instruments ask a candidate to think and behave like a competent boss, instead of simply demonstrating a general trait (e.g., conscientiousness), skill (e.g., Intermediate Java), or ability (e.g., logical reasoning).

Moreover, these tools force the hand of more reliable, standardized data in a selection or promotional process, removing a worthwhile amount of subjectivity from key personnel decisions. There is even scientific evidence of decreased subjective bias on the part of hiring managers when compared with unstructured, ad-hoc interviews.

Another factor that underlies the use of managerial simulations is that they require some budget, effort, and dedication on the part of the HR and line organization, as does any worthwhile human capital initiative. Traditionally, these are expensive, somewhat cumbersome tools that are brought into a company by individuals who are just learning how to use them.

But simulations that realistically judge a candidate’s managerial prowess soon prove themselves to be invaluable to leaders making placement decisions.

Traditional managerial assessments have also demonstrated greater hiring accuracy. They’re more appealing to job candidates than personality inventories and job applications. They have a real relationship with future performance on the job when compared to other hiring tools. They are usually in the form of a statistical correlation with a measure of performance as a boss.

Decades of research in industrial/organizational psychology stands firmly behind these tools.

The Present

So what is the state of managerial hiring tools today? We contacted a wide range of assessment vendors to chat with them about the tools they offer for the selection of managerial to executive-level jobs.

Today’s market for these tools provides a continuum of options ranging from the old school to some thought-provoking new ideas. Here is a summary:

Most assessment vendors and consultants understand that using the same content for lower-level jobs is not an acceptable strategy. Testing and measuring simple competencies does not account for the differences in what it takes to be a good entry-level worker and what it takes to be a leader/manager. We have seen an increasing number of vendors offering new products that leverage traditional approaches to making selection decisions for more complex jobs (i.e., assessment centers, in-baskets, etc.).
Competency models, once thought of as a temporary fad, are now well-entrenched and help support linkages between employee selection and development. While competencies are all the rage for employee selection of all types, the link between selection and development tools for managerial-level and executive jobs is much stronger than it is for hourly jobs. Most managerial-level selection tools can also be used for development (with scores indicating strengths and areas to work on), and almost every provider offers a version tuned for both. This makes sense, as this kind of assessment is commonly used for identifying and grooming high-potential employees and for making internal promotional decisions.
The in-basket is alive and well and living on the Internet. Most vendors offer an online in-basket that provides “assesses” with a realistic representation of a range of daily tasks required of someone at the appropriate job level. A simulation of an email in-box, coupled with organizational charts, employee histories, and process descriptions is most typical here, replacing the giant stack of memos with an MS Outlook-style in-box full of action items and hyperlinks supplemental information.
Technology is being used mostly for delivery and administrative functions. There are many folks out there who are using the same basic content and format as has been used for decades. The biggest difference is that technology has been used to make the delivery of the content easier and to help make the tasks of scoring and report writing simpler as well.
Online roleplay simulations are slowly starting to appear. We did see a decent number of products that provide the ability to use avatars and present assessees with more complex data and problems as part of the assessment experience. However, these are still well in the minority. Some of the technologies that will provide the true next generation of simulations do not yet exist. Forward progress in this area will be slow over the next few years.
The more complex the job level, the higher the level of touch. Most of the standardized products we saw for the lower echelons of management use automatic scoring built into the product, whereas products for more complex and higher-level jobs use trained assessors to do scoring and feedback and act as role players.
There is very little ability to customize content unless you have deep pockets. Most of the tools we saw did not easily support high levels of customization, though some firms sell versions of tools that differ by setting (e.g., healthcare, manufacturing, general office). However, there are different types of assessments for different types of jobs, and the tasks and competencies we saw are general for most leadership situations. As with assessment for individual contributors, you should analyze the job and carefully match assessment content to the actual work performed.
There has been very little work done investigating the ROI of the online versions of traditional assessments. We still believe that asking someone to perform key aspects of a job is still the best way to evaluate how well they will do in their new role. Anecdotally, there are thousands who will testify to the impact of a good assessment program for selection and development of managers and executives.

The Future

So what does the future of managerial-level assessments hold? Here are a few ideas:

More realism via simulations, particularly using avatars and online representation of complex, real-work environments.
More artificial intelligence will be used in scoring, with technology that partially analyzes a written or spoken response by a candidate and, at the very least, speeds up the scoring and rating of completed materials.
More data to support the value of managerial assessments.
More diverse product offerings available as uptake increases in this hot area.
Tighter ties with competency models will be created so that assessment and development will be linked more closely.

Conclusion

Today, best-practice managerial assessment involves a combination of the tried-and-true approaches of the past and the innovative media of the present.

For the organization that’s serious about leadership, managerial assessment tools (especially in-baskets and role plays) represent the key to truly separating those who aren’t qualified to influence one’s workforce from those who will aid revenue, retention, and overall performance.

We expect that as technology evolves, so will the automation of administrative- and scoring-related tasks and the level of realism associated with these assessments.