Is Behavioral Event Interviewing Based on Bad Science?

A few in the I/O psychology scientific community have lambasted me on these pages for suggesting that behavioral event interviewing (BEI) might not be all that it’s cracked up to be. Their comments seem akin to climatologists who discredit anyone who suggests an alternate cause of global warming. To stir the pot even further, we’re holding a public debate on this topic on March 25, 2010, with a bunch of ERE authors (Dr. John Sullivan, Dr. Charles Handler), a BEI luminary Dr. Tom Janz, and your humble recruiter/reporter. This will be a slugfest to finish going all 15 rounds, so you won’t want to miss the excitement.

I’ll lay out my hand and concerns in this article. We’ll address them in the upcoming debate. To get started here are a few of the big problems I have with BEI:

Is a structured BEI all that predictive? According to the oft-cited Schmidt and Hunter Validity and Utility of Selection Methods study, the correlation between a structured interview and on-the-job performance is .51, meaning only 25% (squaring the correlation coefficient) of the person’s performance can be predicted by the interview. This means that 75% is unexplained. Wouldn’t any structured interview give the same results? Also, what’s the predictive value of an unstructured BEI?
Past performance, not past behavior, is the best predictor of future performance. The same study clearly refers to past performance, not past behavior, as the best predictor of on-the-job success. Where’s the research that suggests that past behavior is superior to past performance? As a case in point, I conducted a virtual performance-based interview and assessment comparing Obama vs. McCain before the 2008 election using our 10-Factor Talent Scorecard. If you check it out, the predictions were right on the mark. Using behavior as the criteria, the predictions would have been lopsided.
Why is the criteria used to promote someone from within more predictive than hiring someone from the outside? It seems logical that the methodology companies used to promote those who are successful, which is based on their performance, should be applied to those hired from the outside. If so, this would mean emphasizing a track record of past performance doing comparable work in comparable situations combined with the person successfully taking on bigger roles with less experience. This seems like it would be a better predictor than using behaviors and KSAs.
BEI misses the forest for the trees. The big goal here is to maximize quality of hire, not conduct accurate assessments. While a professional interview and accurate assessment is part of this, more important is having a pool of highly qualified prospects who are willing to go through the assessment process and accept a fair offer of employment if given. This requires great sourcing, great recruiting skills, and strong negotiation skills, plus managers who are strong leaders who can attract top people to work on their team. I haven’t seen any science that looks at hiring from this end-to-end perspective.
If no one is in the forest, can you hear a tree fall? This is a pretty weak analogy, but the point is if no one uses the BEI properly, how can you consider it useful? Most managers find it too clinical, candidates can practice ahead of time, and the best candidates are turned off by it. Plus lack of enforcement and uniformity weakens the pretty weak predictive value even further.
The guidance on making the assessment seems to be limited. What’s a good answer? I’ve looked at dozens of BEI rating sheets and each one is relatively subjective. Statistical process control techniques would suggest that wide variances on any factor are indicative of a process that’s out of control. SPC is a valid scientific technique used in six sigma, but seems to be ignored by those in the I/O psychology community.
Where is the scientific evidence that companies that use BEI outperform their peers? Wouldn’t this be the big Kahuna? In Jim Collin’s Good to Great, there is no indication that BEI was why they hired stronger people. Could it be that there is something else, other than accurate assessments, that drive quality of hire?
There are other techniques that could increase accuracy and improve quality of hire. In Hire With Your Head, I make a strong case that a mashup consisting of a list of pre-hire performance objectives, a few in-depth performance-based interviewing questions, an evidence-based ranking system, strong recruiting and sourcing skills, plus involved hiring managers is the key to maximizing quality of hire. Why don’t the I/O psychologists seek out better techniques, rather than relying on outdated less-reliable methods? One way would be to model all of the managers who consistently hire great people and use this as the framework of a new and better process.
BEI is counterproductive by eliminating the top half from consideration. Since the BEI process assumes the person has 100% or more of the experience required, it eliminates those high-potential people who get more done with less experience from consideration. The best people are looking for a career move involving job stretch, learning, and growth. By forcing everyone through the same funnel, some of the best people voluntarily opt out early, since they find the process demeaning, clinical, and one-directional. By inadvertent default then, the only people considered are those with all of the basic qualifications and those looking for a lateral transfer — aka, the bottom-half.
The BEI is logically flawed. A correlation between two factors doesn’t mean that one is the cause of the other. For example, just because someone has all of the qualifications and behaviors, doesn’t mean the person will be a top performer, even if all top performers have the same qualifications and behaviors. This is similar to the logical “asserting the consequent” argument. Clearly we’d all agree that there is a high correlation between the number of troops required to win a big battle, but having more troops in the field doesn’t mean they’re going to cause the size of the battle. While having the behaviors might necessary, it’s certainly not sufficient. The relationship between the manager might be a problem. The person might not be motivated to do the work, even if competent. The person might not fit with the team, or company culture, or might not want to work with less-than-current technology. These factors, among others, represents the 75% not covered by the BEI.

Now all of this might be the ramblings of an old-line recruiter who has been in the field too long. On the other hand, maybe the scientists never had to close a top performer for a troubled company with limited funding, and then guarantee the person would actually deliver top-notch performance for at least a year. Maybe they should try to do this and then modify their science accordingly. If they do, I suspect they’ll come to the same conclusion that BEI doesn’t improve quality of hire, and in many cases actually causes it to decline.