Receive daily articles & headlines each day in your inbox with your free ERE Daily Subscription.

Not logged in. [log in or register]

Why Interviews Are a Waste of Time

by Sep 27, 2011, 5:38 am ET

Recruiters and hiring managers love interviews. I have never been sure why that’s the case, but it seems to satisfy a human need for power and control. An interviewer has power to recommend for a job or not. Sometimes an interviewer has the power to actually make the hiring decision, and by holding a person’s economic future and career success in your hands, you can feel very powerful.

So anyone wielding such a powerful tool should be certain of its validity and of their skill in using it. The EEOC considers the interview to be a selection test, and requires that it be validated before use. Yet, I would guesstimate that few interviews are validated at all, and the ones that are may not be delivered consistently or by a competent, trained interviewer.

Research has consistently shown that the typical unstructured interview is pretty unreliable. It does not consistently ensure that the most qualified person gets a job or that the person will perform any better than another candidate chosen with less care. In all the studies that I have looked at, the validity of choosing candidates by only using an unstructured interview process is about the same as simply picking someone at random.

Interviews are rarely done consistently from interviewer to interviewer or from candidate to candidate. Yet, we typically consider all the interview inputs for a candidate as if they were done in the same way. Therefore we are comparing apples to oranges, and the hairs we split and the time we spend agonizing over a small detail or a particular answer to an interview question is wasted.

No wonder that candidates often roll their eyes at the absurdity of the interview process.

Assessing candidates is highly subjective, and is based on whatever assumptions (prejudices) the recruiter or hiring manager has, their mood, and the chemistry between them and the candidate. Even factors such as physical appearance, tone of voice, or time of day can impact the interview.

I often ask recruiters to think about what would happen if they selected two candidates for a job who each had the same qualifications and who had known the questions that were going to be asked and had prepared the same answers. If another recruiter interviewed them, would they both receive the same score on the interview, as they should?

DDI and other respected selection organizations offer excellent tools to improve the reliability and validity of interviews, and everyone who does interviewing should be trained in these methods and use them consistently. These well-constructed, validated, and structured interviews given by a trained interviewer can increase reliability by a significant amount, yet they are still only a little more reliable than simply picking a “winner” at random from your final pool.

Carefully constructed interviews, where the questions are directly related to measureable skills, competencies, or past experiences, take a lot of time to prepare and, to be most effective, have to be delivered in a similar way (ideally exactly the same way) to each candidate for the job. This alone would eliminate most of the interviews I have had personally. From my experience as a recruiter and candidate, these rambling, unstructured interviews were far more common. The interviewer ranged over a wide variety of topics, dipping into my resume here and there to ask a question or ask me to validate something they already expected and wanted to hear. In most cases, I could manipulate the interview in subtle ways to make sure my strengths were showcased. In other words, a sophisticated candidate can game the system in many ways to tilt the deck in their favor.

Another Way

On the other hand, there are many tools available to recruiters that could improve their ability to reliably select and recommend candidates that have the depth of skill and aptitude the positions calls for. These tools can save endless amounts of time and free up a recruiter to spend more time sourcing, selling, and ensuring that the candidates are informed and engaged.

These tools include a multitude of screening and testing tools: validated realistic job previews, simulations, aptitude. and skills tests, as well as simple things like asking candidates to actually do something relevant to the job: edit an article, write an advertisement, critique a circuit diagram, locate an error, etc.

Internships are another great way to assess a candidate’s fit into an organization as well as their motivation, interest, and ability – both professionally and to work within a team. While they can be difficult to set up and take time, once they are underway an organization has an almost steady stream of good candidates under assessment.

And still another excellent way to get feedback on past performance and character is to conduct a reference check. Using a tool such as Checkster, you can get anonymous and wide-ranging feedback from many people who have worked with the candidate.

I know many of you use other tools in your evaluation, but I also know you always conduct interviews — often many of them. If the interviews are used to establish a human connection, market the organization or position to the candidate, and are not the primary source of gathering the information to make a decision, I have no issue with them. When they are used as a selection tool — and particularly when we are proud of them as a selection tool — I get concerned. There are many better way to select candidates than the interview, and we should be using them more and more.

This article is provided for informational purposes only and is not intended to offer specific legal advice. You should consult your legal counsel regarding any threatened or pending litigation.

  • http://PeopleAssessments.com Dr. Tom Janz

    Now Kevin… Let’s keep it real! (Where is Lou when you really need him??)

    Of course I agree that traditional unstructured 1-on-1 interviews deliver low validity (it’s validity not reliability that drives hiring accuracy).

    However, I can’t let the following paragraph remain unchallenged because it is simply wrong. And not just a bit wrong. Completely wrong. That paragraph opens with “DDI and other respected selection organizations…”. In that paragraph you assert that even if all the powers of job modeling and behavioral interviewing are marshalled, validity remains close to 0. Simply not true. The average validity across all published referreed research studies for unstructured 1-on-1 interviews comes in at .19 (or captures 19% of the potential talent value of the candidatepool). The average validity for structured behavioral interview comes in at .53 (Schmidt and Hunter, 1998, Psychological Bulletin— and supported again by the Annual Review of Psychology chapter on selection in 2008. So can we “stick to the facts, just the facts”?

    Now I also support the idea that many who are “trained” do not follow all the tenents of good behavioral interviewing, and few outside of research studies actually operate at validity levels approaching .53– with most likely falling above .19 but less than .3. But even that is quite different from asserting that interview accuracy approaches zero (which is what you clearly said in that paragraph. I support Lou Adler’s strong focus on interviewing for performance impact, and not just competency out of context.

    Furthermore, I am a great supporter of Virtual Job Tryouts and simulations, which have shown strong face validity and good predictive validity. I also promote collecting third party viewpoints to triangulate on a candidate’s future performance value. But we are not well served by assertions that can so easily be discounted because they are empirically (and clearly) over-statements.

    “True friends stab you in the front” Tom.

  • http://newnegotiator.tumblr.com New Negotiator

    This article, and others like it, are a bit like Don Quixote tilting at windmills.

    Firms that are hiring think the interview process is valid. Period, and end of story.

    As long as that’s the case, your well reasoned arguments against the present way of interviewing and hiring is wasted breath.

    I’d rather see all of you, in the recruitment business, do something of value for your clients: teach them how to be more effective in the interviews they do have. Or failing that, get to someone like me who can help them.

  • Pingback: Recruiter: Interviews Are A Waste Of Time - MediaJobsDaily

  • http://www.activesolutionsrecruitment.co.uk Darren Ledger

    Generally as a rule the discussion, articles and posts on ERE.net are excellent, thought provoking and of current value from an industry insight perspective.

    The article ‘Interviews Are A Waste of Time..’ is a joke! The article itself is a waste of time. Good recruiters know their clients, know their culture, understand the strategic vision of the actual requirement / vacancy and understand what kind of communication skill, manner, personality, traits and so forth are going to work in that environment.

    I don’t interview a Chartered Accountant and ask them to explain to me the purpose of a General Ledger or which side of a balance sheet a debit and a credit goes! I am looking for far less evident and visible and discernible attributes and traits.

    Is this yet another attempt to totally isolate the entire recruitment process from any human interaction? Is the author providing us with a view of the future of the hiring process one which would look something like this:

    Candidate searches web for job vacancy, candidate registers for job vacancies, completes on-line assesment test, completes personality test, completes relevant legal data protection awareness forms and submist CV through automated screening system and CV shuffling and distribution system (think Taleo), system then submits candidates to organisations internal automated selection system which analyses all tests, CV’s, Academics, and so on. Automated selection system sends candidate a further batch of tests, candidate completes and then gets automated job offer.

    Candidate turns up for work on the first morning and quickly realises that there are no fellow employees because everything is done by ‘Tools’ via automated processes. malfunctioned!

    Give me a break.

  • http://www.activesolutionsrecruitment.co.uk Darren Ledger

    Oh and just to elaborate. How many recruiters can truly say that they have not lost a candidate at offer stage, not failed to fill an assignment, have never lost a placed candidate during the probationary period and similar?

    I’m bragging now but I know that I’m not alone when I say that it is over 4years since I lost a candidate at offer stage, that is because I close my candidates hard on all apsects of a potential offer before I even submit them for a job.

    I haven’t lost a candidate during their probationary period in over 3years. I don’t know why that is, it may be good luck or maybe I do really understand how to define the match between candidate and client.

    Those are the kinds of metric we should have in this industry. But how can you achieve those results withouth INTERVIEWING!

  • Pingback: Are Job Interviews Really a Complete Waste of Time?

  • http://www.shakercg.com Joseph Murphy

    A note about the gap between interviewing concepts and practices.

    I conducted a survey with SHRM and in 2006 published a white paper: The Use of Objective Candidate Evaluation Methods. (Non SHRM members may write me for a copy).

    Here are some stats.
    55% of respondents stated they use behavioral interviews that are based upon questions prepared in advance. 40% of respondents stated they do not conduct interviews with prepared questions. Kevin most certainly has his editorial comments pointed at this second group.

    When I explored further, and asked who uses behaviorally anchored rating scales and numeric summary of interview outcomes. Only 24% of respondents stated they use of this known best practice. This is the group that Tom may be using to support his retort.

    Another way to look at it is this:
    Every company interviews (bold assertion) in their candidate evaluation process.
    40% of companies begin with no preparation for what they want to learn.
    76% of companies have no structured way to evaluate what they hear.

    I think Kevin’s assertion has more generalized weight about the current state of affairs than Tom’s point about results that can be achieved when interviewing is done exceptionally well. The process that Lou Adler teaches is a great example. Lou demands adherence to the process to get the results he purports.

    Kevin’s bigger point is that there are better ways to learn more objective information about a candidate. And thanks Tom for the reference to the Virtual Job Tryout as one example. At issue here is the professional/technical capabilities of practitioners in talent attraction and selection roles and their ability and desire to build rigor, science and discipline into their practice. Kevin is inviting a shift to more thoughtfully developed and objective methods for candidate evaluation.

    Thanks Kevin.

  • http://www.jdlake.com John Lake

    As a DDI client and Master Trainer for over 20 years, I have taught the Targeted Selection process to many managers and employees (yes, employees conducting team interviews). My one caveat is – use the process as it is designed and I’ll support you every step of the way. Deviate from the prescription and you are on your own.

    Quick story…
    One manager/team hired an individual who was gone in 6 months. Since I had trained them in Targeted Selection, I wondered what happened. The manager told me it was not the program’s fault. They ignored the data that was collected because the pressure to hire was so intense. To use his words, “We needed someone really bad, and that’s what we got – someone really bad.”

    So, I’ll modify the title of this post – “Why UNSTRUCTURED Interviews are a Waste of Time.”

  • Keith Halperin

    The fact that unstructured interviews may be ineffective is a moot point until there are alternatives that meet most/all of the following criteria:
    1) Cheap- affordable by virtually all businesses/non-profits
    2) Requires little/no training to do-Highly intuitive/user-friendly
    3) Very wide applicability/effectiveness-works as well for a CEO as it does for a dishwasher
    4) Requires less time than rounds of unstructured interviews
    5) Obviously more effective to the users

    Furthermore, there should be incentives to use it/disincentives not to use it.

    As an alternative, it might makes sense to say to use selection methods which do NOT match the above criteria for positions which pay >$XX/yr.

    -Keith

  • John Miraglia

    Unstructured interviews, will always be with us. That is why best practices in selection and assessment require that hiring managers only be allowed to interview those candidates who have first “passed” objective pre-interview hurdles: i.e meet a valid ideal profile for the job, and demonstrate competencies for the position on an objective, validated assessment test(s).

  • Kevin Wheeler

    Thanks for all the comments.

    Two points:

    1. I have no problem with and support a properly done behavioral interview. I am simply saying that most recruiters and hiring managers do not do a solid behavioral interview. I have been through many interviews personally and have given many. None of the ones I received were even close to a proper and valid interview process. I am not even sure the ones I gave were, although I tried.

    2. Other means of assessing candidates are more valid, faster to administer, and much cheaper. We are kidding ourselves if we think we can intuit or in some way figure out which candidates are better simply through an interview.

    Tom, I leave the science to you and understand I misrepresented the validity percentage. What I was really trying to do was underline that we miss the .5 level most of the time.

  • http://hrtests.blogspot.com Bryan Baldwin

    Just to clarify: the “validity coefficient” (.19, .50) etc. refers to the correlation between interview score and job performance. Squaring this value gets you the percentage of variance that is accounted for. So a correlation of .50, squared, gets your .25. Which means this relationship accounts for 25% of the variance in job performance.

    25% is considered quite good (for many reasons beyond space here) and is ABOVE chance. Even a correlation of .19 means (simplistically) you’re adding value–just not as much as you might using other methods.

    Kevin’s premise remains valid (excuse the pun): unstructured interviews are generally inferior to better assessment methods. But as has been pointed out, they are popular, satisfy a need, and are perceived as being easy.

    Our best hope is to encourage organizational leaders to follow the right path and explain that this is about the bottom line as much as it is about statistics, litigation, candidate perceptions, etc.

  • http://www.pbarrett.net Paul Barrett

    RE: Dr. Tom Janz and his statement:

    “The average validity across all published referreed research studies for unstructured 1-on-1 interviews comes in at .19 (or captures 19% of the potential talent value of the candidate pool)”

    If validity is expressed as a correlation coefficient effect size (the usual way), then the correct expression is: 3.6% of variation in job performance is accounted for by the variation in the measure of interview performance.

    The statement “captures 19% of the potential talent value of the candidate pool” is meaningless, for two reasons. It is r-square which expresses % variation, and what exactly is a ‘measure’ of “potential talent value”?

    Kevin is quite right about the trivial effect size given a 0.19 correlation. It just fails to meet the standard of Recommended Minimum Practical Effect Size suggested in Ferguson, C.J. (2009) An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40, 5, 523-538.

    Clearly, no-one in their right mind would hire somebody without meeting them and evaluating their ‘fit’ with a company, with likely co-workers/supervisors, and whether or not they are completely mad. But, the real business of ‘selection’ should have been achieved before that, with the ‘interview’ as a ‘just making sure about those things that require face-to-face interaction’, and yes, ‘gut feelings’.

    Don’t knock those ‘gut feelings’ … Gigerenzer didn’t demonstrate those ‘fast and frugal processes’ for fun … Gigerenzer, G. (2008) Why heuristics work. Perspectives on Psychological Science, 3, 1, 20-29, and Gigerenzer, G. (2007) Gut Feelings: The Intelligence of the Unconscious. Viking Press. ISBN: 9780-670-03863-3.

    Yes, it’s all imperfect, but assuming the over-corrected hypothetical effect-size estimates from Schmidt and Hunter (1988) are what you will see in the real world is as flawed as using an interview as a primary selection tool.

    A reasoned balance has to be struck between error-laden empirical evidence and error-laden subjective judgment, unless you really do set up very accurate work-samples and other direct behavioral assessments specific to the actual job role (which paradoxically excludes most Assessment Centers using batteries of ‘standardized tasks’.

    There are lots of ways of ensuring such complex judgments are working better than tossing a coin (formally following up the consequences of judgments over time, ensuring multiple interviewers, ensuring some kind of crude person-attribute rating system used by interviewers with rater coherence analysis conducted post-sessions) – but you need to figure out what’s best for your company.

    Regards .. Paul

  • http://www.matchpointcareers.com Paul Basile

    Kevin, this is an excellent post. Anyone reading it carefully would see that the title refers to unstructured interviews, which are – let’s face it – a waste of time or at the very least a very poor use of time. And why is it relevant that just because companies do unstructured interviews that they always will? People can and do change. Using assessments early in the process will improve results a lot. And it doesn’t mean that no other factors will be taken into account – no one has suggested that.

  • Eric Kramer

    I agree with the majority of the posts-the interview is broken. I have just written a book- Active Interviewing:Branding, Selling and Presenting Yourself to Win Your Next Job (published by Cengage Learning). Active Interviewing, based on the premise that an interview is a sales call, is a unique approach that helps candidates become more active in the interview. The cornerstone strategy of Active Interviewing is the development and use of an interview presentation – basically a sales presentation tailored to the requirements of an interview. Using an interview presentation (a printed and bound document), a candidate links their background and experience to the critical requirements of the position and clearly communicates why they are a good candidate for the position. The presentation has 8 sections including a 30/60 day strategic action plan, personal success factors,success stories, value adds, and powerful questions.

    So here is the number 1 question- “Doesn’t the hiring manager mind giving up control in the interview?’ Having had 100s of candidates use a presentation the answer is a resounding “NO”, in fact most hiring managers welcome the opportunity to sit back and be presented with information they need to know to make a good hiring decision. To date there has not been one reported incidence of an interviewier turning down the opportunity to hear an interview presentation.

    Does an interview presentation fix the interview- again no – but it does several important things. First, and perhaps foremost, it is an excellent interview preparation tool for the candidate. Second, it provides structure to the interview and contains a great deal of the information the hiring manager needs to make an informed decision. Third, it overcomes the problem of a lousy interviewer and lets the candidate display communication/presenting/thinking skills in the interview.

    It is definately a different approach to interviewing. It will not replace the rigors of a DDI structured interview but it takes the vast majority of the unstructured and poorly administered structured interviews and improves them dramatically. It is also low cost and easily implemented. If we can improve job interviews by just 10-15% there will be a significant impact on the talent acquisition process.

    If you are interested you can learn more about Active Interviewing and see examples of interview presentations at http://www.activeinterviewing.com. I am interested in what recruiters think of the approach.

  • Keith Halperin

    @ Paul Barrett: Too much statistics, too little conventional, terminology-free English.
    I’m not smart enough-my brain HURTS!

    @Eric K:
    You say that your book is low cost- how low cost?
    Also, will it follow the other criteria?:

    1) Cheap- affordable by virtually all businesses/non-profits
    2) Requires little/no training to do-Highly intuitive/user-friendly
    3) Very wide applicability/effectiveness-works as well for a CEO as it does for a dishwasher
    4) Requires less time than rounds of unstructured interviews
    5) Obviously more effective to the users
    (I’m thinking of something like a Turbo-Tax model, which seems to work pretty well in a large number of cases. While candidate selection may not be simple, I doubt it’s as complicated as the Federal and State Tax Codes.)

    Cheers,

    Keith “Not the Brightest Compact Fluorescent in Home Depot” Halperin

  • http://www.pbarrett.net Paul Barrett

    Keith … I just impart the facts, with an evidence-base where appropriate.

    When “knowledge claims” are made by others, which rely upon a mistaken view of quantified validity, then I can only respond with a correction which uses the correct statistical terminology.

    But I agree with you – this r-square business is practically unintelligible; I no longer use correlations or r-squares myself to express predictive accuracy of any outcome. There are far more simple, straightforward, and obvious ways of imparting such information.

    So, I apologize for the technical language. There is simply no other way of expressing the r-square parameter without distorting its precise meaning, or re-expressing predictive accuracy in a completely different manner.

    Regards .. Paul

  • http://www.kerpartners.com Mitch Heinemann

    Kinda off a an off the wall article. I’ve been hired by going through a battery of tests (hired gun PhD designed) and the results were about the same for that company except they knew everyone’s IQ was above a certain number. Turnover, was no better or worse than any other place I’ve worked. Granted that was early days for me career wise but they used the same tests for “experienced” hires too.

    A big credit card company on the east coast hires in the manner Mr. Wheeler recommends and after recruiting leaders out of there for years I can’t say those folks were “better or worse” on average than candidates from similar companies that recruit via traditional interviews. The point is that the technique is not important but whether it is done well or not.

  • http://PeopleAssessments.com Dr. Tom Janz

    @Paul Barrett
    At the risk of causing further damage to Keith’s brain, there is a fine (and perhaps too esoteric) point to be made in responding to the critique that Paul Barrett leveled at me concerning r (validity) vs. r squared.

    I have a simple question for Paul. If r squared is so meaningful, why is r (and NOT r squared) the term in the utility equation that equates selection accuracy to selection program value?

    As Brogden observed so many years ago, r can be thought of as the percent of predictive value that COULD BE captured by perfect predictor, that is actually captured by an operational predictor—- given the selection ratio and variation in annual performance value among candidates.

    Second question for Paul—– What exactly does the phrase “Percentage of variance accounted for” mean? Beats me. It is the squared average cross product of the standardized predictor and criterion scores— but what does it mean in practice?

  • http://drjohnsullivan.com Dr John Sullivan

    Sacred cows make the best burgers

    Well written Kevin. One tech firm during the war for talent picked five resumes at random out of the qualified stack and hired them all without a single interview. The five turned out to be the highest quality hires that year.

    Structured or unstructured, they are all silly. All you’re really doing is asking people to describe something in their past that may or may not have happened and that they may have had little to do with. Those that actually did the work but that can’t adequately describe what happened fortunately seldom get hired.

    In my experience, those that actually believe that a self-description of past work experience at another firm, in another time, in another environment, under another business model can ever predict the future in a rapidly changing world… mentally still live in and cling to the past. They will defend this sacred cow no matter what evidence is presented.

    John Sullivan

  • http://PeopleAssessments.com Dr. Tom Janz

    Dear Dr. Sullivan
    Curious how you long for evidence, yet your grab bag of glib comments contains nothing that would even remotely be considered evidence– whether by scientists or the legal community.

    I not only cling to the past, I cling to published refereed research studies in respected scientific journals. As impressed as I always am with my own opinion, I know that it isn’t worth squat when compared to the cumulative value of learning over time based on objective research. I would not board a jet designed on the basis of your opinion. I suggest you shouldn’t either.

    Dr. Paul Meehl, Regents Professor at the University of Minnesota, once told me that when someone asserts that they know something, consider asking them three questions: [1] HOW do you know what you know, [2] HOW SURE are you of what you know, and [3] HOW DID YOU ARRIVE at that level of confidence? Other than the N=5 observation cited in your post, subjectively made, what evidence do you have that can counter the dozen or so studies that find substantial predictive validity for structured behavioral interviews? Instead you offer a heart-warming theory (to some) that falls victim to a consistent pattern of nasty little facts.

    Sad how some people will cling to a personally held belief no matter what evidence is presented.

  • http://community.ere.net/blogs/the-careerxroads-annex/ Gerry Crispin

    Kevin, this is beginning to look a little like the reaction to the kid who exclaimed the emperor has no clothes.

    We’ve spent so much effort hyping the latest candidate sourcing technologies, its not surprising…and perhaps a bit embarrassing that few disciplined, multi-faceted and predictively valid selection processes have been established and maintained. Most are neglected, foisted off on the few SIOPs remaining within corporations; buried at pilot stage or, simply impugned as alternatives too difficult, too costly or too likely to piss off candidates. Not my favorite business argument for choosing gut feel over science by the way.

    Too bad, since technology AND communication advances ought to make it even easier to reliably identify the candidate most likely to perform the best as has been pointed out by quite a few folks in this thread (might be worth rereading Joe Murphy’s comments and getting a copy of his whitepaper)

    There are extraordinary initiatives in selection and assessment that, coupled with disciplined interviews (not necessarily replacing them)mitigate the worst effects (of not training or not maintaining the skill once trained). ERE’s journal has dome a good job writing many of them up but maybe this year’s ERE awards for excellence should add a category for Selection and Assessment.

    Then again, I personally think Malcolm Gladwell was right in suggesting all interviews should be done double-blind.

    Maybe, instead of holding a sourcing challenge like one they do at Sourcecon, we could pit trained teams from several companies against each other as they attempt to interview a slate of finalists. Obviously we’ll stick a few candidates in the slate who are well-trained to game the system.

    On second thought, lets not.

  • Rachel Levine

    What about the candidate experience? I’m recruiting for highly skilled professionals in a very competitive market. While we are assessing technical skills and cultural cut, so much of our interviews are educating candidates in the firm, engagements, teams, and long term opportunity. Picking a few resumes out of a pile and calling a candidate to say “you’re hired” would have a 0 percent success rate. I love the absolutes of these articles… When you’re hiring thousands of people a year, it takes a lot of time to interview them, but if you refine the process to efficient and pointed, you get the best results.

  • Ken Schmitt

    Kevin, great article that will sure to get people talking. I agree with your sentiment about the volume of unproductive interviews that take place due to data that is completely unrelated to the job itself. However, for those that are well-crafted, that focus on the specifics of the job & that allow the interviewer – and interviewee – to learn about the other’s style, demeanor, approach to stressful situations & critical reasoning skills, the interview becomes one of several key data points when making a final decision. Our firm works very hard to support our clients in the development of effective interviews & I am proud to say that our PRR (3-year placement retention ratio) is 91%. Somebody has to be doing something right :)
    Ken Schmitt
    President, TurningPoint
    http://www.turningpointsearch.net

  • Richard Cialone

    As is typically the case with complex processes, there is no one “silver bullet” methodology to selecting candidates who will be successful in a given firm.

    Clearly, there’s much evidence, both anecdotal and empirical, that some sort of structured process, done well and with uniformity, can improve hiring results. But brains work differently across the human spectrum, and on that score I agree with Paul Barrett that it’s a mistake to discount so-called “gut feelings”.

    Perceived as unscientific, neuroscience has studied the area for centuries. Neuroscientist David Eagleman relates in his book, “Incognito: The Secret Lives of the Brain”, how in the 1600s, Rene Descartes started thinking about implicit memory, the concept that experience with the world is stored in memory, but that not all memory is accessible. He goes on to cite the 1800s writings of psychologist Hermann Ebbinghaus that posited, “most of these experiences remain concealed from consciousness and yet produce an effect which is significant and which authenticates their precious existence.”

    Sure, behavioral interviewing can be a strong tool in evaluating a candidate’s knowledge and abilities. Or, it can simply be a mechanism by which we can learn how well a candidate can craft a credible story. Depends on the skill level of the tool users.

    With respect to predicting organizational behavior perspective, though, I agree with John Sullivan’s assertion that how a person behaves in one environment may not accurately predict how that same person will behave in a different one. Again citing Eagleman, “While many animals are properly called intelligent, humans distinguish themselves in that they are so flexibly intelligent, fashioning their neural circuits to match the task at hand.” This particularly resonates with me as I’ve personally experienced it, and I know many others who have as well. This flexibility, I believe, is an important missing component in structured methodologies.

  • Keith Halperin

    @ Paul Barrett:
    Thank you, Paul.

    @ Dr. Janz:
    I took ibuprofen, but i think arguing statistics is still not useful to folks like me….
    Also, according to Schmidt (http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.172.1733):
    “This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of general mental ability (GMA) and the 18 other selection procedures. Overall, the 3 combinations with the highest multivariate validity and utility for job performance were GMA plus a work sample test (mean validity of .63), GMA plus an integrity test (mean validity of .65), and GMA plus a structured interview (mean validity of .63). A further advantage of the latter 2 combinations is that they can be used for both entry level selection and selection of experienced employees.”….
    ISTM that creating assessment procedures using GMA plus a structured interview or integrity test. Are there non-biased GMA tests which have been permitted under anti-discrimination law (https://apps.opm.gov/ADT/%28S%28ux0k5w2q3fyfs1453owusva0%29%29/Content.aspx?page=3-04&AspxAutoDetectCookieSupport=1&JScript=1)

    @ Dr. Sullivan:
    I appear to be in the curious situation of agreeing with you with little caveat (I don’t think all interviews are inherently silly or useless)….Perhaps I am asleep and dreaming….

    @ Gerry Crispin:
    IMSM, the double-blind selection referred to was based on a musical audition to measure playing ability without regard to the auditioner’s sex, with little need to consider likeability/cultural fit.

    @Rachel, @ Ken:
    I’m glad things work well for you/your firms, but that doesn’t say what you do will work well for everyone- maybe it will, but maybe it won’t. As we have to remind another ERE contributor from time to time: “the plural of anecdote is not ‘data’”.

    Cheers,
    Keith

  • http://drjohnsullivan.com Dr John Sullivan

    Quality of hire at “your firm” is what really matters
    .
    It’s time to start acting like businesspeople and to come down from the academic and consultant’s cloud. In the business world, correlations carry little weight because they can’t prove cause-and-effect. Generalizations about correlations done in completely different settings are interesting but mostly irrelevant because they don’t reflect what happened at “our firm” under “our” unique process. For the same reason, benchmark results that occurred at other firms are not accepted as proof that “our” process works at “our firm”.

    CFOs don’t put credibility on generalizations about any process. What executives want to know is does their unique process produce results with business impacts at their firm. That means that the relevant metrics are … the quality of hire (the on-the-job performance of new hires) and the ROI of the hiring process that is calculated at your company.
    .
    It’s curious to me that neither quality of hire nor ROI have even been mentioned in this string.

    John Sullivan

  • Sandra McCartt

    Interviews are a waste of time. Job Boards are dead. Third party recruiters are dead, the candidate experience is a bad case of the dismals. Good, now that the learned have identified all that we can all go home and forget that 10% of the population is looking for a job. Internal recruiting can just open the door and hire the first 10 that walk in and if they don’t do the job, send them out the back door and hire ten more. That will take care of quality of hire and time to hire and make the metrics very simple.

  • http://www.pbarrett.net Paul Barrett

    Ok – this is unashamed stats stuff – sorry, but it is the only way to answer Tom Janz ..

    ————————-
    Tom Janz stated: I have a simple question for Paul. If r squared is so meaningful, why is r (and NOT r squared) the term in the utility equation that equates selection accuracy to selection program value?

    As Brogden observed so many years ago, r can be thought of as the percent of predictive value that COULD BE captured by perfect predictor, that is actually captured by an operational predictor—- given the selection ratio and variation in annual performance value among candidates.
    ————————

    [PB]: Hmmm – reading Brogden, H.E. (1946) On the interpretation of the correlation coefficient as a measure of predictive efficiency. Journal of Educational Psychology, 37, 2, 65-76, we see that what he is expressing is:
    “r is the ratio of the increase obtained by selecting above a given standard score on the test to the increase that would be obtained by selecting above the same standard score on the criterion itself.” (p. 68)

    This translates to:
    r = the ratio between the mean standard score of the group selected above a cut (c) on the predictor variable (interview ratings) and the mean standard score of the group selected above the same value of the cut (c) on the criterion variable.

    This definition is valid only when
    1. Both variables (interview ratings, job performance) are expressed as standard scores.
    2. Both variables are measured on continuous-valued magnitude ranges (no censoring or truncation on a non-normally distributed variable).
    3. Both variables are related linearly.
    4. Both variables (interview ratings, job performance) have very nearly or exactly the same distribution form. Ozer is quoted in
    D’Andrade, R., Dart, J. (1990) The interpretation of r verses r² or why percent of variance accounted for is a poor measure of size of effect. Journal of Quantitative Anthropology, 2, 1, 47-59 as having confirmed the ratio holding when one variable was distributed rectangularly and the other normally. I created two simulations generating 10,000 cases, where variable 1 was distributed as a Rayleigh skewed variable (beta = 0.5 and 0.2, then ‘flipped’ to simulate positive skew) and variable 2 as a Normally distributed variable, then standardized with correlation specified as 0.5 … Selecting +1SD above the mean, the Brogden ratio differed by approximately 0.04, overestimating the true correlation. Using a 1-7 truncated but continuous-value Rayleigh distribution for performance ratings, the Brogden formula failed altogether as above a certain cut predictor- score, no performance ratings existed (because of skew and truncation in the performance ratings).

    So, that’s why r is used – as it indexes the ratio between mean standardized scores of predictor and criterion-selected groups, holding the cut-score equivalent. Note this assumes that the mean standardized score is a valid parameter rather then the median, in cases where one or both constituent distributions are non-normal.

    Which is also why this definition does not figure in any statistics textbook I’m aware of, as a formal, straightforward definition of the meaning of a correlation coefficient (D’Andrade and Dart note that it was Lewis Goldberg who brought their attention to Brogden’s interpretation). Clearly it is something very peculiar to organizational psychology/utility analysis. It is fundamentally different from the generic regression definition of correlation as the ratio of explained to total variation, using standardized or non-standardized variables, but is highly relevant when considering the specific issue of the gain associated with use of a predictor scores over and above that from using perfect criterion scores as predictors.

    So, with regard to the interview effect sizes … the generic (meta-analyzed) standardized interview ‘measure’ correlates at 0.19 with the generic (meta analyzed) standardized measure of job performance, indicating the ratio between mean standardized scores in the predictor referenced and criterion-referenced groups. If we actually selected on future job-performance for a candidate, the interview measure predictor would be expected to overlap 19% of this performance selection group in terms of their standard scores (given the assumptions being made actually hold). Put another way, we would expect a gain in performance of 19% using interview scores over and above random selection.

    [Tom Janz]: Second question for Paul—– What exactly does the phrase “Percentage of variance accounted for” mean? Beats me. It is the squared average cross product of the standardized predictor and criterion scores— but what does it mean in practice?
    [PB]: This is trivial. Just refresh your memory with the equations for explained, unexplained, and total variation in simple linear regression. r-square is clearly more easily interpreted within the context of regressing a predictor on a criterion.

    However, given the article by Sturman, M.C. (2000) Implications of Utility Analysis adjustments for estimates of human resource intervention value. Journal of Management, 26, 2, 281-299, Brogden’s incorporation of the correlation into the standard utility analysis formula seems pretty pointless, given the standard utility model equation appears not to work.

    The article abstract:
    This paper argues that most utility analysis (UA) applications are flawed because they employ an overly simplistic formula. This piece reviews the literature on UA adjustments and demonstrates that the adjustments have a large impact on resulting estimates. These results imply that human resources programs do not invariably yield positive returns; rather, intervention success is contingent on program-specific, organizational, and environmental factors.

    So, thank you for bringing up the Brogden definition; that was a new one to me – and it is an interesting definition of a Pearson correlation. But, I’m not sure of its practical significance given the number of assumptions required to be made in order for the estimate to retain its validity.

    Anyway, this has been an interesting exercise, for me at least.

    Regards .. Paul

  • http://superecruiter.blogspot.com/ Morgan Hoogvelt

    I wouldn’t go as far as to say they are a waste of time however, that the interview process is broken. I would agree that the majority of corporate America does not know how to conduct a proper interview.

    I have worked with numerous hiring managers all claim to be expert interviewers only to find they are unprepared, ask illegal and wrong questions and so on.

    I myself laugh at all the crazy interview questions posed to candidates. Those to me are all worthless.

    Good companies will have the interview down to a science.

  • John Buckley

    Here’s a quick case study of the validity of interviews:

    A small company hires 64 employees over a several year period. 9 of these employees are taken sight unseen – they are randomly accepted, without interview, based on a transition of new work to the company. 30 hires are “known” through previous work experience, including sometimes contract work before hiring, although they are still interviewed. 25 hires are “unknowns” who go through the interview process only. The 64 employees are ranked (admittedly subjectively) 20-60-20 so that 13 are identified as top performers, 38 are middle performers, and 13 are bottom performers.

    When sorted into random, interview selected, and known candidates, there are 5% fewer top performers in the random group than in the interviewed group, and there are 9% more bottom performers. The “known” hires constitute an even higher percentage of top performers and lower percentage of bottom performers.

    The obvious conclusion is that while “knowing” your candidate through work experience is the best option, it is an option that isn’t always available, and a robust interview process is a valid improvement over random hiring.

  • G Forse

    Rachel Levine – thank you!

    I’ve been following these comments more for entertainment purposes than anything else, but I was very happy to see someone finally say something about the other side of the interview – the candidate. Also, how about the value of pre-closing a candidate on your process, properly setting expectations, etc.? Find me a tool that will do that effectively…

    G

  • http://www.jdlake.com John Lake

    G Forse:
    In one organization I was with, this was the responsibility of the “host” – the primary contact person that the candidate would be connected to (usually the hiring manager). They would explain the process and the timeline. They would also be responsible for asking for the candidates questions regarding the position, organization, etc.

    However, I am talking about a structured process here – with specific responsibilities for each person/team involved with the candidate(s). They made it work beautifully…
    Hiring Manager: Host and candidate’s primary contact. Kept the interview process moving.
    Employees/Team: Conducted the interviews and made recommendations to hiring manager (yes they were trained and certified).
    Hiring Manager: Final decision.
    HR: Process champion and records keeper.

  • Pingback: Bullet Point: The Question Every Interviewer Should Be Able to Answer | The Hired Guns Blog

  • http://PeopleAssessments.com Dr. Tom Janz

    Dear Dr. (Sullivan, that is)
    The silly assertion in the room is the one that corporate contexts are so unique and different that an assessment method that works quite well in one would not work well in others. It is not a new silliness. Even us pointy headed people scientists (IO Psychologists) used to think that you needed to carefully select a different set of test scales and then prove their effectivenss for each job title and company, because every hiring situation is fundamentally different. Then we learned how to separate signal from noise (called meta-analysis) so we could sort out the noise (the random variance that naturally occurs in small sample research) from the signal concerning which types of tests work for which types of jobs. Now, no self-respecting people scientist remains ignorant of validity generalization and meta-analysis.

    You are welcome to your opinions. Just please stop trying to pass them off as practical knowledge, when they in fact mislead, misguide, and misrepresent what we know about the power of different types of interviews and other forms of pre-employment assessment.

    Clearly, the factors that shaped the species do not vary from one corporation to another. The fundamental underlying skill sets

  • http://www.pbarrett.net Paul Barrett

    Tom, you respond to Dr.Sullivan with such certainty about meta-analysis and validity generalization. Clearly, you truly believe meta-analyses are providing accurate estimates of effect sizes.

    But, we now know from both the social and medical sciences, that meta-analyses are not always so ‘robust’ as one might think from Frank Schmidt’s and others’ recent bravura positions and claims.

    Obviously, meta-analysis of exactly the same defined predictors and criteria work, given a reasonable number of constituent =random= samples; they have to via statistical sampling theory.

    And therein lies the problem, it is actually very rare to find a meta-analysis in psychology which uses studies composed of exactly the same predictors (say a single-brand personality test) and an outcome which is the exactly the same across all studies e.g. absenteeism rate, income, no. of widgets made etc.

    All too often, those who implement meta-analyses mix studies which comprise different predictors (but given the same name) with outcomes such as “job performance”, which are also constituted from many kinds of subjective ratings and other measures of performance within/across the studies. That confounds the statistical properties of meta-analysis with the peculiar problem of predictor and outcome ‘mixture error’. By mixture error, I mean the problem of including same-name predictors or outcomes which are not quantitatively proxies of one another.

    While not necessarily agreeing entirely with Dr. Sullivan, I am more wary about dismissing out of hand the ‘specificity’ of validity, partly because having spent time evaluating clinical interventions within forensic high-security psychiatric institutions (which use carefully documented therapeutic treatment guidelines from others in various institutions) and watching how situational attributes (the therapist, the institutional mindset, the treatment of patients by nursing staff, and even the initial diagnostic precision of psychiatrists) can render what ‘should have worked’ into the question “we are doing everything laid down in the manual but it’s not working as expected”, you realize ‘meta-analysis’ of multiple studies are only giving you part of the picture, unless these other diffuse and difficult-to-capture attributes are also part of the meta-analysis.

    There is also the body of published evidence below:

    Bailar III, J.C. (1997) The promise and problems of meta analysis. New England Journal of Medicine, 337, 8, 559-561.

    LeLorier, J., Gregoire, G., Benhaddad, A., Lapierre, J., Derderian, F. (1997) Discrepancies between meta-analyses and subsequent large randomized, controlled trials. The New England Journal of Medicine, 337, 8, 536-542.

    Schonemann, P.H., & Scargle, J.D. (2008) A generalized publication bas model. Chinese Journal of Psychology, 50, 1, 21-29.

    Bobko, P., & Roth, P.L. (2008) Psychometric accuracy and (the continuing need for) quality thinking in meta analysis. Organizational Research Methods, 11, 1, 114-126.

    Shercliffe, R.J., Stahl, W., & Tuttle, M.P. (2009) The use of meta-analysis in psychology: A superior vintage or the casting of old wine in new bottles? Theory and Psychology, 19, 3, 413-430.

    Rich, G.A., Bommer, W.H., MacKenzie, S.B., Podsakoff, P.M., & Johnson, J.L. (1999) Apples and apples or apples and oranges? A Meta-Analysis of objective and subjective measures of salesperson performance. Journal of Personal Selling & Sales Management, XIX, 4, 41-52.

    Hennekens, C.H., & DeMets, D. (2009) The need for large-scale randomized evidence without undue emphasis on small trials, meta-analyses, or subgroup analyses. Journal of the American Medical Association, 302, 21, 2361-2362.

    Hennekens, C.H., DeMets, D., Bolland, M.J., Grey, A., Read, I., Vosk, A. &, Sacristan, J.A. (2010) Commentaries and reply to the Hennekens and DeMets Commentary on Meta-Analysis. Journal of the American Medical Association, 303, 13, 1253-1255.

    Levine, T.R., Asada, K.J., & Carpenter, C. (2009) Sample sizes and effect sizes are negatively correlated in meta-analyses: Evidence and implications of a publication bias against nonSignificant findings. Communication Monographs, 76, 3, 286-302.

    Pace, V.L., & Brannick, M.T. (2010) How similar are personality scales of the “same” construct? A meta-analytic investigation. Personality and Individual Differences, 49, 7, 669-676.

    Fanelli, D. (2011) Negative results are disappearing from most disciplines and countries. Scientometrics (Online First – DOI: 10.1007/s11192-011-0494-7 … http://www.springerlink.com/content/bt125xw73v72282w/), , , 0-0.

    and a little white paper of mine and J.P. Rolland, documenting the range of the estimated population correlation from multiple meta-analyses, between “The” Big Five personality attributes Emotional Stability and Conscientiousness, also including a large (n=156,000) sample of data from the Hogan database.
    Barrett, P.T. and Rolland, J.P. (2008) The Meta-Analytic Correlation between the Big Five Personality Constructs of Emotional Stability and Conscientiousness: Something is not quite right in the woodshed.
    http://www.pbarrett.net/#whitepapers

    For example, with correlations around 0.2, many companies instituting a particular ‘intervention, (say our interviews), will actually show a negative benefit of an interview to eventual job performance. It’s just the rate of the positive to the negative over all such companies will favour the positive. The effect can be seen clearly in the Gallup Q12 validity article Harter, J.K., Schmidt, F.L., & Hayes, T.L. (2002) Business unit-level relationship between employee satisfaction , employee engagement, and business outcomes: a meta analysis. Journal of Applied Psychology, 87, 2, 268-279 (check out my whitepaper which explains the problem visually and with numbers … The ROI of the Gallup Q12: Assessing the true value of high-cost HR interventions) at:
    http://www.pbarrett.net/#whitepapers.

    So, I would propose that the reality of ‘generalizable validity’ is in reality, rather more complicated than Tom Janz’s snappy corrective applied to Dr. Sullivan.

    Regards .. Paul

  • Pingback: What is a job interview really about? - Jobs, employers, employees, hiring, resumes, occupations, government, laws, unions, contracts, workers, part-time - Page 5 - City-Data Forum

  • E J

    Mr. Wheeler,
    I read your blog article “Why Interviews Are A Waste of Time”, and I have to admit I agree with you and disagree with you on several statements you make in the article. It is true that the interview can be a powerful tool and that many hiring managers do not possess the adequate skills and competency to conduct an effective interview. I can also appreciate you pointing out the legalities of the interview process, by stating the EEOC requires that the “interview be a selection test and requires that it be validated before use”. But I disagree with you that the interview process is a waste of time. I think that over the course of time we have just gotten off track with the purpose of the interview process and the basic how to of selecting a candidate versus the “let me pull a rabbit out of the hat” technique that many interviewers use.
    As a current job seeker, and after reading your article, I realize that I can take the basic “show and tell” with me into an interview. This was a trick I used many years ago. The next time I have an interview, I am going to volunteer to do the actual job at hand right on the spot”. What a great way to show excitement and willingness to get and do the job!
    The holidays are coming up and I recently interviewed for a retail position. Had I read your article before the interview I would have volunteered to show that I could make the sale. I don’t think the store manager would have let me, but it sure shows willingness and bodaciousness!

  • Pingback: AudreyPodrey.com » How Not to Write a Resume

  • Pingback: Dumb Ways to Decide…? People | X or Y

  • Pingback: Boston reputation management

  • Pingback: Ace Parking Keith Jones

  • Pingback: Long Beach reputation management

  • Pingback: Lean Recruitment - Hire faster and better - JOBBOX.io