I am not sure about you, but I am reading more and more about the power of “big data.” Forrester, McKinsey, and IBM have all issued white papers or reports in the last month or two discussing the impact that the analysis of big data will have on business.
Big data refers to the totality of information available. This includes data in emails, instant messages, in video, and in audio files — all data that might help create a more complete understanding about an issue or person or provide an answer to some question. All the spreadsheets and databases we are currently using are made up of structured data, data that can be organized into columns or rows and then added or otherwise analyzed.
And, while this type of data is incredibly useful, access to unstructured data would add dimensions and depths that only the CIA can currently realize.
Historically, the volume and unstructured nature of so-called “big data” prevented much in way of analysis. An individual had to listen to the audio, watch the videos, read all the material, and integrate and analyze to form a conclusion. This is obviously very time-consuming, and requires training and the ability to assimilate many kinds of media. But we now have computers that are close to being able to look at large amounts of this kind of data and draw inferences, make suggestions, and provide summaries. The CIA and other government agencies undoubtedly already are using these tools to analyze email, voice mail, and phone calls in search of terrorists.
But these capabilities are about to be available to everyone. In the past few months Oracle announced it had acquired Endeca, a company that does dataanalysis and is building a Big Data Appliance — a computer specially designed to handle the volume of information found in unstructured data. IBM developed Watson, the computer that played against humans and won at Jeopardy, as a big data analysis machine.
HP announced a few days ago that it is integrating Autonomy, which it purchased earlier this year, into a new hardware platform for data analysis, SAS has developed a number of big-data applications, and EMC recently acquired Greenplum, another data analysis firm. Each of these firms is looking to mine the potential of the massive amounts of data that exist and that are being created.
Imagine the power these tools will potentially give to marketing and advertising folks. They may be able to specifically target individuals with messages that, based on the analysis of what they are writing or talking about, will entice them to buy a product or choose a suppler. On the more positive side, this level of understanding will make it possible for computers to take over call centers, much of customer support, and other jobs where knowing a lot about the caller as well as the products will be most useful.
For recruiters, this may change everything about what we do and how we do it.
The capability this analytic software has is scary and threatening. To thrive in the coming world of big-data analysis, we will all have to learn to adapt and to develop very different skills from the ones we now have.
Here is just a cursory sampling of what may be in store.
By analyzing what a department produces, what data is gathered and used, who people talk to and interact with in meetings or in social networks, these programs will be able to identify key characteristics of successful people and from that develop a list of competencies, skills, and attitudes that are most likely to be successful. They can match this against current requirements and suggest changes or skills that might improve or complement whatever exists. But a job description or analysis will be much more complete and accurate than they are today.
This capability will be here in a year or two.
By tapping into a larger data-set than we can access or analyze today, we can find more people and learn more about them than ever. We can perhaps get referrals from whoever a person calls, what they talk about, and who they refer to in the conversations.
These tools will also completely eliminate the need for Boolean search or experts in using the various forms of search that are popular today. These will all be automated to a great extent. Imagine a computer akin to the Hal 9000 in 2001:Space Odyssey that can understand human languages (similar to Siri on the Apple iPhone 4S) and conduct a search independently of a recruiter. They will be able to dig much deeper and make inferences based on data that would be impossible for a human.
I look for some of this capability within two to three years at the most.
We have an increasing ability to learn more and more about people by gleaning bits of information about someone from scraping or extracting data from websites/public information/social networks and from information about the products or services someone buys or uses, and from their interests extracted from comments, Tweets, locations, and so forth.
This, combined with better analysis of the job as described above, will let us choose people with a higher probability of success than we can do today with all of our tools.
This is just around the corner and could probably be put into practice at some level today if the machines were available for commercial use (some are) and the costs were reasonable.
With the power that these tools are already capable of, everything we do will be tracked and can be correlated to our performance.
We will be able to measure and track which calls resulted in the most candidates, what methods yielded the greatest returns, and how well our candidates performed once hired. Some of this capability is available today with tools incorporated into HRIS systems like SAP and Oracle.
Legal and Privacy Concerns are huge and will require another article to discuss.
What are your thoughts about all of this? Is this all just a pipe-dream or do you agree that it will happen?
