What Do Car Platforms and Employment Assessments Have In Common? Systemic Risk

A major manufacturing trend in the automotive industry has been the focus on global platform-based vehicle design and manufacturing. One of the trade publications predicts that by 2017 VW will build over 40 models on its MQB platform (Audi A3, VW Golf, etc.) totaling over 4 million vehicles worldwide. All of the global car makers will be above 2 million vehicles per year with their major platforms.

The savings from faster vehicle development time, lower tooling costs and more advantageous supply contracts (by leveraging higher volumes) could arguably add up to billions of dollars in savings. Suppliers also benefit as they service larger, more stable supply contracts that support broad-based utilization of their global manufacturing facilities.

And the risks? The same global scale that can yield big savings benefits could drive huge costs and reputation damage if a product defect or manufacturing quality issue surfaces. Instead of the defect being confined to a single plant, single model or even a single vehicle segment, the potential exists for the defect to be multiplied across thousands of vehicles around the world in a very short amount of time.

Employers and assessment companies face analogous risks. For many applicants, especially those applying for entry-level positions in retail and food service, employment assessments offer a standardized experience for job applicant. While this "one size fits all" approach helps to reduce an employer's costs and may reduce the impact of overtly biased or discriminatory behavior on the part of one or more recruiters, the inclusion of one or more potentially "defective components" in the assessments means that employers face the risk that a finding of bias or discrimination in one of the assessments will put all tests at risk. Please see When the First Domino Falls: Consequences to Employers of Embracing Workforce Assessment Solutions.

These "defective components" in assessments may be either design defects (i.e., the adoption and use of the five-factor model of personality) or manufacturing defects (i.e., coding errors in the assessment software). The latter is analogous to the coding error at 23andMe that resulted in notices going out to some customers informing them that they had a chronic and life-shortening condition - limb-girdle muscular dystrophy - when they did not. Please see On Not Dying Young: Fatal Illness or Flawed Algorithm?

Gut Check: How Intelligent Is Artificial Intelligence?

In 6 Ways to Create a Smarter Workforce in 2014, Tim Geisert, Kenexa's Chief Marketing Officer, writes:

Use science, precision and data to hire the right people for the job the first time. According to a 2012 IBM study, 71 percent of CEOs surveyed cited human capital as their greatest source of sustained economic value. So, why does HR continue to rely on gut instinct alone to make such important decisions?

Perhaps Mr. Geisert should have spoken with Rudy Karsan, one of Kenexa's founders and its CEO, who wrote in Listening to Your Gut Feeling:

On the big decisions I have gone against my gut on a couple of occasions and it’s been a train wreck. My gut is made up of my instinct, my faith, my intuition, my experiences and data that is currently inaccessible because it’s tucked away in the deep recesses of my brain.

Then again, perhaps Mr. Karsan should have spoken with Troy Kanter, President of Human Capital Management for Kenexa, who stated in a press release:

Now, instead of making hiring decisions based on 'gut' feelings and personal likes and dislikes, hiring managers and HR can select candidates based on objective data, which also prevents potential legal ramifications and mitigates risk in the hiring process."

So, two of the Kenexa executives believe that going with your gut is bad idea, but the most senior Kenexa executive believes that going against your gut is an accident waiting to happen. Who's right?

What Does Data Tell Us?

The benefits provided by the use of pre-employment assessments, whether called workforce science, talent analytics or any other name, should be readily apparent and quantifiable. For example, has the rising use of pre-employment assessments over the past 10-15 years resulted in greater employee engagement?

Gallup has measured employee engagement since 2000 and it defines “engaged” employees as those who are involved in, enthusiastic about, and committed to their work and contribute to their organization in a positive manner. The 2013 Gallup report shows that 70% of American workers are “not engaged” or “actively disengaged” and are emotionally disconnected from their workplaces.

Is the employee engagement data from the 2013 report an anomaly? No. As shown in the following chart taken from the 2013 Gallup report, there has been little change workplace engagement levels since 2000.

Contrast the lack of change in employee engagement with the marketing of pre-employment assessments, like this selection from the Kronos website:

Your employees are the face of your brand and the most vital asset of your business. They drive your productivity and profitability. What’s more important than selecting the right ones? Take the guesswork out of employee selection with industry-specific, behavioral-based assessments and interview guides [from Kronos].

Gallup’s research shows that employee engagement is strongly connected to business outcomes essential to an organization’s financial success, including productivity, profitability, and customer satisfaction. Yet, as the report states, "workplace engagement levels have hardly budged since Gallup began measuring them in 2000."

Brain vs Computer

In "Thinking In Silicon," a December 2013 article in the MIT Technology Review, Tom Simonite writes:

Picture a person reading these words on a laptop in a coffee shop. The machine made of metal, plastic, and silicon consumes about 50 watts of power as it translates bits of information—a long string of 1s and 0s—into a pattern of dots on a screen. Meanwhile, inside that person’s skull, a gooey clump of proteins, salt, and water uses a fraction of that power not only to recognize those patterns as letters, words, and sentences but to recognize the song playing on the radio. 

All today’s computers, from smartphones to supercomputers, have just two main components: a central processing unit, or CPU, to manipulate data, and a block of random access memory, or RAM, to store the data and the instructions on how to manipulate it. The CPU begins by fetching its first instruction from memory, followed by the data needed to execute it; after the instruction is performed, the result is sent back to memory and the cycle repeats. Even multicore chips that handle data in parallel are limited to just a few simultaneous linear processes. 

Brains compute in parallel as the electrically active cells inside them, called neurons, operate simultaneously and unceasingly. Bound into intricate networks by threadlike appendages, neurons influence one another’s electrical pulses via connections called synapses. When information flows through a brain, it processes data as a fusillade of spikes that spread through its neurons and synapses. You recognize the words in this paragraph, for example, thanks to a particular pattern of electrical activity in your brain triggered by input from your eyes. Crucially, neural hardware is also flexible: new input can cause synapses to adjust so as to give some neurons more or less influence over others, a process that underpins learning. In computing terms, it’s a massively parallel system that can reprogram itself.

Okay, but what about computing at the bleeding edge, like the "cognitive computing" of IBM's Watson?

Not So Elementary

According to a January 9, 2014 article in CIO.com, IBM says cognitive computing systems like Watson are capable of understanding the subtleties, idiosyncrasies, idioms and nuance of human language by mimicking how humans reason and process information.

Whereas traditional computing systems are programmed to calculate rapidly and perform deterministic tasks, IBM says cognitive systems analyze information and draw insights from the analysis using probabilistic analytics. And they effectively continuously reprogram themselves based on what they learn from their interactions with data.

Said IBM CEO Ginni Rometty, "In 2011, we introduced a new era [of computing] to you. It is cognitive. It was a new species, if I could call it that. It is taught, not programmed. It gets smarter over time. It makes better judgments over time." "It is not a super search engine," she adds. "It can find a needle in a haystack, but it also understands the haystack."

This "new species" of computing has its challenges. According to "IBM Struggles to Turn Watson Computer Into Big Business," a recent Wall Street Journal article:

Watson is having more trouble solving real-life problems than "Jeopardy" questions, according to a review of internal IBM documents and interviews with Watson's first customers. 

For example, Watson's basic learning process requires IBM engineers to master the technicalities of a customer's business—and translate those requirements into usable software. The process has been arduous.

Klaus-Peter Adlassnig is a computer scientist at the Medical University of Vienna and the editor-in-chief of the journal Artificial Intelligence in Medicine. The problem with Watson, as he sees it, is that it’s essentially a really good search engine that can answer questions posed in natural language. Over time, Watson does learn from its mistakes, but Adlassnig suspects that the sort of knowledge Watson acquires from medical texts and case studies is “very flat and very broad.” In a clinical setting, the computer would make for a very thorough but cripplingly literal-minded doctor—not necessarily the most valuable addition to a medical staff.

As Hector J. Levesque, a professor at the University of Toronto and a founding member of the American Association of Artificial Intelligence, recently wrote:

 "As a field, I believe that we tend to suffer from what might be called serial silver bulletism, defined as follows:

the tendency to believe in a silver bullet for AI, coupled with the belief that previous beliefs about silver bullets were hopelessly naıve. 

We see this in the fads and fashions of AI research over the years: first, automated theorem proving is going to solve it all; then, the methods appear too weak, and we favour expert systems; then the programs are not situated enough, and we move to behaviour-based robotics; then we come to believe that learning from big data is the answer; and on it goes."

Similarly, assessment companies have marketed the benefits of "science, precision and data" over the past fifteen years under the guise of neural networks, artificial intelligence, big data and deep learning, yet what has changed? Employee engagement levels have hardly budged and employee turnover remains a continuing and expensive challenge for employers.

The more things change, the more they remain the same or, in deference to Monsieur Levesque "plus ça change, plus c'est la même chose."

By The Numbers: What Employee Engagement and Stock Price Performance Tell Us About Pre-Employment Assessments

The benefits provided by the use of pre-employment assessments, whether called workforce science, talent analytics or any other name, should be readily apparent and quantifiable. For example, has the rising use of pre-employment assessments created greater employee engagement? If pre-employment assessments are designed to find those employees with the best fit for the company culture, shouldn't companies who use those assessments outperform their peer companies who do not use the assessments?

Employee Engagement

Gallup defines “engaged” employees as those who are involved in, enthusiastic about, and committed to their work and contribute to their organization in a positive manner. The information in this section come from Gallup's State of the American Workforce 2013 report.

The report shows that 70% of American workers are “not engaged” or “actively disengaged” and are emotionally disconnected from their workplaces and less likely to be productive. Currently, 52% of workers are not engaged, and worse, another 18% are actively disengaged in their work. Gallup estimates that these actively disengaged employees cost the U.S. between $450 billion to $550 billion each year in lost productivity.

Having the vast majority of American employees not engaged with their workplaces is troublesome as the country attempts to recover ground lost during the financial crisis and get back on track to pre-recession levels of prosperity. Even more troubling is that workplace engagement levels have hardly budged since Gallup began measuring them in 2000, with fewer than one-third of Americans engaged in their jobs in any given year. 

So, notwithstanding the exponential growth in pre-employment assessments over the past 10-15 years, "workplace engagement levels have hardly budged" since 2000. Contrast the lack of growth in employee engagement with the marketing of pre-employment assessments, like this selection from the Kronos website:

Your employees are the face of your brand and the most vital asset of your business. They drive your productivity and profitability. What’s more important than selecting the right ones? Take the guesswork out of employee selection with industry-specific, behavioral-based assessments and interview guides [from Kronos].

Gallup’s research shows that employee engagement is strongly connected to business outcomes essential to an organization’s financial success, including productivity, profitability, and customer satisfaction. And engaged employees are the ones who are the most likely to drive the innovation, growth, and revenue that their companies desperately need. Yet, the purported benefits of pre-employment assessments have failed to move the needle on employee engagement, meaning companies have not received the promised productivity and profitability "bumps" from using pre-employment assessments.

Stock Price Performance

The chart below compares stock price performance of CVS Caremark (CVS) and Walgreens Co.(WAG) for the period from July 31, 2011 to January 11, 2014.

The reason for selecting the July 31, 2011 start date is that in July 2011, CVS and the Rhode Island Civil Liberties Union (ACLU) entered into a voluntary settlement addressing the ACLU’s complaint challenging CVS’s use of a pre-hire questionnaire that the ACLU claimed could have a discriminatory impact on people with certain mental impairments or disorders.

The settlement came after the Rhode Island Commission for Human Rights had issued a finding in February 2011 that there was "probable cause" to believe that the questionnaire used by CVS violated state anti-discrimination laws that bar employers from eliciting information that pertain to job applicants' mental or physical disabilities.Pursuant to the settlement agreement, CVS agreed to permanently remove the questions at issue from its online application. Since that time, CVS has not utlized online pre-employment assessments as part of its hiring process.

The reasons for comparing CVS stock price performance with Walgreens are (i) that Walgreens and CVS are direct competitors and (ii) Walgreens continues to use online pre-employment assessments, including personality tests similar to those CVS was using prior to the Rhode Island settlement agreement. CVS stock price performance is shown by the black line and Walgreens stock price performance is shown by the brown line.

The numbers don't lie, do they? Since eliminating the use of pre-employment assessments, CVS stock price performance has increased by approximately 90%. Walgreens, in contrast, continues to use pre-employment assessments and its stock price performance has increased by approximately 50%. As an investor, where would you have rather put your money? As a person with a mental illness or their family member, loved one, friend and colleague, where would you have rather shopped, at a company that engages in hiring discrimination against persons with mental illness or at one that does not (CVS)?

Market Success ≠ Product Effectiveness

Pre-employment personality tests are marketed by hundreds of assessment companies and used by thousands of employers, including many of the largest employers in the U.S. There has been significant growth in the usage of personality tests over the past 10-15 years and their widespread adoption evidences market success, but does it evidence product effectiveness?

One might argue that product effectiveness is demonstrated by market success, that one could not have the latter without the former. That argument fails in light of the many instances in which market success did not correlate with product effectiveness.

Vioxx, DePuy, Polybutylene ...

Vioxx, a non-steroidal anti-inflammatory drug developed by Merck, received FDA approval on May 20, 1999. The drug gained widespread acceptance among physicians treating patients with arthritis and other conditions causing chronic or acute pain. Merck recorded more than $11 billion in Vioxx sales during the drug's years on the market from mid-1999 to September 2004. The drug was withdrawn from the market after a study showed it increased the risk for heart attacks and strokes. At the time of the withdrawal, Vioxx was Merck’s second-best selling drug, generating $2.3 billion in sales the previous year. Since the withdrawal, the company has paid nearly $6 billion in litigation settlements, not including legal-defense costs and possible payments from pending litigation. That pending litigation includes a derivative class action lawsuit on behalf of all shareholders who lost money on Merck common stock or options trades between May 1999 and September 2004.

In 2005, DePuy, a Johnson & Johnson company, started selling its Articular Surface Replacement, or A.S.R., hip for use in standard hip replacements in the United States. Close to 40,000 patients in the U.S. received a DePuy ASR hip implant from its introduction to the market in August 2005 through August 2010, when DePuy issued a recall for the device. On November 19, 2013, Johnson & Johnson announced its agreement to pay at least $2.5 billion to resolve thousands of defective DePuy ASR hip implant lawsuits. In addition, Johnson & Johnson may pay as much as $1 billion to Medicare and private health insurers who covered the medical costs of removing its recalled hip implants

Beginning in the late 1970s, polybutylene plastic plumbing systems—touted as being cheaper and more durable than copper pipe systems—were installed in new homes nationwide. Over the years, several million homes were built with polybutylene plumbing systems. Before long, the plumbing systems began to experience failures of the fittings and of the pipe itself. Consumers nationwide attributed the failures to various causes, including inadequate design, defective manufacturing, improper installation, and degradation of the materials from chemicals in the drinking water. More than ten years of litigation, and bankruptcy for one company, would follow, and hundreds of millions of dollars would be spent before reaching a final class action resolution.

Market Success = Greater Risk Exposure

Market success does not equate with product effectiveness, but market success may multiply the unintended consequences of an inherently flawed design due to path dependencies and systemic effects.

The success of assessment companies in marketing pre-employment personality tests over the past 10-15 years has created systemic risk for their employer customers. If one employer has violated the law and subjected itself to significant liability as a consequence of its use of an assessment provided by an assessment company, then all customers of that company are similarly at risk.

Similarly, the lack of diversity in the psychological model underlying many of the personality tests offered by assessment companies (the five-factor model of personality or Big Five) means that if one assessment company's personality tests that use the Big Five is found to screen out persons with mental illness or to be an illegal medical examination under the ADA, the personality tests marketed by other assessment companies that use the Big Five (and, more importantly, their employer customers) are similarly at risk.

As set out in When the First Domino Falls: Consequences to Employers of Embracing Workforce Assessment Solution, the potential risks to employers are substantial and include:

• Damages and injunctive relief for violating the ADA and Rehabilitation Act of 1973;
• Brand damage and lost revenues; and
• Illusory indemnification by the assessment companies.

Additional risks include those referenced in the Vioxx, DePuy and polybutylene examples discussed above, risks like shareholder derivative class actions, defense costs and claims by state and federal governments for costs incurred as a consequence of the illegal acts (i.e., unemployment insurance, income support payments - SSDI and SSI, Medicare and Medicaid costs).