Employment Assessments: 21st Century Snake Oil?

The Oxford English Dictionary defines snake oil as "a quack remedy or panacea." The origins of snake oil as a derogatory phrase trace back to the latter half of the 19th century, which saw a dramatic rise in the popularity of "patent medicines." Despite the name, most patent medicines were not officially patented. They were medicines with questionable effectiveness whose contents were usually kept secret.

By the middle of the 19th century the manufacture of patent medicines had become a major industry in America. Often high in alcoholic content and fortified with cocaine, morphine or opium, many of these concoctions were advertised for infants and children. Some level of exoticism in the contents of the preparation was deemed desirable by their promoters and nearly any scientific discovery could inspire a key ingredient or principle in a patent medicine.

From the beginning, some physicians and medical societies were critical of patent medicines. They argued that the remedies did not cure illnesses, discouraged the sick from seeking legitimate treatments, and created negative consequences like alcohol and drug dependency.

By the end of the 19th century, Americans favored laws to force manufacturers to disclose the remedies' ingredients and use more realistic language in their advertising. These laws met with fierce resistance from the manufacturers. Finally, with strong support from President Theodore Roosevelt, a Pure Food and Drug Act was passed by Congress in 1906, paving the way for public health action against unlabeled or unsafe ingredients and misleading advertising.

Are there any similarities between the sales and marketing of snake oil/patent medicine and the sales and marketing of employment assessments? Do the assessments have questionable effectiveness? Are their contents kept secret? Do they market scientific discoveries, in the form of buzzwords, as inspiring key ingredients? Is there public action challenging the assessments?

Questionable Effectiveness?

According to a 2012 study by Oracle and Development Dimensions International (DDI), a global human resources consulting firm whose expertise includes designing and implementing selection systems, more than 250 staffing directors and over 2,000 new hires from 28 countries provided the following perspectives on their organization’s selection processes (the following are excerpts from the study):

• [O]nly 41 percent of staffing directors report that their pre-employment assessments are able to predict better hires.
• Only half of staffing directors rate their systems as effective, and even fewer view them as aligned, objective, flexible, efficient, or integrated. 
• [T]he actual process for making a hiring decision is less effective than a coin toss.

In a 2007 article titled, “Reconsidering the Use of Personality Tests in Employment Contexts”, co-authored by six current or former editors of academic psychological journals, Dr. Kevin Murphy, Professor of Psychology at Pennsylvania State University and Editor of the Journal of Applied Psychology (1996-2002), states:

The problem with personality tests is … that the validity of personality measures as predictors of job performance is often disappointingly low. A couple of years ago, I heard a SIOP talk by Murray Barrick … He said, “If you took all the … [factors], measured well, you corrected for everything using the most optimistic corrections you could possibly get, you could account for about 15% of the variance in performance [between projected and actual performance].” … You are saying that if you take normal personality tests, putting everything together in an optimal fashion and being as optimistic as possible, you’ll leave 85% of the variance unaccounted for. The argument for using personality tests to predict performance does not strike me as convincing in the first place.

Secret Contents?

Using terms like patent-pending, proprietary and trade secret, employment assessment companies claim that they cannot disclose information about their assessment processes. Are these claims legitimate or, like the Wizard of Oz, are the claims used to mask the lack of relevant and legal substance behind the assessments? Or is it a bit of both? 

Even assuming that the confidentiality claims by the assessment companies are appropriate, are there no circumstances in which the companies must disclose information regarding how their assessments are developed and implemented and the results of the assessment usage across a broad population of job applicants? There are. 

In a 2012 decision, a federal appeals court stated the Americans with Disabilities Act (ADA) prohibits employment tests when such tests screen out or tend to screen out disabled people and the use of the test is not job-related for the position in question and consistent with business necessity. The court went on to order that the employer and testing company, Kroger and Kronos, respectively, provide the Equal Employment Opportunity Commission (EEOC) with:

• Any and all documents and data constituting or related to validation studies or validation evidence pertaining to Kronos assessment tests purchased by Kroger, including but not limited to such studies or evidence as they relate to the use of the tests as personnel selection or screening instruments, even if created or performed for other customer(s);
• The user’s manual and instructions for the use of assessment tests used by Kroger;
• Any and all documents (if any) related to Kroger, including but not limited to correspondence, notes, and data files, relating to Kroger; its use of the assessment test; results, ratings, or scores of individual test takers; and any validation efforts made thereto; and
• Any and all documents discussing, analyzing or measuring potential adverse impact on persons with disabilities.

So, it's quick and easy for the regulatory agency (EEOC) to obtain the necessary information, right?No. The EEOC investigation leading to the appeals court decision referenced above has been ongoing for more than six years. It has generated a number of district court and appellate court decisions. None of those decisions have addressed substantive claims of discrimination. They are all decisions relating to the unwillingness of Kroger and Kronos to provide information to the EEOC that would allow the EEOC to determine whether the Kronos assessments illegally discriminated against persons with disabilities.

Please see Kroger and Kronos: Chaos and Disorder.

Negative Consequences?

Employment personality tests discriminate against applicants with mental illness. These applicants are ready, willing and able to work, but are being illegally screened out from employment consideration by personality tests that use a non-validated stereotype of the capabilities of persons with mental illness. Please see What Are The Issues, ADA, FFM and DSM, and Employment Assessments Are Designed to Reveal an Impairment.

Employment discrimination on the basis of mental illness affects all demographic groups. Mental illness is no respecter of age, gender, geography, income, occupation, military status, race, religion or sexual orientation. Persons with mental illness include military veterans returning to the civilian workforce, new and expectant mothers, LGBTs and young adults. Please see Tests Discriminate Against Returning Veterans, Tests Discriminate Against New and Expectant Mothers and Employment Tests Discriminate Against LGBTs.

The illegal screening out of applicants with mental illness has come at a cost of tens of billions of dollars to taxpayers and the U.S. Treasury.  The prevalence of mental disorders has generally remained unchanged over the past 15 years and substantially increased rates of treatment should have resulted in a decline in the percentage of persons receiving disability awards who are diagnosed with mental illness. Sadly, no. People with psychiatric impairments constitute the largest and most rapidly growing subgroup of income support program awards (SSDI, SSI). Please see Costing Taxpayers Billions of Dollars Each Year.

Every year since 1999, more Americans have killed themselves than the year before, making suicide the nation’s greatest untamed cause of death. Being unemployed is associated with a 2-3X increase in the relative risk of death by suicide, compared with being employed. Given that more than 90% of persons who attempt suicide have mental illnesses, a tool like personality testing that illegally excludes persons with mental illness from employment consideration leads to an increase both in perceived burdensomeness and thwarted belongingness/social alienation, two critical elements tied to the risk of suicide. Please see Does the Rising Use of Employment Personality Tests Contribute to An Increase in Suicides?

Marketing by Scientific Buzzwords?

New entrants in the assessment field include ConectCubed, Good Co., Evolv, Knack and Prophesy Sciences They compete with incumbents like Kenexa (IBM), Kronos, SHL, Success Factors (SAP),  and Taleo (Oracle). Marketing claims include:

Our games are fun, but our technology is rock solid. We design and develop our games using state-of-the-art behavioral science, then we use data-mining tools and massive amounts of data to validate and compute the Knacks you earn. (Knack)

We use a powerful combination of cognitive games, biometric signals, and machine learning algorithms to compile actionable insights about you and your teammates. (Prophesy Sciences) 

We have combined decades of academic and business research with sophisticated statistical models to create our Proprietary Psychometric Algorithm. (Good Co.)

Evolv’s patent-pending technology platform unifies and supplements existing data from current systems, then utilizes that dataset to identify fact-based workforce insights that drive measurable ROI. (Evolv)

Kronos helps organizations find value in big data with enhanced analytics. (Kronos)

Public Action?

The EEOC has been attempting to investigate the use of Kronos assessment by Kroger for more than six years. As noted previously, that investigation has resulted in a number of court decisions, not on the substance of the claims of alleged discrimination, but on the requirement of Kronos and Kroger to provide the EEOC with relevant information regarding the assessment and its usage.

That investigation has evolved into a systemic investigation by the EEOC. Systemic investigations involves pattern or practice, policy, and/or class cases where the alleged discrimination has a broad impact on an industry, profession, company, or geographic area. As stated by the court in the 2012 appellate decisions referenced previously, it is “a proper inquiry for the EEOC to seek information about how these tests work, including information about the types of characteristics they screen out….“ 

In connection with systemic investigations, the EEOC’s enforcement tools include issuing broad information requests and subpoenas on employers that are named as respondents in EEOC charges, particularly when the EEOC suspects systemic discrimination, and filing pattern or practice class lawsuits in federal court. For some employers, the potential class size can be measured in the millions of plaintiffs. 

The EEOC's systemic investigation of Kroger and Kronos, as well as its investigation of other employers and their assessment companies, is consistent with the EEOC's implementation of its Strategic Enforcement Plan (SEP) for 2013-2016. The first national priority of the SEP is “eliminating systemic barriers in recruitment and hiring.” The SEP goes on to state that “people with disabilities continue to confront discriminatory policies and practices at the recruitment and hiring stages. These include … the use of screening tools (e.g., pre-employment tests …)."

Disability discrimination, including employment assessment litigation, is a significant element of the EEOC's enforcement activities. As shown in the chart below, ADA claims constituted the largest percentage of the EEOC’s yearly litigation filing activity for FY 2013 - almost half of all cases. 

The Importance of This Issue

The long-term fiscal stability of the United States of America depends, in part, on ensuring that Americans with disabilities have meaningful opportunities to contribute to our collective well-being and on eliminating outdated policies that keep people in cycles of poverty and dependency.

More than two decades after the passage of the ADA, the unemployment rate for Americans with disabilities stubbornly remains nearly double that of people without disabilities, while their rate of labor force participation has continued to be abysmally low. Figures from the Bureau of Labor Statistics show that labor force participation for workers with disabilities was 20.3 percent, while the total for workers without disabilities was 69.1 percent—more than three times higher. 

There are many benefits of employment—work enhances skills such as communication, socialization, academics, physical health, and community skills; it factors into how one is perceived by society; it promotes economic well-being (reducing government expenditures on income support programs, Medicare and Medicaid); it leads to greater opportunity for upward mobility; and it contributes to greater self-esteem.