Systemic Risks for School Systems Employing TeacherInsight

According to Gallup Inc., its TeacherInsight (TI) assessment is an automated online interview used by many school districts to help those schools identify the best potential teachers. School districts across the country use the TI assessment as part of their teacher application and selection process.

Need to Independently Review Assessment.

In The (Non)Predictive Ability of the Gallup TeacherInsight Assessment, there is discussion of a study of the TI assessment undertaken by Michael T. Novotny. The Novotny study includes the following recommendations: 

School districts that are using, or considering the use of, the TI should require the Gallup Organization to show them the research in support of the instrument. The lack of independent research on the TI and Gallup’s unwillingness to publish their own research does not help support the credibility of the TI. If Gallup was to publish their own studies, then independent researchers could attempt to replicate those studies in order to either confirm or contradict their findings regarding the validity of the TI. Most large school districts have a research department with the capability of conducting such research.

As discussed under the headings "Ignoring EEOC Guidance" and "Disregarding Industry Standards" in Risks to Kroger Shareholders, a key element in guidance from both the EEOC and EEAC (an employer association) is the responsibility of the employer (e.g., local school district) to independently review the hiring assessment(s) it uses and to avoid reliance on the representations of the assessment provider (e.g., Gallup).

Accordingly, as Novotny recommends, school districts that are using, or considering the use of, the TI assesment should require Gallup to show them the research in support of the instrument. The Gallup-provided research should include information on the ability of the TI assessment to predict teacher performance, as well as validity studies undertaken by or on behalf of Gallup and evidence that the assessment does not discriminate against individuals who are members of classes protected by laws like Title VII and the ADA. 

What's Missing?

According to Gallup, a TI interview development study, originally completed in January 2002, demonstrated content, construct, and criterion-related validity as well as fairness across classifications of race, gender, and age. There are a number of other "protected classes," including persons with disabilities (including physical, developmental and mental disabilities), national origin, and religion, that are missing from Gallup's list.

As an example, Dr. Melanie Schneider has identified three primary concerns relevant to nonnative and bilingual speakers of English (raising issues of discrimination on the basis of national origin): 

• the timed nature of the TI assesment, 
• possible inequalities associated with limited access to computers or the Internet, and
• little perceived consideration for cultural and linguistic.

Dr. Schneider writes:

As mentioned earlier, parts of the TI contain timed questions. Unlike other standardized tests, there is no accommodation for applicants for whom English is a second or additional language or others who may have a disability that prevents a rapid-fire response. Waiting too long to respond to a question results in a missing response, which counts against an applicant’s total score. Although the timed nature of some types of questions affects all applicants who take the TI, nonnativespeakers of English are especially penalized when speed of response is required. 

Linked to the timed format of the TI, which may disadvantage some nonnative speakers of English, are possible inequalities due to the digital divide between low-income and middle-income students. ... English language learners in the United States are more likely to come from lower income families than their native-English-speaking peers. Inequalities between low-income and middle-income children in the use of and access to computers and the Internet have been well documented. ... Even among undergraduate students, values, attitudes, and beliefs about access to and facility with certain technologies may disadvantage certain groups of students, such as those from low-income immigrant families. 

Finally, a third concern for bilingual speakers of English is the belief that a singular view of talents characterizes successful teachers. Assuming that there is a single, preferred set of values, attitudes, beliefs, and behaviors associated with teacher success in the classroom ignores the role of culture in teaching and assessment

In Albemarle Paper Company v. Moody, the Supreme Court addressed a case in which an employer implemented a test (Wonderlic) on the theory that a certain verbal intelligence was called for by the increasing sophistication of the plant's operations. The company made no attempt to validate the test for job-relatedness, and simply adopted the national "norm" score as a cut-off point for new job applicants.

The Supreme Court cited the Standards of the American Psychological Association and pointed out that a test should be validated on people as similar as possible to those to whom it will be administered. The Court further stated that differential studies should be conducted on minority groups wherever feasible.

Ongoing Investigations of Assessments for ADA Compliance by EEOC

For more than six years, the EEOC has been investigating Kroger and Kronos, Kroger's assessment provider.  The investigation focuses on whether the Kronos assessment illegally screens out persons with mental illness. Please see Kroger and Kronos: Chaos and Disorder.

The case arose from a charge filed by Vicky Sands with the EEOC in 2007. The EEOC has converted Ms. Sands' investigation into a systemic investigation and has since started at least two additional systemic investigations involving companies using the Kronos assessment.

The EEOC investigations involve claims that the Kronos assessment is an illegal pre-employment medical examination, that the data collected by the assessments is confidential medical information, and that the assessment screens out or tends to screen out persons with mental illness. 

Systemic Risks

For many school districts, employment assessments like TI offer a standardized experience for all applicants. While this "one size fits all" approach helps to reduce a school district's costs and may reduce the impact of overtly biased or discriminatory behavior, the inclusion of one or more potentially "defective components" in the assessments means that school districts face the risk that a finding of bias or discrimination of an assessment used in one school district will put all school districts that use the assessment at risk. Please see When the First Domino Falls: Consequences to Employers of Embracing Workforce Assessment Solutions.

These "defective components" in assessments may be either design defects (i.e., the adoption and use of certain personality models) or manufacturing defects (i.e., coding errors in the assessment software). The latter is analogous to the coding error at 23andMe that resulted in notices going out to some customers informing them that they had a chronic and life-shortening condition when they did not. Please see On Not Dying Young: Fatal Illness or Flawed Algorithm?

Each day a school district continues to use the TI assessment, there are more potential plaintiffs with claims against that school district.  Labor and employment laws like Title VII and the ADA, permit a school district to use a third party like Gallup to undertake the assessment of job applicants. The use of a third party, however, does not insulate a school district from any claims arising from the assessment usage. Under those laws, a school district is responsible (and liable) for any failures on the part of an assessment or assessment provider to comply with the provisions of those laws.

Risks to Kroger Shareholders

There is material risk to Kroger shareholders and other stakeholders that the company's use of the Kronos assessments over the past 7+ years has created millions of potential plaintiffs, including every person who has applied for a position at one of the many Kroger companies.

Kroger is being investigated by the Equal Employment Opportunity Commission (EEOC) as a consequence of multiple charges filed with the EEOC alleging non-compliance with labor and employment laws, including the Americans with Disabilities Act (ADA). The investigation has been ongoing for more than five years and has generated a number of district court and appellate court decisions as Kroger and the employment assessment company it uses, Kronos, have sought to avoid disclosing information about the assessment and its impact on persons protected by the ADA.

These actions by Kroger and Kronos have generally been unsuccessful; the appellate court decisions have sided with the EEOC and have ordered Kronos to disclose massive amounts of information, including:

1. Any and all documents and data constituting or related to validation studies or validation evidence pertaining to Kronos assessment tests purchased by Kroger, including but not limited to such studies or evidence as they relate to the use of the tests as personnel selection or screening instruments, even if created or performed for other customer(s).
2. The user’s manual and instructions for the use of assessment tests used by Kroger.
3. Any and all documents (if any) related to Kroger, including but not limited to correspondence, notes, and data files, relating to Kroger; its use of the assessment test; results, ratings, or scores of individual test takers; and any validation efforts made thereto.
4. Any and all documents discussing, analyzing or measuring potential adverse impact on persons with disabilities.
5. Any and all documents related to any and all job analysis performed by any person or entity related to any or all position(s) at Kroger.

 See Kroger and Kronos: Chaos and Disorder.

Claims and Damages

Claims against Kroger include:

• Claims that the Kronos assessment is an illegal pre-employment medical examination whose use is prohibited by the ADA.
• Claims that responses to the Kronos assessment, as a medical examination, are confidential medical information, subject to strict safeguards that Kroger has been recklessly ignoring.
• Claims that the Kronos assessment illegally screens out persons with mental disabilities in violation of the ADA.
• Claims that Kroger failed to ensure that the Kronos assessment results accurately reflect the skills, aptitude and other factors that the assessment purports to measure, rather than reflecting an applicant’s impairment.

As a medical examination, it is unlawful for the Kronos assessment to be administered by Kroger to any job applicant prior to a conditional offer of employment being made to that applicant.  Each time the Kronos assessment is so administered to any such applicant, Kroger is violating Title I of the ADA and subjecting itself to claims for back pay, front pay, compensatory damages and injunctive relief. Each person that has been subject to the illegal medical examination of Kroger under Article 1 of the ADA is entitled to seek compensatory and punitive damages of $600,000, in addition to back pay, front pay and recovery of attorneys’ fees.

An employer utilizing the Kronos assessment, like Kroger, is at risk for punitive damages where, as held by the court in Kolstad v. American Dental Association, the “employer has engaged in intentional discrimination and has done so with malice or reckless indifference to the federally protected rights of an aggrieved individual.”

Kroger’s unlawful use of the Unicru assessment subjects the company to material and publicly-disclosable liability and it shareholders to a potentially significant diminution in the value of their shares.

Ignoring EEOC Guidance

Reckless indifference may be show by a variety of methods, including an employer’s failure to comply with the EEOC guidance for testing and selection, pursuant to which employers are encouraged to:

1. Administer tests without regard to race, color, national origin, sex, religion, age, or disability;
2. Ensure that tests are properly validated;
3. Ensure that tests are job-related;
4. Ensure that tests are appropriate for the employer's purpose;
5. If selection procedures screen out a protected group, determine whether there is an equally effective alternative selection procedure that has less adverse impact and adopt that alternative procedure;
6. Keep abreast of changes in job requirements and update the test specifications or selection procedures accordingly; and
7.  Ensure that tests and selection procedures are not adopted casually by managers who know little about these processes.

In a court filing submitted several years ago in connection with its ongoing EEOC litigation, Kronos stated that “[n]o adverse impact or validation studies have been performed by Kronos, or to Kronos’ knowledge, by or for Kroger with respect to potential adverse impact on individuals with disabilities.” This filing was served on both the EEOC and Kroger.

Consequently, Kroger cannot confirm that the Kronos assessments have been administered without regard to disability, nor can it confirm that the Kronos assessments have been properly validated. Even more recklessly, Kroger has known that the Kronos assessments have not been validated with respect to potential adverse impact on individuals with disabilities and yet has continued to use the unvalidated assessment for years.

Disregarding Industry Standards

Reckless indifference may also be shown by an employer’s failure to meet industry standards. The general counsel of the Equal Employment Advisory Council, an employer association comprised of more than 300 major corporations and staffed by experienced lawyers and HR professionals with in-depth knowledge in handling EEO and affirmative action compliance issues, set out the following industry standards in a statement made to the EEOC in 2007:

• Tests are to be based on "objective" criteria
• Tests are attractive screening methods "when administered properly"
• A "carefully selected" test that is "properly validated" can provide a great deal of relevant information when "[u]sed in conjunction with other sources of information"
• A few "basic principles that EEAC member companies strive to apply:“
• Ensure each employment test has been properly validated.
• Ensure that the validity study is current and properly documented.
• Avoid overreliance on representations made by test manufacturers regarding test validity and suitability for a particular job.
• Conduct periodic audits of employment selection testing procedures to monitor for
• possible disparate impact
• significant changes in jobs
• outdated validity studies
• other potential problems

As set out in the many documents filed in the ongoing litigation among the EEOC, Kroger and Kronos, few, if any, of these basic industry principles have been followed by Kroger. For example:

1. Kroger was unable to provide the EEOC with any validation studies, whether current or not or properly documented or not; 
2. There was no information provided that demonstrated periodic audits by Kroger of the Kronos assessment to monitor for possible disparate impact or outdated validity studies; and
3. Kroger utilizes the Kronos assessment, on its own, as the basis for rejecting applicants – the assessment was not “used in conjunction with other sources of information.” 
4. Kroger seemingly relied solely on the representations of Kronos regarding test validity and suitability.

Kroger’s knowing failure to meet many the basic principles set out by the industry and turning a blind eye towards the known discriminatory impact of the Kronos assessment is demonstrably reckless.

He Said, She Said

A key element in both the EEOC and EEAC guidance is the responsibility of the employer (Kroger) to independently review the assessment and avoid reliance on the representations of the assessment provider.  Did Kroger conduct such an independent review? It would appear not.

In a letter to the EEOC dated February 9, 2009, sent by Kroger’s counsel it states, “Unicru, the company that developed the assessment [and subsequently acquired by Kronos], has informed … [Kroger] that the assessment has been fully and appropriately validated, and that there is no disparate impact.” That statement admits a reliance on Kronos's representations and indicates no independent validation effort on the part of Kroger.

The quoted sentence from the February 9, 2009 letter of Kroger's counsel is puzzling, given the Kronos statement mentioned previously that it is “impossible” to measure disparate impact on people who have disabilities. Which is it? How can Kronos assure Kroger that there is no disparate impact, as Kroger’s counsel stated to the EEOC, if, according to Kronos, it is impossible to measure disparate impact?

Ignoring Supreme Court Precedent

In a petition to revoke an EEOC subpoena filed on October 16, 2008, Kronos claims that there is “no known method … to ascertain adverse impact against the entire generic category of disabilities.” As noted above, such a claim is a red herring. Given the wealth of scholarship and data concerning the interrelationship between the FFM model and diagnosing mental illness (please see ADA, FFM and DSM), it seems more likely that Kronos was unwilling to spend the money to determine the impact of the Unicru assessment on persons with disabilities.  Either that, or Kronos knows what the results would demonstrate – that the assessment is a medical examination and its administration by employers, including Kroger, results in disparate treatment of, and has a disparate impact on, persons with mental illness.

As evidenced by the scholarship on the use of the FFM to diagnose mental illness previously discussed, there are no barriers to testing the use of the FFM in the Kronos assessment to determine its impact on persons with mental illness. As noted by the authors of “Veiled Barriers: Pre-EmploymentTesting and Disability” published by the Journal of Rehabilitation Administration in 2006:

[S]tudies of sub-groups, such as individuals with mental illnesses or cognitive impairments could be conducted to determine the potential, and perhaps likelihood for, pre-employment test results unfairly penalizing these individuals in the employee selection and hiring stages …

In Albemarle Paper Company v. Moody, the Supreme Court addressed a case in which an employer implemented a test (Wonderlic) on the theory that a certain verbal intelligence was called for by the increasing sophistication of the plant's operations. The company made no attempt to validate the test for job-relatedness, and simply adopted the national "norm" score as a cut-off point for new job applicants. The Supreme Court cited the Standards of the American Psychological Association and pointed out that a test should be validated on people as similar as possible to those to whom it will be administered. The Court further stated that differential studies should be conducted on minority groups wherever feasible.

Substitute (i) Kroger for Albemarle Paper Company and (ii) Kronos assessment for the Wonderlic test in the Albemarle Paper Company decision and one sees few, if any, differences. Like Albemarle Paper Company, Kroger made no attempt to validate the test for job-relatedness and arbitrarily adopted cut-off points. Like Albemarle Paper Company, neither Kroger nor Kronos conducted validation studies on persons with mental illness.

Kroger’s failure to follow Supreme Court precedent that (i) requires validation of an employment test (Unicru assessment), (ii) prohibits arbitrary cut-off points, and (iii) requires differential studies of the impact of the test on protected classes (persons with mental illness)? Reckless.