Systemic Risks for School Systems Employing TeacherInsight

According to Gallup Inc., its TeacherInsight (TI) assessment is an automated online interview used by many school districts to help those schools identify the best potential teachers. School districts across the country use the TI assessment as part of their teacher application and selection process.

Need to Independently Review Assessment.

In The (Non)Predictive Ability of the Gallup TeacherInsight Assessment, there is discussion of a study of the TI assessment undertaken by Michael T. Novotny. The Novotny study includes the following recommendations: 

School districts that are using, or considering the use of, the TI should require the Gallup Organization to show them the research in support of the instrument. The lack of independent research on the TI and Gallup’s unwillingness to publish their own research does not help support the credibility of the TI. If Gallup was to publish their own studies, then independent researchers could attempt to replicate those studies in order to either confirm or contradict their findings regarding the validity of the TI. Most large school districts have a research department with the capability of conducting such research.

As discussed under the headings "Ignoring EEOC Guidance" and "Disregarding Industry Standards" in Risks to Kroger Shareholders, a key element in guidance from both the EEOC and EEAC (an employer association) is the responsibility of the employer (e.g., local school district) to independently review the hiring assessment(s) it uses and to avoid reliance on the representations of the assessment provider (e.g., Gallup).

Accordingly, as Novotny recommends, school districts that are using, or considering the use of, the TI assesment should require Gallup to show them the research in support of the instrument. The Gallup-provided research should include information on the ability of the TI assessment to predict teacher performance, as well as validity studies undertaken by or on behalf of Gallup and evidence that the assessment does not discriminate against individuals who are members of classes protected by laws like Title VII and the ADA. 

What's Missing?

According to Gallup, a TI interview development study, originally completed in January 2002, demonstrated content, construct, and criterion-related validity as well as fairness across classifications of race, gender, and age. There are a number of other "protected classes," including persons with disabilities (including physical, developmental and mental disabilities), national origin, and religion, that are missing from Gallup's list.

As an example, Dr. Melanie Schneider has identified three primary concerns relevant to nonnative and bilingual speakers of English (raising issues of discrimination on the basis of national origin): 

• the timed nature of the TI assesment, 
• possible inequalities associated with limited access to computers or the Internet, and
• little perceived consideration for cultural and linguistic.

Dr. Schneider writes:

As mentioned earlier, parts of the TI contain timed questions. Unlike other standardized tests, there is no accommodation for applicants for whom English is a second or additional language or others who may have a disability that prevents a rapid-fire response. Waiting too long to respond to a question results in a missing response, which counts against an applicant’s total score. Although the timed nature of some types of questions affects all applicants who take the TI, nonnativespeakers of English are especially penalized when speed of response is required. 

Linked to the timed format of the TI, which may disadvantage some nonnative speakers of English, are possible inequalities due to the digital divide between low-income and middle-income students. ... English language learners in the United States are more likely to come from lower income families than their native-English-speaking peers. Inequalities between low-income and middle-income children in the use of and access to computers and the Internet have been well documented. ... Even among undergraduate students, values, attitudes, and beliefs about access to and facility with certain technologies may disadvantage certain groups of students, such as those from low-income immigrant families. 

Finally, a third concern for bilingual speakers of English is the belief that a singular view of talents characterizes successful teachers. Assuming that there is a single, preferred set of values, attitudes, beliefs, and behaviors associated with teacher success in the classroom ignores the role of culture in teaching and assessment

In Albemarle Paper Company v. Moody, the Supreme Court addressed a case in which an employer implemented a test (Wonderlic) on the theory that a certain verbal intelligence was called for by the increasing sophistication of the plant's operations. The company made no attempt to validate the test for job-relatedness, and simply adopted the national "norm" score as a cut-off point for new job applicants.

The Supreme Court cited the Standards of the American Psychological Association and pointed out that a test should be validated on people as similar as possible to those to whom it will be administered. The Court further stated that differential studies should be conducted on minority groups wherever feasible.

Ongoing Investigations of Assessments for ADA Compliance by EEOC

For more than six years, the EEOC has been investigating Kroger and Kronos, Kroger's assessment provider.  The investigation focuses on whether the Kronos assessment illegally screens out persons with mental illness. Please see Kroger and Kronos: Chaos and Disorder.

The case arose from a charge filed by Vicky Sands with the EEOC in 2007. The EEOC has converted Ms. Sands' investigation into a systemic investigation and has since started at least two additional systemic investigations involving companies using the Kronos assessment.

The EEOC investigations involve claims that the Kronos assessment is an illegal pre-employment medical examination, that the data collected by the assessments is confidential medical information, and that the assessment screens out or tends to screen out persons with mental illness. 

Systemic Risks

For many school districts, employment assessments like TI offer a standardized experience for all applicants. While this "one size fits all" approach helps to reduce a school district's costs and may reduce the impact of overtly biased or discriminatory behavior, the inclusion of one or more potentially "defective components" in the assessments means that school districts face the risk that a finding of bias or discrimination of an assessment used in one school district will put all school districts that use the assessment at risk. Please see When the First Domino Falls: Consequences to Employers of Embracing Workforce Assessment Solutions.

These "defective components" in assessments may be either design defects (i.e., the adoption and use of certain personality models) or manufacturing defects (i.e., coding errors in the assessment software). The latter is analogous to the coding error at 23andMe that resulted in notices going out to some customers informing them that they had a chronic and life-shortening condition when they did not. Please see On Not Dying Young: Fatal Illness or Flawed Algorithm?

Each day a school district continues to use the TI assessment, there are more potential plaintiffs with claims against that school district.  Labor and employment laws like Title VII and the ADA, permit a school district to use a third party like Gallup to undertake the assessment of job applicants. The use of a third party, however, does not insulate a school district from any claims arising from the assessment usage. Under those laws, a school district is responsible (and liable) for any failures on the part of an assessment or assessment provider to comply with the provisions of those laws.