The (Non)Predictive Ability of the Gallup TeacherInsight Assessment

Gallup states that the TeacherInsight (TI) assessments have "been thoroughly researched and tested to be sure they identify potentially superior teachers." While it is to be expected that the company marketing the TI assessment would make such a statement, is there any independent support for the predictive ability of the TI assessment?

Establishing the predictive validity of an assessment usually requires that applicants who "pass" the assessment perform satisfactorily in practice, and those who do not pass do not perform satisfactorily in practice. The challenge with assessing the predictive validity of the TI assessment, however, is that teacher applicants who do not meet the cutoff score may not be hired. Consequently, support for the predictive validity of the TI assessment is determined by carefully documenting the evidence from teachers who “pass,” through correlational and regression analyses that compare performance on the target measure, TI scores, with one or more other measures.

Doctoral dissertations by Robert Jacob Koerner and Michael T. Novotny are two of the few independent, published research studies to examine the predictive validity of the TeacherInsight (TI) assessment:  that teachers who score higher on it will be more successful teachers.

Novotny Study


The Novotny study involved 527 teachers hired into a North Texas school district for the 2006-2007 school year. The study analyzed the relationships between the TI assessment scores and the eight Professional Development Appraisal System (PDAS) domain scores for those teachers. PDAS is the instrument used by the State of Texas for appraising its teachers and identifying areas that would benefit from staff development.

The Novotny study concluded that:

The TeacherInsight scores produced a statistically significant correlation with only one of the eight PDAS domain scores. However, even that correlation (r = 0.14) was weak. ... The findings do not support the ability of the TeacherInsight to identify more effective teachers, based on Professional Development Appraisal System scores. 

Koerner Study

The Koerner study examined the relationship between the TI assessment and student achievement as measured by the Texas Growth Instrument (TGI), which is an estimate of a student’s academic growth based on the Texas Assessment of Knowledge and Skills (TASK) scores for two consecutive years.More specifically, the study focused on the predictive validity of the TI assessment, that is, how well teacher TI scores predict student achievement gains, as measured by the TGI in reading, English language arts, and mathematics at the primary and secondary school levels. Participants in the study were 132 teachers from one Texas school district who taught reading, English language arts, or mathematics in Grades 3–11 during the 2005–2006 school year and had taken the online TI assessment.

According to Koerner:

The findings [of his study] provide little support to the validity of TeacherInsight in terms of its ability to predict student achievement scores and its usefulness as a tool for the selection of teachers by school systems. 

Wasting Resources, Eliminating Good Teachers

Both the Koerner and Novotny studies found statistically significant, but very weak, relationships with a small number of variables related to teacher success. Those very low correlations suggest a weak link at best between the TI assessment and student achievement and teacher effectiveness.

As Novotny states in the introduction to his study:

It is critical that schools and districts identify highly effective, highly qualified teachers to raise student achievement. School districts have limited resources such as time, money, and manpower to achieve this task. If standardized interview tools such as the TI are effective at identifying better teachers, the time and money spent on them are worthwhile. However, if these tools are not effective, then the time and money spent could be better utilized elsewhere. Furthermore, if the TI does not effectively identify better teachers it could be preventing good candidates from being hired or from being accepted into alternative certification programs.

Not only are there concerns that the TI assessment may waste resources and screen out good candidates, as set out in Systemic Risks for School Systems Employing TeacherInsight, the use of the TI assessment may also violate the non-discrimination provisions of labor and employment laws like Title VII and the ADA that are enforced by the U.S. Equal Employment Opportunity Commission.

As Dr. Melanie Schneider writes in TeacherInsight and the Selection and Hiring of Bilingual Speakers of English:

Should teachers, teacher educators, and professional teacher organizations passively accept the increasingly pivotal role of online assessment tools in the hiring process, or should they press for more research and a balanced selection process that includes multiples sources of information,including human interaction?

Zappos: The Future of Hiring and Hiring Discrimination?

Zappos has launched a new careers site and removed all job postings. Instead of applying for jobs, persons interested in working at Zappos will need to enroll in a social network run by the company, called Zappos Insiders. The social network will allow them to network with current employees by digital Q&As, contests and other means in hopes that Zappos will tap them when jobs come open.

"Zappos Insiders will have unique access to content, Google Hangouts, and discussions with recruiters and hiring teams. Since the call-to-action is to become an Insider versus applying for a specific opening, we will capture more people with a variety of skill sets that we can pipeline for current or future openings," said Michael Bailen, Zappos’ head of talent acquisition.

The idea is that recruiters will monitor the interactions, and alert promising applicants as job positions become available. "Our recruiters are focusing on proactive sourcing…so that we know EXACTLY who we want to interview once a position becomes available," Stacy Donovan Zapar, Zappos' Social Recruiting and Employer Branding Strategist, wrote in a posting explaining Zappos Insiders.

Ideally, the Zappos Insider strategy will result in an engaged, talented applicant pool from which the company can seamlessly fill positions. More importantly, at a company where cultural fit is 50 percent of the hiring equation, encouraging candidates and current employees to interact organically allows the company to gauge whether someone is a good fit before the hiring process officially begins, theoretically saving both time and money.

Zappos’ Bailen said he is surprised that Zappos appears to be the first to drop job listings. "We're hoping a lot of other companies jump on board," he said, but there are compelling reasons why Zappos may find itself alone.

Bias and Discrimination

In a comment to Donovan Zapar’s posting explaining the Zappos Insider initiative, Steve Kuzner asked:

Ultimately, the [Zappos Insider] policy makes the hiring process less transparent for job seekers, who will no longer even know there is an open position until they are deemed worthy to apply for it. While well-intentioned, such opacity provides a shield under which bias, conscious or not, can potentially flourish.

Donovan Zapar responded, in part, by acknowledging that there are laws prohibiting an employer from basing hiring decisions on the race, religion, national origin, gender, pregnancy, disability, genetic identity, or age of applicants (Protected Persons). That response is correct, as far as it goes, but it fails to address the broad scope of Title VII of the Civil Rights Act of 1964 (Title VII), its progeny (Age Discrimination in Employment Act (ADEA), Americans with Disabilities Act (ADA), Genetic Information Non-Discrimination Act (GINA), etc.), their implementing regulations, and court decisions over the past fifty years.

In addition to prohibiting discrimination in hiring, employment laws prohibit discrimination against Protected Persons in regard to recruiting, the work environment, or any other term, condition, or privilege of employment.  Not only do the laws prohibit intentional discrimination, they also prohibit neutral policies that disproportionately affect Protected Persons and that are not related to the job and the needs of the business.

Job requirements must be uniformly and consistently applied to all persons. Even if a job requirement is applied consistently, if it is not important for job performance or business needs, the requirement may be found unlawful if it excludes Protected Persons significantly more than others. Examples of potentially unlawful practices include: (1) soliciting applications only from sources in which all or most potential workers are of the same race or color; (2) requiring applicants to have a certain educational background that is not important for job performance or business needs; and, (3) testing applicants for knowledge, skills or abilities that are not important for job performance or business needs.

[Dis]Connection

Donovan Zapa wrote in her post:

In preparation for this day, we've built a brand new "Inside Zappos" social recruiting presence over the past couple of months on Twitter, Facebook, Instagram, LinkedIn and Pinterest …. We've used these social channels to share our company culture, our people, our campus, our events, the way we do business and also to get to know the people who might like to work for us someday.

Similarly, in response to the question, “How can I stand out from the pack and stay front-and-center in the Zappos Recruiters’ minds?” on the Zappos' Insider site, the company lists six ways to stand out, including: using Twitter, Facebook, Instagram, Pinterest and Google Hangouts; participating in TweetChats; following Zappos’ employees on various social media platforms; and, reaching out to Zappos’  “team ambassadors.” For the most part, all of the foregoing activities require broadband internet access and devices (tablets, smartphones, etc.) that run on those access networks.  A number of Protected Classes will be challenged by both the broadband access and social media participation requirements:

• As noted  by a PewResearch Internet Project Research report, African Americans have long been less likely than whites to have high speed broadband access at home, and that continues to be the case. Today, African Americans trail whites by seven percentage points when it comes to overall internet use (87% of whites and 80% of blacks are internet users), and by twelve percentage points when it comes to home broadband adoption (74% of whites and 62% of blacks have some sort of broadband connection at home).  
• The gap between whites and blacks when it comes to traditional measures of internet and broadband adoption is pronounced. Specifically, older African Americans, as well as those who have not attended college, are significantly less likely to go online or to have broadband service at home compared to whites with a similar demographic profile.
• According to the PewResearch Internet Project, even among those persons who have broadband access, the percentage of those using social media sites varies significantly by age:

Age	Social Media Participation
18-29	90%
30-49	78%
50-64	65%

Social medial participation is not solely a function of age. "Social media is transforming how we engage with customers, employees, jobseekers and other stakeholders," said Kathy Martinez, Assistant Secretary of Labor for Disability Employment Policy. "But when social media is inaccessible to people with disabilities, it excludes a sizeable segment of our population." Persons with disabilities (e.g., sight or hearing loss, paralysis), whether physical, mental, or developmental, face challenges accessing social media. Each of the social media platforms promoted by Zappos - Twitter, Facebook, Instagram, Pinterest, and Google Hangouts - have differing levels of support for those with disabilities (e.g., close captions or real live captions on image content that utilize sound/voice). 

Career transition and talent development consulting firm Lee Hecht Harrison asked hundreds of job seekers via an online poll, “How active are you on social networking sites?” Only forty-eight percent said they're active on a daily basis, with an additional nineteen percent saying they log on about two or three times per week. This leaves thirty-three percent who use social media sites one to three times a month, if at all. 

As Aristotle wrote, “There is nothing so unequal as the equal treatment of unequals.” It is for this reason that employment laws like Title VII prohibit facially neutral policies (e.g.,  enrolling in Zappos Insider, necessitating broadband internet access, and using social media platforms with varying degrees of accessibility ) that disproportionately affect Protected Persons and that are not related to the job and the needs of the business. Zappos' Insider may be a club that is "open" to everyone, but African Americans, persons over forty, and persons with disabilities, among others, will have a harder time being admitted and will be limited in their ability to use all the club has to offer.

Donovan Zapa, in her posting announcing Zappos' Insider, wrote:

[O]ur recruiters are focusing on proactive sourcing, driving people to join our Insider program, having two-way conversations, meeting people, networking, chatting with Insiders, answering questions, engaging on social media, employer branding and proactive pipelining so that we know EXACTLY who we want to interview once a position becomes available. It's old-school recruiting, made new and fresh again. 

It is likely that "old school recruiting" was part of the reason that laws like Title VII, ADEA, ADA and GINA were enacted. Persons making hiring decisions knew exactly who they wanted to interview and that did not include persons of color, women, older Americans, persons with disabilities, and persons of certain faiths. While likely not the intent, Zappos' Insider platform makes discrimination against "those persons" new and fresh (and painful and illegal) again.