Need to Measure and Assess Artificial Intelligence and Decision-Support Systems

Set out below are extracts from Artificial Intelligence is Hard to See, a post by Kate Crawford and Meredith Whittaker. AI refers to artificial intelligence and boldface text is taken from the article.

AI and decision-support systems are reaching into everyday life: determining who will be on a predictive policing ‘heat list’, who will be hired or promoted, which students will be recruited to universities, or seeking to predict at birth who will become a criminal by the age of 18. So the stakes are high. 

[T]here are no agreed-upon methods to assess the human effects and longitudinal impacts of AI as it is applied across social systems. This knowledge gap is widening as the use of AI is proliferating, which heightens the risk of serious unintended consequences. 

The core issue here isn’t that AI is worse than the existing human-led processes that serve to make predictions and assign rankings. Indeed, there’s much hope that AI can be used to provide more objective assessments than humans, reducing bias and leading to better outcomes. The key concern is that AI systems are being integrated into key social institutions, even though their accuracy, and their social and economic effects, have not been rigorously studied or validated. 

AI systems are being integrated into key social institutions, even though their accuracy, and their social and economic effects, have not been rigorously studied or validated.

There needs to be a strong research field that measures and assesses the social and economic effects of current AI systems, in order to strengthen AI’s positive impacts and mitigate its risks. By measuring the impacts of these technologies, we can strengthen the design and development of AI, assist public and private actors in ensuring their systems are reliable and accountable, and reduce the possibility of errors. By building an empirical understanding of how AI functions on the ground, we can establish evidence-led models for responsible and ethical deployment, and ensure the healthy growth of the AI field. 

If the social impacts of artificial intelligence are hard to see, it is critical to find rigorous ways to make them more visible and accountable. We need new tools to allow us to know how and when automated decisions are materially affecting our lives — and, if necessary, to contest them.

EEOC, Systemic Investigations, and Assessments

The Equal Employment Opportunity Commission (EEOC) issued a review of its systemic program titled "Advancing Opportunity" in July 2016. The review marks the 10th anniversary of EEOC's 2006 Systemic Task Force Report. 

According to a press release accompanying release of the review:

"EEOC has transformed its systemic program in the past decade by investing in staff, training, and technology to build systemic expertise in every EEOC district," reflected EEOC Chair Jenny R. Yang. These investments have produced a 250 percent increase in systemic investigations in the past five years. 

Highlighting EEOC's significant achievements in resolving systemic cases, the review reports a 94% success rate in systemic lawsuits. In addition, EEOC tripled the amount of monetary relief recovered for victims in the past five fiscal years from 2011 through 2015, compared to the monetary relief recovered in the first five years after the Systemic Task Force Report of 2006.  EEOC also tripled the rate of successful voluntary conciliations of systemic investigations from 21% in fiscal year 2007 to 64% in fiscal year 2015.  

EEOC's Successes in Systemic Litigation

 Regarding pre-employment assessments, the press release states:

EEOC's systemic investigations have also led to changes in hiring assessment screens that discriminated based on race, sex and disability. In a public conciliation with Target Corporation, EEOC found that four hiring assessments formerly used by the retailer were not job-related and consistent with business necessity as required by Title VII and the ADA. Target agreed to pay $2.8 million to resolve a Commissioner's charge of discrimination alleging the assessments affected thousands of applicants and agreed to ensure that future hiring screens were validated to prevent discrimination against future applicants.

In "EEOC Burnishes Systemic Successes and Intentions," Jackson Lewis, a management side labor and employment law firm, writes:

The EEOC believes that employers too often ignore its pronouncements. Therefore, the EEOC considers the best way to obtain compliance is to leverage its resources by making an example of certain employers through systemic enforcement and lawsuits. 

The EEOC defines systemic discrimination as pattern or practice, policy, or class cases where the discrimination has a broad impact on an industry, profession, company, or geographic location. 

According to the Jackson Lewis article, "The [EEOC review] provides clues to the agency’s intentions in aspirational statements and disclosures about the EEOC’s investments and nationwide teams." These include:

Tests. Like the EEOC’s challenges to background checks, the EEOC’s concern with tests and assessments is that these selection criteria have an unlawful disparate impact. The [EEOC review] lists only one recent success challenging an employer’s use of a test as a selection device. However, it makes several references to the EEOC’s interest in scrutinizing tests and assessments.

While the EEOC review only lists the public conciliation with Target Corporation noted above, as noted in a September 2014 cover story in the Wall Street Journal, there are at least two ongoing systemic investigations relating to the use of pre-employment assessments and claims under the Americans with Disabilities Act that the assessments unlawfully screen out persons with mental disabilities and that the assessments are illegal pre-employment medical examinations.

The EEOC review states:

Moving forward, EEOC will focus on three key areas in order to expand the agency's impact and better serve the public: 1) executing national strategies to address persistent and emerging systemic issues; 2) advancing solutions that promote lasting opportunity in the workplace; and 3) strengthening the agency's technology and infrastructure.

Persistent and emerging systemic issues include those listed as national priorities in the EEOC's Strategic Enforcement Plan (SEP). First on the list of national priorities in the SEP  is:

Eliminating Barriers in Recruitment and Hiring. The EEOC will target class-based intentional recruitment and hiring discrimination and facially neutral recruitment and hiring practices that adversely impact particular groups. Racial, ethnic, and religious groups, older workers, women, and people with disabilities continue to confront discriminatory policies and practices at the recruitment and hiring stages. These include exclusionary policies and practices, the channeling/steering of individuals into specific jobs due to their status in a particular group, restrictive application processes, and the use of screening tools (e.g., pre-employment tests, background checks, date-of-birth inquiries). Because of the EEOC's access to data, documents and potential evidence of discrimination in recruitment and hiring, the EEOC is better situated to address these issues than individuals or private attorneys, who have difficulties obtaining such information.

(Emphasis added) 

Online-Only Employment Application Processes Systematically Discriminate Against Poor

The use of online employment application processes as the sole means to apply for jobs systemically discriminates against persons of lower socioeconomic status, many of who are protected classes under equal employment opportunity (EEO) laws. The discrimination arises from the limited Internet access for many of those persons.

EEO laws prohibit discrimination against protected persons in regard to recruiting, the work environment, or any other term, condition, or privilege of employment.  Not only do the laws prohibit intentional discrimination, they also prohibit neutral policies that disproportionately affect protected persons and that are not related to the job and the needs of the business (e.g., requiring job applicants submit their application online through a web-based interface)..

The National Digital Inclusion Alliance released its list of the Top 25 Worst Connected U.S. Cities for Poor Households (households with incomes below $35,000) in September 2015. More than 50% of poor households in Birmingham, Buffalo, Chicago, Cleveland, Dallas, Detroit, Greensboro, Louisville, Memphis, Miami, New Orleans, St. Louis, Toledo, and Washington, D.C. have no Internet access. 

Top 25 Worst-Connected Cities Poor Households ACS 2014 from Angela Siefer

Detroit

A New York Times article from May 2015 notes that "Detroit has the worst rate of Internet access of any big American city, with four in 10 of its 689,000 residents lacking broadband." Many of the persons without broadband access - whether at their residences or on smartphones - rely on public libraries for Internet access

In Hope Village, a 100-block area of Detroit, half of the 5,700 residents live in poverty. Many are not getting basic digital literacy skills or access to educational resources for entry-level jobs, much less the growing number of jobs that require more tech skills and vocational certificates.

Julie Rice, a Hope Village resident for the last seven years, has found having limited web access a major obstacle in her search for full-time employment after losing her retailing management job more than two years ago. With a part-time job at a furniture store paying $10.88 an hour, Ms. Rice cannot afford a service to connect to the web, which can cost more than $70 a month. 

So Ms. Rice has made Hope Village’s public library, Parkman, her career center. She regularly comes on the five days the library is open to search retailing openings, arrange interviews and take employment tests. The library typically extends her time online over the one-hour session limit. Even so, during a recent online exam for a store manager job at Ann Taylor, she ran out of time and was locked out of the test.

Every day it becomes harder to find opportunities in Detroit without using the web. Sean Person, a Hope Village resident, has gone to stores more than a dozen times and asked to fill out paper applications, only to be told to apply online. Most listings on Michigan’s biggest private and public jobs site require email, uploads of resumes, and online assessments. 

Zappos

At Zapppos, the company has created "Zappos Insiders." According to the company:

Zappos Insiders are simply people who might want to work for Zappos someday… now, tomorrow or sometime down the road. It’s like a special membership for people who want to stay in touch with us, learn more about our fun, zany culture, know what’s happening at our company, get special insider perspectives and receive team-specific updates from the areas you’re most interested in. There is no better way to stay in-the-know and for us to get to know each other than by becoming an Insider.

What are the benefits?  

• Be the first to know about job opportunities in your desired job family

• Stay in-the-know about the latest news and happenings here at Zappos

• Chat with the Zappos recruiting team during our bi-weekly Tweetchats

• Gain exclusive access to online & in person events with current Zappos employees

People who do not have broadband access at home or on their phones are unable to obtain the benefits of being a Zappos Insider - they are the last to know about job opportunities, they are unable to stay in-the-know about the latest Zappos news, they cannot chat with Zappos recruiting teams during their bi-weekly Tweetchats, and  they are denied exclusive access to online events with current Zapppos employees.

Zappos' website has the following line:

Poke. Like. Share. Join the Conversation @InsideZappos

The only way to poke, like, share or join the conversation is with a broadband connection. Poor persons, and others with limited broadband connectivity seemingly need not apply.

As Aristotle wrote, “There is nothing so unequal as the equal treatment of unequals.” It is for this reason that employment laws like Title VII prohibit facially neutral policies (e.g.,  enrolling in Zappos Insider, necessitating broadband internet access, and using social media platforms with varying degrees of accessibility ) that disproportionately affect protected persons and that are not related to the job and the needs of the business.

Zappos' Insider may be a club that is "open" to everyone, but persons with lower socioeconomic status, disproportionately Blacks, Hispanics, persons with mental illness, and women will have a harder time being admitted and will be limited in their ability to use all the club has to offer.

See also Zappos Insider: The Death of Job Postings and the Rise of the Borg and Zappos: The Future of Hiring and Hiring Discrimination.

Knack

According to Knack, it is a game-based, science-driven, data-powered talent-matching platform.

Knack "games" like Wasabi Waiter, Dash Dash, Bomba Blitz and Meta Maze purportedly analyze every millisecond of player behavior, measuring conscientiousness, emotion recognition, and other attributes that, according to the company, academic studies show correlate with job performance. The games then score each player’s likelihood of becoming an outstanding employee.

Knack's assessments are based on games developed by the company that may be "played" on computers and mobile devices. According to the company:

Knack uses its breakthrough, scalable technology to provide disadvantaged and marginalized people around the world with an empowering gateway to social and economic mobility. Helping people tap into their true potential will increase their well-being, break persistent cycles of poverty, and make them hopeful about the future.

And yet Knack's fundamental structure - online games played by persons with broadband Internet connections over computers and mobile devices - further disadvantages those on the other side of the digital divide: those who do not have ready access to broadband, those who must rely on public libraries for limited Internet access.

See also Knack Testing Illegal Under ADA?