When Algorithms Discriminate Without Asking About Disability

 Employment Testing: Failing to Make the Grade — Ten Years Later, Part 2

Suppose an employer uses an algorithm to rank job applicants. The system does not ask about race, age, gender, or disability. It uses employment history, education, work experience, and other seemingly neutral information.

Does that mean the system cannot discriminate?

Not necessarily.

One of the central concerns raised in this blog more than a decade ago was that employment-screening systems can disadvantage protected groups without explicitly identifying—or intentionally targeting—those groups.

The problem becomes particularly important when automated systems use information that may serve as a proxy for a protected characteristic.

The Hidden Meaning of Neutral Data

Consider employment gaps.

An applicant may have been out of the workforce while receiving treatment for depression, recovering from a psychiatric hospitalization, or managing another disability. Another may have taken time away to care for a family member.

An automated screening tool might treat an extended employment gap as a negative indicator, even when the applicant possesses all the qualifications necessary for the position.

Similarly, an assessment might penalize applicants whose work histories involve frequent job changes, unconventional educational paths, or periods of reduced employment.

These characteristics may have legitimate relevance to some positions. But their relevance should be established rather than assumed.

The central concern is not whether an algorithm uses a particular data point. It is whether the criterion is meaningfully related to job performance and whether its use unlawfully disadvantages qualified applicants.

Learning From Yesterday's Employees

In its August 2013 discussion of Kroger and Kronos, this blog examined an assessment system that reportedly used neural networks to identify characteristics associated with an employer's successful employees.

The problem was straightforward: What happens when an algorithm learns from a workforce shaped by past hiring practices?

If an employer historically favored particular educational backgrounds, employment patterns, personality characteristics, or demographic groups, a system trained on those employees may reproduce those preferences.

The resulting recommendations may appear objective because a computer generated them.

But mathematical consistency is not the same as fairness.

A system can apply the same rule to every applicant and still produce discriminatory outcomes.

Different Legal Questions

Not every statistical correlation involving a protected group constitutes unlawful discrimination.

Under Title VII, an employment practice producing a disparate impact on a protected group may require the employer to demonstrate that the practice is job-related and consistent with business necessity. Other requirements and defenses may also apply.

The ADA has separate provisions addressing qualification standards and selection criteria that screen out individuals with disabilities, subject to the applicable statutory standards.

And an algorithm that indirectly disadvantages applicants with disabilities does not necessarily constitute a prohibited medical examination.

These distinctions matter. The legality of an automated screening system depends on its design, operation, justification, and actual effects.

What Should Employers Be Asking?

Before using an automated applicant-screening tool, employers should understand:

• What information does the system evaluate, and why?

• How were its criteria selected and validated?

• Do those criteria accurately measure the abilities required for the particular job?

• Could they disadvantage applicants with disabilities or other protected characteristics?

• Can applicants obtain appropriate accommodations?

• Are the results monitored over time, including after software changes?

These are not merely questions for software developers. They are questions for employers, lawyers, human resources professionals, and executives responsible for hiring practices.

In 2016, this blog's final post called attention to the need to measure and assess artificial intelligence and decision-support systems.

Ten years later, that responsibility is no less important.

An algorithm's failure to ask about disability does not establish that its decisions are free from disability discrimination.

Next: Why using a third-party screening vendor may create legal risks far beyond a single employer.