There are two doctrines in anti-discrimination law: disparate treatment, and disparate impact. Let's take a minute to look at each of these:
- Disparate treatment: This is one kind of unlawful discrimination. Intentionally discriminating against ZIP codes with the hope of discriminating against race is not legal. Disparate treatment problems have less to do with the algorithm and more to do with the organization running it.
- Disparate impact: This can be a problem if an algorithm is deployed that has a different impact on different groups, even without the organization knowing about it. Let's walk through a lending scenario in which disparate impact could be a problem. Firstly, the plaintiff must establish that there is a disparate impact. Assessing if there's a disparate impact is usually done with the four-fifths rule, which says that if the selection rate of a group is less than 80% of the group, then it is regarded as evidence of adverse impact. If a lender has 150 loan applicants from group A, of which 100, or 67%, are accepted, and 50 applicants from group B, of which 25 are accepted, the difference in selection is 0.5/0.67 = 0.746, which qualifies as evidence for discrimination against group B. The defendant can counter this by showing that the decision procedure is justified as necessary.
After this is done, the plaintiff has the opportunity to show that the goal of the procedure could also be achieved with a different procedure that shows a smaller disparity.
Note
Note: For a more in-depth overview of these topics, see Moritz Hardt's 2017 NeurIPS presentation on the topic at http://mrtz.org/nips17/#/11.
The disparate treatment doctrine tries to achieve procedural fairness and equal opportunity. The disparate impact doctrine aims for distributive justice and minimized inequality in outcomes.
There is an intrinsic tension between the two doctrines, as illustrated by the Ricci V. DeStefano case from 2009. In this case, 19 white firefighters and 1 Hispanic firefighter sued their employer, the New Haven Fire Department. The firefighters had all passed their test for promotion, yet their black colleagues did not score the mark required for the promotion. Fearing a disparate impact lawsuit, the city invalidated the test results and did not promote the firefighters. Because the evidence for disparate impact was not strong enough, the Supreme Court of the United States eventually ruled that the firefighters should have been promoted.
Given the complex legal and technical situation around fairness in machine learning, we're going to dive into how we can define and quantify fairness, before using this insight to create fairer models.