Algorithmic Bias

A bias is an inclination to prefer or disfavor an individual, group, idea, or thing. Biases against people based on their religion, race, socioeconomic status, gender identity, or sexual orientation are particularly unfair and therefore especially problematic. But all human beings hold such biases both consciously and unconsciously. Because human bias has led to unfair discrimination in hiring, promoting, housing, health care, lending, criminal sentencing, and many other areas of human endeavor, many hoped that replacing human decision-makers with computers would remedy, or at least minimize, such bias.  But we create the algorithms that guide computers’ decision-making, so those algorithms often reflect our biases. This is one form of algorithmic bias, and it arises in the design, testing, and application of computer systems.

Another form of algorithmic bias can happen with artificial intelligence where computers create their own code after being trained on vast amounts of data. These AI systems “learn” with training data and follow the principle of “garbage in, garbage out.” So, if an AI-based system is fed faulty or incomplete training data, its predictions will also be faulty.

Unfortunately, examples of algorithmic bias abound. For instance, one company built an AI-based computer system to handle its hiring processes. But the system was trained on a database from a 10-year period where the resumes were predominantly from white men, so the algorithm penalized resumes that were submitted containing the word “women’s.”

Another company created an algorithm used by judges to determine bail and sentencing decisions that systematically discriminated against people of color. And yet another algorithm used for clinical evaluations made black patients appear healthier than they were, which kept those patients from being fairly placed on the national kidney transplant waitlist.

The hazards of algorithmic bias are perhaps most obvious in the development of AI-based facial recognition software. These early AI systems were fed an overrepresentation of pale and male faces. The training data failed to represent the full range of human diversity. Consequently, facial recognition systems perform better in identifying men and people with lighter skin tones than in identifying women and people with darker skin tones. In fact, some facial recognition systems tagged black women as “men.” So although we tend to assume that using AI will lead to fair and neutral decisions, this is clearly a dangerous assumption.

Also of concern is the “black box” problem. We don’t know how Deep Learning AI systems make their decisions – the process is invisible to us. This lack of transparency is troubling and can lead to products that reinforce stereotypes and exacerbate explicit and implicit biases.

Fairness advocates urge companies to guard against algorithmic bias by evaluating their training data, making that data subject to public evaluation, disclosing the accuracy of their systems’ decisions, and allowing third-party auditing of their systems and tools.

Other advocates for ethical AI, like the Algorithmic Justice League founded by Dr. Joy Buolamwini, also encourage algorithmic justice. Dr. Buolamwini argues for inclusive coding and design teams, inclusive data sets, and more thoughtful consideration of the implications of AI-based systems in general.

So, while we can all agree that minimizing algorithmic bias is desirable – as with human bias – eliminating computer-based bias will be challenging. But with an emphasis on algorithmic justice, and with transparent AI development and design, it’s possible that instead of being an opponent, AI can become an ally in our efforts to create a just society.