Posted by 
We are surrounded by algorithms – they decide what we see in our Facebook Feeds, screen our job applications, calculate probabilities and drive facial recognition software. While they are faster and sometimes more accurate than us at processing large amounts of data, there is one area where they can be just as flawed as humans – they are biased.
While it sounds strange to assign an inanimate object with attributes usually reserved for people, the core of AI is, ultimately, human. It is humans that design it, humans that feed it information to train it, and humans that decide what the algorithm is going to be used for. And in every step, they are inadvertently shaping the bias that one day might cost you a job interview or an extra hour at the airport due to an AI-powered, racially-profiled secondary search.
In what ways are algorithms biased?
In 2014 Amazon started working on a computer program that would screen job applications and suggest the top candidates. The program was trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period and since technology tends to be a male-dominated industry, the data contained a disproportionate amount of male-submitted resumes vs. female. From the data, the program taught itself that male candidates were preferred and started to penalize resumes that contained the word “women’s” as well as downgrade candidates who attended all-women’s colleges.
Tools like Google Translate learn languages by crawling the web and analyzing billions of words, word combinations and sentences to determine how a specific word should be translated. I recently read about sexism in some of the translations, so I decided to test it myself. My native language, Croatian, has gendered nouns, so when it comes to professions we will have two versions of the same word – a male and a female (for example, a male doctor is liječnik while the female is liječnica). I started putting in different professions into Google Translate and its bias aligned with common stereotypes. Professions such as doctor, lawyer and engineer were translated in the male form of the word, while kindergarten teacher, secretary and cleaner were translated in the female form.
While some cases of algorithm bias are more annoying than dangerous, sometimes algorithm bias can have life-altering consequences. In 2016, ProPublica released a report on algorithm bias in software used in US courtrooms to rate a defendant’s risk of committing future crimes. Their research found that the software was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants while white defendants were mislabeled as low risk more often than black defendants. According to their research, the defendants whose risk rate the algorithm marked as higher were more likely to be sentenced to longer prison sentences regardless of circumstances or facts. The reason this particular program is biased is that it was fed decades’ worth of police records that are riddled with systemic racism.
The need for algorithmic transparency and accountability
One of the major issues surrounding algorithm bias is the lack of transparency. We know that the programs are making biased decisions, but we don’t know how they came to those conclusions. Better yet, sometimes we are not even aware algorithms are being used to make decisions about us.
Algorithms are often developed by private companies that do not disclose their proprietary secrets – even when their methods and results are being questioned. The aforementioned risk rating system used in courtrooms was developed by a private company, Northpointe, and sold to the US judicial system. Neither the defendants whose lives the results are altering nor the judges making the rulings based on them know how the system works, which types of data it uses, or how it comes to its predictions.
As with most things digital, lawmakers are lagging behind the rate at which new algorithm-driven technologies are being developed and are struggling to come up with a framework that would ensure proper regulation of algorithmic systems. Various institutions and agencies are trying to address this issue (for example, the European Centre for Algorithmic Transparency (ECAT) but in the meanwhile, it is unclear where the responsibility for negative outcomes from algorithms lies.
Who is at fault when a person becomes radicalized from being in a social media filter bubble where they only receive content that an algorithm deemed suiting or when someone is racially discriminated against? Is it the coder, the data set that was fed to the AI or the company that decided to use it?
Ultimately, there is a great need for regulation and openness when it comes to programs that have more and more impact on our lives every day. And, same as how the machines are learning about us, it is important that we learn about them too so that we as individuals can have a better understanding of how our data is used and what our rights are.
Image credit: rawpixel.com/freepik.com
