Incident · Mid
Amazon (2018): recruiting AI penalizes women
Real incident. Key points: historical bias in labels, protected-attribute proxies, and fairness audit.
Short answer
Real incident. Key points: historical bias in labels, protected-attribute proxies, and fairness audit. According to Reuters, in a report reproduced by CNBC, machine learning specialists at Amazon discovered a big problem: the company's new recruiting engine did not like women.
01 · CONTEXT
A machine for picking résumés
According to Reuters, in a report reproduced by CNBC, machine learning specialists at Amazon discovered a big problem: the company's new recruiting engine did not like women. Amazon declined to comment on the engine or its challenges; the account comes from people familiar with the project, who spoke on condition of anonymity.
The team had been building programs to review résumés since 2014, with the goal of mechanizing the search for talent. The experimental tool gave candidates scores from one to five stars, and the stated goal was an AI that could crawl the web and find candidates worth recruiting.
The work was handled by a team at the engineering hub in Edinburgh, which grew to about a dozen people.
02 · THE SYSTEM
From résumé history to star score
In the simplified design below, there are two inputs: the résumés already received, which serve as examples for training, and the candidates found on the web, which the models score. The recruiter sees the score; the diagram is our simplification of what the report describes.
The candidate search feeds the models, which return to the recruiter a score from one to five stars.
The models were trained by observing patterns in résumés submitted to the company over a ten-year period.
The group created 500 models, focused on specific job functions and locations.
Recruiters looked at the tool's recommendations, but never relied solely on those rankings.
03 · THE BIAS
What the model learned from the past
In 2015, the company realized the system was not rating candidates for developer and other technical posts in a gender-neutral way.
The report explains the origin: the models were trained on résumés submitted over a ten-year period, and most came from men, a reflection of male dominance in the tech industry. In practice, the system taught itself that male candidates were preferable.
| Signal in the résumé | How the model reacted |
|---|---|
| The word women's | Penalized the résumé |
| All-women's colleges | Downgraded the graduates |
| Verbs more common on male engineers' résumés | Favored those who used them |
| Skills common among IT candidates | Received little weight |
The training examples came mostly from men.
From these examples, the system taught itself that male candidates were preferable.
The models penalized terms such as the word women's and downgraded graduates of all-women's colleges.
The scores reached the recruiter already carrying this pattern, even though the recruiter never relied solely on them.
Two concepts help in reading the case. Historical bias is the pattern of the past that stays recorded in the training data; a protected-attribute proxy is a signal that accompanies that attribute even without naming it. The names of the concepts are ours.
04 · THE FIX
Neutralizing terms was not enough
Amazon edited the programs to make them neutral to those specific terms. According to the sources, this did not guarantee that the machines would not devise other ways of sorting candidates that could also be discriminatory.
Gender bias was not the only problem. Flaws in the data underpinning the models' judgments meant that unqualified candidates were often recommended for all kinds of jobs, and, with results close to random, Amazon shut the project down.
According to anonymous sources, the team was disbanded because executives lost hope in the project. The report places this by the start of the year before the publication.
05 · LESSONS
Fairness as a design requirement
The report presents the case as a study of the limitations of machine learning. A Carnegie Mellon professor said that guaranteeing a fair, interpretable, and explainable algorithm was still a long way off, and a LinkedIn executive said he would not trust any AI system today to make a hiring decision on its own.
There is also a difficulty of external oversight: critics acknowledge that it can be extremely hard to sue an employer over automated screening, because candidates may never know it was used.
From Amazon, the report says part of the learning was put to use, and that a heavily reduced version of the engine does rudimentary tasks, such as removing duplicate profiles. Another source said a new team in Edinburgh was formed to try again, this time with a focus on diversity.
The diagram below is our own reference design, not what Amazon built. It shows where a fairness audit could fit in: on the data, before training, and on the scores by group, after it.
The data goes through an audit before training, which measures composition and looks for signals that reveal a protected attribute.
Only the audited data reaches training, and the resulting model is treated as a candidate, not as a final version.
The scores by group go to a metrics dashboard that compares the distribution across groups of candidates.
A disparity between groups raises an alert that goes back to the audit, closing the loop.
The recruiter uses the score as support and decides with other evidence.
Frequently asked questions
What is Amazon (2018): recruiting AI penalizes women?
Real incident. Key points: historical bias in labels, protected-attribute proxies, and fairness audit.
What does the Amazon (2018): recruiting AI penalizes women design cover?
It is organized in these sections: A machine for picking résumés, From résumé history to star score, What the model learned from the past, Neutralizing terms was not enough, and Fairness as a design requirement.
How long is Amazon (2018): recruiting AI penalizes women and what level is it?
It takes about 12 minutes to read and is rated Mid level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.