How Does Bias Get Into AI Hiring Tools?

AI Recruiting Academy › Module 4: Risk, law and fairness

What does bias in an AI hiring tool actually mean?

It means the tool’s scores, rankings or rejections fall unevenly on a protected group for reasons that have nothing to do with doing the job. A tool can be accurate on average and still be biased, because it can be right for most people and consistently wrong for one group.

In US employment law this is called disparate impact. A practice that looks neutral, applied to everyone the same way, is still unlawful if it screens out a protected group disproportionately and the employer cannot show it is job related and necessary. You do not need a biased person anywhere in the process.

Where does the bias come from?

What happened with Amazon’s recruiting tool?

Amazon built an experimental tool to rate resumes, trained on ten years of resumes submitted to the company. Because most of those came from men, the system taught itself that male candidates were preferable. According to Reuters’ 2018 report, it penalized resumes containing the word “women’s”, as in “women’s chess club captain”, and downgraded graduates of two all women’s colleges. Engineers edited out those specific terms, but could not be sure the tool would not find other ways to sort candidates, and the project was dropped.

The lesson for recruiters: removing the obvious signal does not remove the bias. The pattern is in the data, and a model will find another route to it.

Do tools built on ChatGPT style models have the same problem?

Yes, in a different way. Large language models, which power most new screening and matching features, learned from huge amounts of human writing, associations included. A University of Washington study swapped names associated with white and Black men and women across more than 550 real resumes and had three language models rank them against over 500 job listings. The models favored white associated names 85% of the time and Black associated names 9% of the time. They never favored Black male associated names over white male associated names.

Results vary by model and task, and bias does not always run in the same direction. The point is that no one can assume a model is neutral. It has to be tested on your roles and your candidates.

Who is liable when an AI tool discriminates?

The employer, first. Using a vendor’s tool does not transfer your responsibility for the hiring decision. Vendors can be on the hook too. In Mobley v. Workday, job seekers allege Workday’s screening tools rejected them because of race, age and disability. The court let the age claim proceed as a nationwide collective in 2025, and in June 2026 it allowed disability and California state law claims against Workday itself to continue.

In 2023 the EEOC settled with iTutorGroup, whose software automatically rejected women 55 or older and men 60 or older. It was a simple programmed rule, not advanced AI, and it cost the company $365,000.

Federal enforcement has shifted since. An April 2025 executive order told federal agencies to stop pursuing disparate impact cases, but as Duane Morris explains, it does not stop private lawsuits, and states such as New York, California and Illinois keep disparate impact in their own laws. The risk has moved, not gone away.

What can a recruiter do about it?

Know what the tool looks at. Ask the vendor which inputs drive the score and whether names, photos, addresses, graduation dates or employment gaps are used.

Ask for test results. A serious vendor can show how selection rates compare across sex, race and age groups. Lesson 2 explains how that test works.

Remove what is not needed. Turn off fields that are not required for the job, and avoid hard knockout filters you cannot justify.

Keep a human in the loop. Review rejections, not just the shortlist. Spot check who the tool passed over.

Watch your own numbers. Compare pass rates by group at each stage over time. Bias can appear after launch as your applicant pool changes.

Offer an alternative. Give candidates who need an accommodation, such as someone who cannot do a timed or video assessment, another way through.

Quick answers

Is AI more biased than human recruiters?

Not necessarily. People are biased too. The difference is scale: one biased tool applies the same pattern to every applicant, every day, so a small skew becomes a large one fast.

If we remove names and photos, is the tool fair?

It helps, but it is not enough. Proxies such as schools, addresses, clubs and dates can carry the same signal. Testing outcomes is the only way to know.

Does the law only apply to fancy AI?

No. Anti discrimination law covers any selection method, from a simple keyword filter to an advanced model. The iTutorGroup case was a single age rule.

Can I use a tool the vendor says is bias free?

Treat that as a claim to check, not a fact. Ask for the method, the data, the date of the last test and the results by group.

This lesson explains the law in plain English. It is not legal advice; check your own situation with an employment lawyer.