AI Recruiting Academy › Module 4: Risk, law and fairness
The short answer
An AI bias audit checks whether a hiring tool selects or scores one group of people at a noticeably lower rate than another. The core test compares selection rates across sex, race and ethnicity groups and turns them into an impact ratio. A ratio below 0.8, the long standing four fifths rule, is a warning sign worth investigating.
New York City requires an independent bias audit every year for automated tools used in hiring or promotion there, with a public summary of the results. Elsewhere an audit is not always required by law, but it is the best evidence you have that a tool treats candidates fairly.
What does a bias audit actually measure?
It measures outcomes, not intentions. The auditor takes real or test data from the tool, splits candidates into groups, and asks two questions: what share of each group did the tool move forward, and how does each group’s rate compare to the group that did best?
A good audit then goes further and asks why. If one group falls behind, is it because of something the job truly needs, or because of a proxy such as a zip code or a gap in employment? Lesson 1 covers where those proxies come from.
How is the impact ratio calculated?
Take each group’s selection rate and divide it by the selection rate of the most selected group. Here is a simple example for a resume screening tool:
| Group | Applicants | Passed the screen | Selection rate | Impact ratio |
|---|---|---|---|---|
| Group A | 400 | 200 | 50% | 1.00 |
| Group B | 300 | 135 | 45% | 0.90 |
| Group C | 200 | 70 | 35% | 0.70 |
Group C’s ratio of 0.70 falls below 0.8. Under the four fifths rule in the federal Uniform Guidelines on Employee Selection, that is generally treated as evidence of adverse impact. It is not proof of discrimination on its own, and small numbers can mislead, but it is the point where you stop and look closely.
When a tool gives scores rather than a pass or fail, audits use a scoring rate instead: the share of each group that scores above the median.
What does New York City’s law require?
New York City’s Local Law 144, enforced since July 2023 by the Department of Consumer and Worker Protection, covers automated employment decision tools used to substantially assist or replace discretionary decisions on hiring or promotion for jobs in the city.
| Requirement | What it means |
|---|---|
| Independent audit | Done by someone with no financial interest in the tool, within one year before use and every year after |
| Groups tested | Sex, race and ethnicity, and each combination of them |
| Public summary | Audit date, data used, selection or scoring rates and impact ratios, posted on your website |
| Candidate notice | Tell candidates the tool is used and what it assesses, with a way to request an alternative or accommodation |
| Penalties | Up to $1,500 per violation, and each day of use without an audit can count separately |
Enforcement has been weak so far. A December 2025 audit by the New York State Comptroller, summarized by DLA Piper, called it “ineffective”: the city reviewed 32 posted audits and found one problem, while the Comptroller found at least 17 possible violations in the same set. The city agreed to tighten enforcement, so employers should expect closer checks.
Who has to do the audit, the employer or the vendor?
Under the New York City law, the duty sits with the employer or employment agency using the tool. A vendor can pay for an audit and share it, and many do, but the employer is the one who must make sure an audit exists, covers its use, and is posted. If the vendor’s audit used other companies’ data, check that the rules allow it for your situation and that it is recent.
California’s employment regulations, in force since October 2025, do not require an audit, but they say bias testing, or the lack of it, is relevant evidence in a discrimination claim, according to Seyfarth. In practice that makes testing a defense you want to have.
What should you ask a vendor about bias testing?
| Question | A good answer sounds like |
|---|---|
| When was the last audit, and who did it? | A named independent firm, within the last twelve months |
| Can I see the full results? | Yes, with rates and ratios for every group, not just a pass badge |
| What data was used? | Real candidate data from similar roles, with the sample size stated |
| Which groups were tested? | At least sex, race and ethnicity; ideally age and disability too |
| What happened when a gap was found? | A clear description of the fix and a retest |
| Can we test on our own data? | Yes, with exports or a built in fairness report |
What are the limits of a bias audit?
It is a snapshot. Your applicant pool, job mix and the tool itself change. A tool that passed last year can drift.
It tests limited groups. The New York City audit does not cover age or disability, both common sources of hiring claims.
Small numbers mislead. With only a few candidates in a group, one hire can swing the ratio. Good auditors flag groups too small to judge.
Passing is not permission. An audit shows outcomes are within a range; it does not prove the tool measures anything useful for the job.
Quick answers
Do I need a bias audit if I am not in New York City?
Not always by law, but audits are the strongest evidence you can have if a candidate challenges a rejection. Laws in other states are moving toward notice and testing duties; Lesson 3 has the map.
Does a keyword filter in my ATS need an audit?
Under New York City’s rules it depends on how much weight the output carries. If the tool’s result is relied on alone or outweighs other factors, it likely counts. When in doubt, test it anyway.
How much does an audit cost?
It varies with the number of tools and the data involved. Ask the vendor first, since many already commission one and share the summary.
Can I run a basic check myself?
Yes. Export pass rates by group at each stage and apply the four fifths calculation above. It will not replace an independent audit where one is required, but it shows you where to look.
This lesson explains the law in plain English. It is not legal advice; check your own situation with an employment lawyer.