AI Recruiting Academy › Module 5: Putting AI to work
The short answer
Start from the problem, not the demo. Write down the one task you want the tool to fix and how you will know it worked. Then test it on your own jobs and your own data in a short pilot, check what it does without asking you, ask for proof of bias testing and security, and confirm the rules where you hire.
A tool that looks brilliant in a demo but cannot explain its choices, will not let you switch off automatic rejections, or trains on your candidate data is not a bargain at any price.
Why do so many AI pilots fail?
Because teams buy before they decide what success looks like. MIT’s 2025 study of business AI, reported by Fortune, found only about 5% of AI pilots delivered rapid revenue gains, and the rest stalled with little measurable impact. The same research found that buying from specialist vendors worked about two thirds of the time, while building your own worked about a third as often.
HR is no different. SHRM’s 2026 HR AI research found 56% of organizations do not formally measure whether their AI is working, and only 16% use return on investment as a measure. You cannot tell a good tool from a bad one if you never agreed what “good” means.
What should you decide before you see a demo?
| Decide | Example |
|---|---|
| The problem | “Recruiters spend six hours a week scheduling first round interviews” |
| The measure | “Hours spent scheduling per hire, and days from application to first interview” |
| The roles in the pilot | “Customer service hires in two locations” |
| Who approves what | “The tool books and reschedules; a recruiter handles every exception” |
| Deal breakers | “No automatic rejections, no training on our data, must work with our ATS” |
Share this with every vendor. It turns a sales demo into a test of whether the product solves your problem.
What should an AI recruiting tool scorecard include?
Score each vendor on the same areas, using evidence from your pilot rather than the sales deck.
| Area | What to check | Red flag |
|---|---|---|
| Fit for the task | It works on your real jobs, not the vendor’s sample data | They will only demo their own examples |
| Human control | You choose which steps run without approval | Automatic rejection is on by default |
| Explanations | Every score or pick shows the reasons | A number with no reasons |
| Bias testing | An independent audit, with a date and a summary you can read | “Our AI is unbiased” with no evidence |
| Data use | Your candidate data is not used to train their models without written consent | The contract is silent on training |
| Security | A current SOC 2 Type II report or ISO 27001 certificate | Only a security page on their website |
| Integration | Reads and writes to your ATS without copy and paste | “Export a CSV” is the integration |
| Candidate experience | You can see exactly what candidates see and receive | You cannot preview messages or screens |
| Total cost | Licenses, setup, training and the time to check its work | Price only after a long sales process |
Which frameworks can you borrow?
You do not need to invent your own checklist. Three public frameworks are written for exactly this job.
NIST AI Risk Management Framework. The US NIST framework organizes AI risk into four functions: Govern, Map, Measure and Manage. It includes checking fairness and bias, managing risk from outside vendors, and being able to switch a system off. It is voluntary and is being revised, but it is still the reference most vendors recognize.
UK Responsible AI in Recruitment guide. The UK government’s 2024 recruitment guide walks through what to check when buying and using AI for sourcing, screening, interviews and selection. It is written for non technical readers and is the most practical of the three.
AI and Inclusive Hiring Framework. Published in September 2024 for the US Department of Labor, it covers responsible vendors, impact checks, accommodations, explainability and human oversight. The department has since flagged it as possibly out of date, so use it as a checklist rather than a rulebook.
One change to know: in January 2025 the EEOC removed its AI hiring guidance from its website. As Cooley points out, the anti discrimination laws themselves did not change, and they still apply to decisions made with AI.
What security proof should you ask for?
| Proof | What it tells you |
|---|---|
| SOC 2 Type II report | An independent auditor tested the vendor’s security controls over a period of time. Ask who did the audit, because the AICPA now warns buyers to evaluate SOC reports carefully |
| ISO/IEC 27001 | The vendor runs a certified information security system |
| ISO/IEC 42001 | The first AI management standard, published in 2023. Still rare, and a good sign when present |
| Data processing agreement | Where data is stored, who can see it, how long it is kept, and whether it trains models |
Which laws change what you should ask?
This is an overview, not legal advice. Module 4, Lesson 3 maps every AI hiring law in detail.
New York City. Automated hiring tools need a yearly independent bias audit and notice to candidates. Module 4, Lesson 2 explains how to read an audit. Ask for the audit summary.
European Union. The EU AI Act lists AI used to advertise jobs, filter applications and evaluate candidates as high risk. Under the change that took effect in July 2026, those duties now apply from 2 December 2027, according to the European Commission. Some vendor pages still show the old August 2026 date, which is a quick test of how closely a vendor follows the rules.
Everywhere. You remain responsible for discriminatory outcomes, even when a vendor’s tool made the call.
How do you run a fair pilot?
Keep it short and real. Two to four weeks, one or two live roles, your own data.
Run it side by side. Have the tool and a recruiter work the same requisition, then compare results and hours.
Check the bottom of the list. Read the candidates the tool ranked lowest. If you would have interviewed some of them, write that down.
Ask candidates. A two question survey after the process tells you whether the tool helped or annoyed the people you want to hire.
Score, then negotiate. Fill in the scorecard before you talk price. Pilot results are your strongest lever.
Quick answers
Should I trust vendor case studies?
Use them to decide what to test, not what to believe. Ask for a reference customer with a similar hiring volume and call them yourself.
Is a free trial the same as a pilot?
No. A trial shows the features. A pilot tests whether the tool fixes your problem, against a measure you agreed in advance.
What if the vendor will not share a bias audit?
Treat it as a no for any tool that scores, ranks or rejects candidates. For tools that only schedule or draft messages, it matters less.
Can I use ChatGPT or Claude instead of buying a tool?
For drafting and summarizing, often yes. For screening at volume, a general chatbot has no hiring specific bias testing, so be careful what you ask it to decide.