Plans cold email tests a recruiter's list can really support: big angle and offer swaps before wording, one variable at a time, split inside one campaign, interested replies as the measure and a kill rule written first. On small agency lists it says so and steers to reading replies instead. It reads results honestly and says too early when numbers are small.
Pipeline and ATScategory
No scriptsinstructions only, sends nothing
Any AI appupload, or copy and paste
Free, no signup
Upload the file to Claude, ChatGPT Business or Gemini Enterprise. On ChatGPT Plus, the free Gemini app or Grok, copy and paste it instead.
Nothing hidden: this is the whole file. Want it with your agency's details filled in, or formatted for ChatGPT, a Gemini Gem or Grok? Use the copy tool on the Cold Email Pack page.
---
name: ab-test-plan
description: "Plans honest testing for a recruitment agency's cold email. Big angle and offer swaps before wording, one variable at a time, interested replies as the measure, a sample size the list can really support and a kill rule, then reads the result without over-claiming. Use when a recruiter asks \"what should I A/B test\", \"test subject lines\", \"which email version won\", \"how many emails do I need for a test\", or \"is this result significant\"."
license: MIT
metadata:
author: Work Insiders
version: "1.0"
pack: cold-email-pack
---
# A/B test plan
Most cold email "tests" compare two versions on 40 sends each and call a winner from two replies. You plan tests the recruiter's list can actually support, and say plainly when a result is luck.
If `references/agency-profile.md` exists, read the Cold email lines first.
## First, the honest maths
With reply rates of a few percent, small differences need very large samples. At reply rates of 2 to 5%, spotting that one version gets about a third more replies needs roughly **3,000 to 8,000 sends per version**. At about 1,000 per version only a near-doubling shows up reliably, and under about 500 per version a "winner" is rarely more than noise. Treat small-list results as hints, not proof.
Most agency lists are much smaller (an aging-role list might be 150 to 300 companies; an MPC campaign 15 to 40 managers). Say so, and plan for that:
- **Small list (a few thousand per version is out of reach):** do not run formal tests. Run one clear angle at a time, read every reply, and swap to a very different angle when one is clearly dead.
- **Large list:** run a proper split, below.
## What to test, in order
Move down only when the step above is settled:
1. **The segment** (who): two buyer titles or two triggers, same email.
2. **The angle or offer** (why reply): blind profiles vs a market snapshot vs a job-ad second opinion. Different angles, not different wordings of one.
3. **The ask:** "want the profiles?" vs "reply 'yes' and I'll send them".
4. **The first line.**
5. **The subject line.** Test a creative subject against a plain one (`{{company}} hiring`); test it rather than assume.
6. **Follow-up timing or number of steps.**
## A fair test
- **One variable per test**, everything else identical: same segment split randomly, same mailboxes, same days.
- **Split inside one campaign** in the sequencer as variants of the same step, not as two campaigns or two weeks.
- **Measure interested replies**: yes, send details, a question about the offer, or "what are your fees?". Count meetings and job orders when there are enough. Ignore opens.
- **Write the kill rule first.** A practical one from practitioners: about 2,000 sends of a version (or two full runs of the sequence on a small list) with no interested reply means the angle is dead; one strong interested reply in the first 500 earns it more volume, not a verdict.
## Reading the result
Ask for sends, human replies (out-of-office and bounces removed), interested replies, meetings, job orders, bounces and opt outs per version.
- **Interested rate = interested replies / human replies**, and also interested replies per 1,000 sends.
- Fewer than about 10 interested replies per version: **too early**, whatever the percentages say.
- A big lead with 10+ interested replies each: **likely winner**; run it against a new challenger.
- Close: **no difference**; test something higher on the list.
- A version with more opt outs or bounces is not a winner, even with more replies.
- Read who replied, not only how many: competitors, job seekers and people pitching back are not buyers.
Offer a two-proportion test on the recruiter's numbers if they want significance; never claim it without one.
## Output
The plan (variable, the versions, segment, split, sends per version, measure, kill rule, the date to read it), or for results: the verdict, the numbers behind it, and the next test.
## Rules
- No invented benchmarks. Compare versions with each other and with the recruiter's own past campaigns.
- Write one line on why the losing version may have lost, so the lesson is kept.
The agency profile it reads (fill this in so it uses your real details)
# Agency profile
Fill this in once. Every skill in the pack reads it, so every output comes out in your agency's voice with your real terms. Delete any line you do not want the AI to use. Leave a line blank rather than guessing.
## Who we are
- Agency name:
- Recruiter name and title (for sign-offs):
- Phone:
- Mailing address (needed for US commercial email):
- Years in business:
- Specialties (roles we fill):
- Markets (cities, regions or "US remote"):
- Industries we know best:
## Terms
- Direct hire fee basis (e.g. % of first-year base):
- Guarantee period and type (free replacement / pro-rata refund / credit; conditions, e.g. fee paid within terms):
- Contract markup or bill rate approach:
- Exclusive or retained options:
- Payment terms (e.g. net 30 from start date):
- Introduction period (how long a candidate we introduce stays ours):
- Contract-to-hire conversion fee:
- Concessions we allow, and their limits:
## Proof points we can share
- Placements, time to fill, or retention figures we are happy to quote:
- Client names we may mention (only with permission):
## How we work
- ATS:
- Do we text candidates? (yes/no):
- Name the client in first candidate messages? (yes/no):
- Blind profiles: always / new clients only / never:
- Include pay expectation in submittals? (yes/no):
- Job post apply instructions:
- EEO statement:
## Cold email
- Sequencer we use (e.g. Instantly, Smartlead, EmailBison, lemlist, Apollo):
- Sending domains (lookalikes only, never our main domain):
- Mailboxes and the sender names on them:
- Countries we email (sets which email and data laws apply):
- Opt-out line we use (e.g. "Reply 'no' and I won't email again."):
- Booking link (sent only after someone replies yes):
- Where our current clients, opt outs and do-not-contact list are kept:
## Words we never use
-
Yes. It is part of the free Work Insiders Cold Email Pack, MIT licensed, so you can edit and share it.
Does A/B Test Plan work in Gemini?
Yes. Upload it in Gemini Enterprise or drop the folder into Gemini CLI's skills folder. In the regular Gemini app, paste its instructions into a Gem.
Can I trust it with my data?
It has no scripts and sends nothing anywhere. Whatever you paste goes only to your AI assistant, under that assistant's own data terms, so remove candidate details you do not need first. It can still steer how the assistant answers, so read it before you use it.
Written by Work Insiders for agency recruiters. Fill in the agency profile before you use it so it never invents fees or proof points. How we make money.
