A/B Test Email Subject Lines in Gmail on a Small List

You can A/B test email subject lines on a small list, but only if you accept what small numbers can and cannot tell you. The method is simple: split your recipients at random into two groups, send the same email with two different subject lines at the same time, and compare how many people in each group opened it. The hard part is the last step. With 25 people per group, one extra open moves the open rate by four percentage points, so most single tests end in a tie. The fix is to test bigger differences, repeat tests over several sends, and keep a log, rather than trusting any one result.

This guide covers the whole loop: which subject line patterns are worth testing, how to run a fair split from Gmail (manually or with a Google Sheets mail merge), how to measure opens, and how to decide whether the difference you see is real or noise.

Why an A/B test of email subject lines on a small list is different

Large senders run subject line tests on tens of thousands of addresses. At that scale, a two-point difference in open rate is probably real. If you send a newsletter to 80 clients, or a cold outreach batch to 40 prospects, the maths changes completely.

Three things make small-list testing harder:

• Every person is a big percentage. In a group of 20, each recipient is 5 percent of the open rate. Whether one person happened to be on holiday matters more than your wording.
• Open tracking is not perfect. Opens are usually detected with a tiny image loaded when the message is displayed. Recipients who block images, and anyone on Apple Mail with Mail Privacy Protection, either do not register or register as opened whether they read it or not. On a big list this blurs the numbers evenly. On a small list it can tip a test either way.
• Your list is not uniform. A company with ten people at the same firm, or a list mixing warm clients and cold leads, means a random split can accidentally put all the likely openers on one side.

None of this means you should skip testing. It means you should run the test to learn a pattern over several weeks, not to crown a winner after one send.

Subject line patterns worth testing

Pick variants that are genuinely different from each other. Testing "Quick question" against "A quick question" wastes a send. Below are patterns that tend to produce a noticeable gap in one direction or the other, which is what a small test needs.

Specific versus generic

Generic: Update on the project Specific: Draft contract for the Henley site, ready for your review

Specific lines tell the reader what is inside and why it concerns them. This is often the single biggest lever for business email, and the one to test first.

Question versus statement

Statement: Invoice 2041 is now overdue Question: Can you confirm invoice 2041 is in your payment run?

Questions invite a reply and can feel less like a demand. They can also read as pushy if the relationship is cold.

With and without the recipient's context

Plain: Proposal for your website With context: Proposal for the Bakehouse site redesign

Including the company, project or place the recipient actually uses signals that the email is not a template. If you are using mail merge, this is a merge field, so it costs nothing to add.

Short versus descriptive

Short: Thursday? Descriptive: Can you do Thursday 2pm for the kick-off call?

Short lines stand out in a crowded inbox but give no reason to open. Descriptive lines compete on relevance. Which wins depends heavily on whether the reader already knows you.

Number versus no number

Three changes to the October schedule against Changes to the October schedule. Numbers set an expectation of a quick, scannable read.

Things not worth testing on a small list: emoji, capitalisation, and punctuation tweaks. The effect, if any, is too small to see with fewer than a few hundred recipients.

How to run the split from Gmail step by step

Gmail has no built-in A/B feature, so you set the split up yourself. The two parts are a fair split and sending both groups at the same time.

Build the split in Google Sheets

1. Put your recipients in a Google Sheet with columns for Email, First name, and any merge fields you want (company, project).
2. In a new column called Group, add a random number with =RAND() in each row, then copy the column and Edit > Paste special > Values only so the numbers stop changing.
3. Sort the sheet by that column. The top half is group A, the bottom half is group B. Type A or B into a column called Variant.
4. Add a column called Subject and fill it with subject A for the A rows and subject B for the B rows. You now have one sheet that drives one send.

Randomising with a number, rather than alphabetically or by date added, avoids accidentally grouping people from the same company together.

Send both variants at the same time

If the list is small enough to send by hand, write the email once, then send it to each person individually (never as one message with everyone in BCC, which makes per-person opens impossible to read). Use the subject from your sheet. Keep the two groups within the same half hour, because send time affects opens more than most subject line changes.

If that is too many emails to send by hand, use a mail merge. Gmail can do this for free with Apps Script (we have a separate guide on Google Sheets mail merge), or with an extension. Zestmail runs a mail merge from a Google Sheets URL directly inside Gmail and lets you schedule the send, so group A and group B leave at the same minute without you sitting at the keyboard. On the free plan the limit is 20 emails per day with a small "Sent with Zestmail" footer; for a 60-person test you would need the Pro plan or to split the send across days, which weakens the test.

Measure opens per group

You need per-recipient open data, not a total. Options:

• Native Gmail read receipts only exist on Google Workspace accounts and only work when the recipient's mail system cooperates and the person agrees to send one. Not usable for a test.
• A tracking extension adds an invisible image to each message and shows you who opened it and when. Zestmail does this inside Gmail, filters out your own opens and common bot opens (security scanners that "open" mail before the person does), and shows the results in a side panel. It does not ask for permission to read your inbox. Its AI-assisted subject-line A/B tests can also suggest variants and keep the comparison in the same place as the opens, which saves the spreadsheet work above; note that AI features send the draft to Zestmail's server to generate suggestions.

Whatever tool you use, wait a fixed window before counting. Forty-eight hours covers most business email. Count opens at the same point for both groups.

Reading the results: when is a difference real?

This is where most small-list tests go wrong. Here is a practical way to judge it without a statistics course.

1. Count the opens in each group and work out how many recipients one open represents. With 25 per group, each open is 4 points of open rate.
2. Ask: could two or three people flipping change the winner? If group A got 11 opens and group B got 9, then two people decide the result. Treat that as a tie.
3. Only call a winner when the gap is clearly bigger than a handful of people. On groups of 25, something like 14 versus 7 is worth believing. On groups of 10, almost nothing is.
4. If you want a formal check, free online "A/B test significance calculators" accept opens and recipients per group and tell you how confident you can be. Most small tests will come back inconclusive. That is the honest answer, not a failure.

A rough guide for what you can realistically detect:

These are deliberately conservative rules of thumb for a single send. The more sends you combine, the smaller the gap you can trust.

Also check timing, not just totals. If one variant's opens all happen in the first hour and the other's trickle in over two days, the first subject line is doing more work even if the 48-hour totals are close.

Keep a running log, not a scoreboard

The way to get real answers from a small list is to accumulate them. Set up a sheet with one row per test: date, list segment, variant A, variant B, recipients per group, opens per group, and a one-line note on what pattern you were testing (specific versus generic, question versus statement, and so on).

Then:

• Test the same pattern across several sends, with different wording each time. If "specific" beats "generic" in four out of five sends, you have learned something you can use, even though no single send was conclusive.
• Reuse the winning pattern as the control and test a new pattern against it next time. One variable per test.
• Note anything that could have skewed the group, such as half of group B being at one company that was closed for a public holiday.
• Re-test winners occasionally. A pattern that worked in cold outreach may not work on the same people once they know you.

After eight or ten sends you will have a short, evidence-based house style for subject lines that fits your readers, which is more useful than any list of universal rules. The tools involved are modest: a spreadsheet, a way to send both versions at once, and a way to see opens per person. Gmail plus a free Apps Script merge gets you most of the way; an open tracker such as Zestmail removes the manual counting, with a 7-day trial on Pro if 20 emails a day is not enough for your test. Either way, the discipline of random splits, same-time sends and honest reading of small numbers is what makes the test worth running.