How to test outreach

Change one thing at a time

Test one ICP, message angle or ask against another and keep everything else the same. If you change two things, you won't know which one made the difference.

Pick one success metric and watch for harm

Judge each test by one result that leads to revenue: positive replies, then meetings. Watch unsubscribes, spam complaints and bounces too, so a winner doesn't hurt your domain. Opens are unreliable and any reply can be a no.

Sources: Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments, 2020; Apple Mail Privacy Protection, 2021

Test big differences

Telling a 3% positive reply rate from 6% takes about 750 companies per version, counting each company once, by the standard two-proportion calculation at 5% significance and 80% power. Telling 3% from 4% takes about 5,300. So test different ICPs, angles and offers, not small wording changes.

Read early results by what replies say

A warmed mailbox sends 30 to 50 emails a weekday, depending on its history, so first emails alone to the 1,500 companies a two-version test needs take six to ten weeks from one mailbox, and follow-ups share that limit. Until then, read the replies. If 100 companies finish the sequence with no positive reply, you can be about 95% sure the true rate is below 3%, so change the ICP or angle. That is a reason to stop a miss, not to declare a winner. Pause and check the list if bounces or opt-outs rise.

Source: Hanley and Lippman-Hand, JAMA, interpreting zero numerators, 1983

Set the size and length before you start

Decide how many prospects each version gets and stick to it, except to stop a clear miss. Stopping as soon as one version looks ahead can make a false winner several times as likely, about five times in one analysis.

Sources: Johari, Koomen, Pekelis and Walsh, KDD, 2017; Johari et al., Always valid inference, 2015; Evan Miller, How not to run an A/B test, 2010

Split by company, not by person

People at one company talk to each other, so give everyone at a company the same version and count results by company.

Expect most ideas to lose

Only about a third of ideas tested at Microsoft improved the result they aimed at, and 10 to 20% at Google and Bing. A loss still tells you something, so write down each test, its result and what you changed.

Source: Kohavi and Thomke, Harvard Business Review, 2017

Treat benchmarks as starting points

Most outreach benchmarks, including the vendor studies cited here, describe other teams' campaigns rather than controlled tests. Use them to choose what to test, then trust your own results.