A/B Testing Your Email Campaigns

A/B testing — also called split testing — is the practice of sending two or more versions of an email to separate segments of your audience to determine which version performs better. It removes guesswork from email marketing decisions and lets data drive continuous improvement in your campaigns.

What Can You A/B Test?

How to Set Up an A/B Test

Step 1 — Define Your Hypothesis

Before testing, articulate what you expect to happen and why. A clear hypothesis keeps your test focused:

"Personalising the subject line with the recipient's first name will increase the open rate by at least 3 percentage points because recipients are more likely to notice an email that addresses them by name."

Step 2 — Split Your Audience

Divide your send list into two equal, randomly selected groups:

• Group A (Control): Receives the current version — your baseline.
• Group B (Variant): Receives the version with the one change you are testing.

For large lists, testing with 20% of your list (10% per variant) and sending the winner to the remaining 80% is a common approach that protects the majority of sends while still gathering meaningful data.

Step 3 — Determine Sample Size

Step 4 — Run the Test via MigoSMTP API

// Send Variant A (Control)
POST /v1/email/send/bulk
{
  "from":    "newsletter@yourcompany.com",
  "subject": "Our summer sale starts today",
  "html":    "...template A...",
  "recipients": [...group_a_recipients...],
  "tags":    ["ab-test:summer-sale", "variant:A"]
}

// Send Variant B (Test)
POST /v1/email/send/bulk
{
  "from":    "newsletter@yourcompany.com",
  "subject": "{{first_name}}, our summer sale starts today 🎉",
  "html":    "...template A (same)...",
  "recipients": [...group_b_recipients...],
  "tags":    ["ab-test:summer-sale", "variant:B"]
}

Step 5 — Measure Results

Wait at least 24–48 hours after sending before analysing results. Most opens happen in the first 4 hours but a 48-hour window captures late openers and ensures fair comparison.

Use the Campaign Tags filter in Delivery Reports to compare metrics between variants:

Step 6 — Declare a Winner and Apply Learnings

• Variant B wins with a 5.6 percentage point improvement in open rate.
• Send the remaining 80% of your list Variant B immediately.
• Document the finding: "Personalised subject lines increase open rate by ~5.6% for this audience."
• Apply the winner's approach to your next campaign as the new baseline.
• Test the next variable — build a continuous improvement cycle.

A/B Testing Pitfalls to Avoid

• Declaring a winner too early — wait at least 24 hours before reading results. Early data is often misleading.
• Testing with too-small a sample — small list differences may be random noise, not meaningful signals.
• Testing multiple variables at once — you will not know what caused the difference.
• Not documenting results — without records, you repeat the same tests and forget past learnings.
• Only testing subject lines — subject lines are easiest but CTA, content, and timing tests can yield equally valuable insights.

Next Steps

• Email campaign scheduling
• Understanding campaign metrics
• Building your first email campaign