A/B and multivariate testing with rigorous statistics

Most marketing tools call any A/B test with a higher number on one side a 'winner'. We do the math properly. Tests run until they hit either a Bayesian confidence threshold you set or a minimum sample size — whichever comes first — and the platform tells you when a result is not statistically meaningful instead of pretending it is.

Lifecycle & Automation
·
Available in the United States · English documentation

Capabilities you don't have to build, monitor, or babysit

Each capability ships on the standard plan. Nothing in this list is an upsell or a separate product line.

01

Bayesian and frequentist outputs side by side

Some teams want a p-value, others want the probability that variant B beats variant A. We compute both, show the credible interval, and warn explicitly when sample sizes are too small for either method to mean anything.

02

Multivariate testing without an explosion of cells

Test subject, hero image, and CTA simultaneously with a fractional-factorial design that uses far fewer recipients than a naive full grid. Results show main effects and interactions separately.

03

Holdout groups and incrementality measurement

Hold back 5% of recipients and measure what would have happened without any campaign. Indispensable for showing leadership that the cost of the sequence is justified by incremental revenue, not background conversion.

04

Per-channel and cross-channel tests

Run the same test on email and SMS to see which channel each variant performs best on — sometimes the 'losing' subject in email is the winning SMS opener.

Where US customers actually use this

  • Subject-line tests for onboarding and dunning emails
  • Send-time optimization per audience and time zone
  • Hero-image and CTA tests on transactional templates
  • Channel-fallback tests (email-only vs email-then-SMS)
  • Frequency-cap tests for re-engagement audiences

Winning variants are promoted automatically into the parent flow or drip; losing variants archive with their full result attached so a year from now you don't re-run a test that already failed.

More in Lifecycle & Automation

The rest of this product family

Explore the rest of the platform

Other 4notify product areas

Messaging APIs

Reach customers on the channel they actually open — email, US-native SMS, WhatsApp Business, and any system you can point a webhook at.

Deliverability & Observability

Know what happened to every message — and why — without piecing it together from carrier exports and inbox screenshots.