# Email A/B Test Calculator > Compare two email variants with effect-size intervals, practical lift thresholds, sample-ratio diagnostics, and sample-size planning. - **Category:** Measurement - **Updated:** July 2026 - **Canonical page:** [https://emailbump.com/tools/email-ab-test-calculator](https://emailbump.com/tools/email-ab-test-calculator) - **Interactive tool:** [Open the calculator](https://emailbump.com/tools/email-ab-test-calculator) ## What the tool does Enter randomized units and binary outcome events for a control and variant. The calculator reports: - Each arm's observed outcome rate - A Wilson score confidence interval for each rate - Absolute improvement in the direction defined as better - Relative lift against the observed control rate - A score-based confidence interval for the difference between independent proportions - A conventional two-sided pooled two-proportion z-test p-value - Outcome difference per 10,000 randomized units - Whether the interval clears a predefined minimum worthwhile effect - Sample progress against a fixed-horizon plan - A sample-ratio mismatch diagnostic - A sparse-event warning when the normal approximation is weak - An equal-allocation sample-size estimate for a future experiment Calculations run in the browser. Experiment counts are not submitted to Email Bump or an external service. ## Required inputs ### Randomized units Use the independent units assigned to A or B. Depending on the design, this may be a: - Unique recipient - Customer account or workspace - Household - Organization - Other stable experimental unit Do not substitute sends, deliveries, opens, page views, or repeated events when the same person or account can appear more than once. Repeated observations from one unit are correlated and violate the independence assumption of a simple two-proportion comparison. ### Outcome events Each randomized unit should contribute a binary result for the analysis window: event or no event. Examples include: - At least one qualified click - Purchase within seven days - Completed onboarding - Replied to the message - Unsubscribed - Submitted a spam complaint If one recipient can create several events, define the primary outcome as whether the recipient generated at least one event or use a model appropriate for counts and repeated observations. ### Direction For clicks or conversions, higher is normally better. For complaints or unsubscribes, lower is normally better. The calculator transforms the effect so a positive improvement always points in the selected desirable direction. ## Observed rates For an arm with `x` events among `n` randomized units: ```text observed rate = x / n ``` The absolute raw difference is: ```text variant rate - control rate ``` When lower is better, the calculator reverses the sign for its improvement display: ```text improvement = control rate - variant rate ``` An absolute improvement of 0.8 percentage points means eight additional desirable outcomes per 1,000 independent units, or 80 per 10,000, assuming the estimate generalizes. ## Absolute difference and relative lift Absolute and relative effects answer different questions. Suppose the control rate is 8% and the variant rate is 8.8%: ```text absolute difference = 8.8% - 8.0% = 0.8 percentage points relative lift = 0.8% / 8.0% = 10% ``` Relative lift can sound large when the baseline is small. Always report the baseline and absolute change beside it. Relative lift is undefined when the observed control rate is zero. ## Rate intervals The calculator uses a Wilson score interval for each arm rather than a basic Wald interval of: ```text p ± z × standard error ``` The simple Wald interval performs poorly for small samples and rates near zero or one and can extend below zero or above one. Wilson intervals remain within valid probability bounds and generally provide better coverage behavior. Changing the confidence level changes the interval width: - 90% is narrower - 95% is the conventional default - 99% is wider A 95% confidence interval is not a 95% posterior probability that the realized interval contains the true value. Under repeated use of the same procedure and assumptions, approximately 95% of constructed intervals would cover the corresponding fixed effect. ## Effect interval The calculator uses a score-based interval for the difference between two independent proportions, constructed from the arms' Wilson score limits. It displays the interval after transforming the sign into the selected better direction. The interval is the primary decision view: - **Entirely above zero:** evidence favors the variant in the selected direction. - **Crosses zero:** the result is inconclusive; the data remain compatible with some benefit and some harm. - **Entirely below zero:** evidence favors the control. The width communicates precision. An observed lift can look attractive while the interval remains wide enough to include negligible benefit or meaningful harm. ## P-value The displayed p-value comes from a conventional two-sided pooled two-proportion z-test under a no-difference model. It asks how incompatible the observed difference, or a more extreme difference, would be with the null model under the assumptions of the test. It does not report: - The probability that B is better - The probability that the null hypothesis is true - The probability the result happened “by chance” - The size or importance of the effect - The probability that a replication will win - Whether the experiment was randomized or instrumented correctly The calculator reports the p-value continuously and does not turn `p < 0.05` into a universal truth label. ## Inconclusive does not mean equal When an interval crosses zero, the experiment has not distinguished the observed effect from zero at the selected interval level. This does not prove equality. Read the interval: - If it still includes large benefit and large harm, the test is imprecise. - If it is tightly concentrated around zero, the variants may be practically similar. - If the product requires a formal claim of equivalence or non-inferiority, define an acceptable margin in advance and use an appropriate equivalence or non-inferiority design. Failing to reject a no-difference null is not the same as establishing equivalence. ## Minimum worthwhile effect The minimum worthwhile lift is the smallest relative change that would justify acting. Define it before examining the result. Consider: - Engineering and creative implementation cost - Revenue or customer value - Support burden - Complaint and unsubscribe guardrails - Brand trust - Reversibility - Opportunity cost - Whether the treatment complicates future campaigns The calculator converts the relative threshold to an absolute threshold at the observed control rate. If the control rate is 8% and the minimum worthwhile relative lift is 10%: ```text minimum absolute improvement = 8% × 10% = 0.8 percentage points ``` The decision states whether: - The entire effect interval exceeds the worthwhile threshold - The interval crosses the threshold - Even the optimistic end remains below the threshold This separates evidence of a difference from evidence of a difference large enough to matter. ## Sample-ratio mismatch A sample-ratio mismatch, or SRM, occurs when observed allocation is unusually far from the planned assignment ratio. For a planned 50/50 experiment, 50,050 and 49,950 units are ordinary random imbalance. A large departure such as 57/43 can indicate: - Assignment failure - Eligibility rules applied after randomization - One treatment failing to render or deliver - Tracking loss that differs by variant - Deduplication or identity-resolution differences - Bots, security scanners, or internal traffic concentrated in one arm - Variant-specific suppression - Query or join errors The calculator uses a two-sided normal approximation to the binomial allocation under the expected variant share. It flags `p < 0.01` as a diagnostic threshold. This threshold is not a guarantee: a plausible split does not prove correct instrumentation, and a flagged split requires investigation rather than automatic deletion or rebalancing. Interpret outcomes only after a strong allocation mismatch has been explained. ## Sparse outcomes The ordinary z-test relies on a large-sample normal approximation. The calculator warns when any of these cells contains fewer than five observations: - Control events - Control non-events - Variant events - Variant non-events Rare outcomes such as spam complaints often trigger this warning even with a large audience. Use an exact test, rare-event model, or qualified statistical review instead of trusting the normal approximation. The Wilson arm intervals remain useful descriptions of uncertainty, but the displayed ordinary p-value should not drive a decision. ## Sample-size planning The planner estimates equal-allocation units per arm for a future two-sided comparison of independent proportions. It uses: - The current observed control rate as the baseline - The minimum worthwhile relative lift as the target effect - The selected confidence level to define alpha - 80% or 90% target power - A standard normal-approximation formula Power is the long-run probability that the planned test rejects the no-difference null when the specified target effect is the true effect and the assumptions hold. The estimate is not a guarantee. Increase the plan for: - Attrition and missing outcomes - Clustered assignment - Repeated measurements - Multiple variants or primary metrics - Sequential monitoring - Conservative operational headroom - Baseline uncertainty - A full business cycle and delayed conversion window Planning from the effect observed in the same small experiment can be unstable. Use historical baseline data and a product-defined minimum effect when available. ## Fixed-horizon stopping A fixed-horizon experiment specifies before launch: - Primary metric - Randomization unit - Variants and allocation - Inclusion and exclusion rules - Minimum worthwhile effect - Sample-size target - Minimum duration - Outcome attribution window - Analysis method - Guardrail metrics Read the primary result at the planned end. Repeatedly looking at an ordinary p-value and stopping as soon as it crosses 0.05 changes the false-positive behavior of the procedure. Early stopping can be valid when the experiment uses a preplanned group-sequential, alpha-spending, always-valid, or other sequential method with boundaries appropriate to repeated looks. A standard fixed-horizon calculator does not create those boundaries. ## Duration still matters Reaching a numerical sample target does not guarantee representative exposure. Email behavior varies by: - Weekday and weekend - Time zone - Send-time optimization - Pay cycles - Billing cycles - Delayed purchase windows - Product seasonality - Frequency and prior campaign exposure Run long enough to cover the prespecified business cycle and outcome window. Do not end a Monday-morning send test after the fastest responders convert if the product decision concerns the full audience. ## Multiple comparisons False-positive risk increases when the team searches across: - Several subject lines - Many variants - Multiple primary metrics - Numerous segments - Different outcome windows - Repeated interim looks - Post-hoc exclusions Predefine one primary comparison when possible. Otherwise use a design and analysis that account for multiplicity. Treat unplanned segment findings as exploratory hypotheses to verify in a new experiment. ## Email-specific measurement concerns ### Opens Open tracking depends on a remote image request. Mailbox privacy systems can preload images, proxy requests, block images, or obscure the actual reader. Opens can therefore be generated without human attention or absent despite a real read. Use open rate as a client-dependent signal, not a universal measure of human attention. Pair it with downstream outcomes such as clicks, qualified replies, conversion, retention, complaints, or unsubscribes. ### Clicks Security scanners can visit links before a person does. Deduplicate events at the randomized unit and use bot filtering that is defined before reading the variant result. ### Conversions Use the same attribution window and event eligibility for both arms. Allow delayed conversions to mature before analysis. ### Complaints and unsubscribes These outcomes are often rare but important. A test powered for clicks may be incapable of detecting a meaningful complaint increase. Treat them as guardrails with appropriate rare-event uncertainty rather than assuming “no observed difference” means safe. ## Assumptions The calculator assumes: - Assignment was randomized - The two arms are independent - Each unit contributes one binary primary outcome - Assignment and outcome logging are accurate - Eligibility and exclusions were defined without using future outcomes - The selected analysis was not chosen after seeing results - The ordinary p-value is read at a fixed horizon - There are no unaccounted multiple comparisons - The large-sample approximation is suitable unless warned otherwise If assignment is clustered by household, account, organization, geography, or campaign batch, use cluster-aware methods. ## Recommended workflow 1. Write the hypothesis and customer mechanism. 2. Choose one primary outcome and direction. 3. Define the independent randomization unit. 4. Define the minimum worthwhile effect. 5. Plan sample size, duration, allocation, exclusions, and outcome window. 6. Validate assignment and event instrumentation before launch. 7. Monitor operational failures and guardrails without optional stopping on the primary fixed-horizon p-value. 8. At the planned end, check sample allocation first. 9. Read the effect interval, baseline, absolute effect, and practical threshold together. 10. Review complaints, unsubscribes, delivery, and downstream quality. 11. Document all variants, exclusions, dates, and deviations from plan. 12. Replicate surprising, expensive, or strategically important results. ## Frequently asked questions ### Is p=0.03 a 97% probability that B is better? No. The p-value is computed under a no-difference model and is not a posterior win probability. ### Does p=0.20 prove there is no difference? No. Read the effect interval. It may include a broad range of benefit and harm, or it may be tightly centered around zero. Equivalence requires a predefined margin and an equivalence procedure. ### Should I ship whenever the interval excludes zero? Not automatically. Compare the complete interval with the minimum worthwhile effect and guardrails. A precise but tiny improvement may not justify complexity or risk. ### Can I use sends as the sample? Only if each send is independently randomized and contributes an independent outcome. Repeated email to the same recipient or account normally violates that assumption. ### Why check allocation before conversions? A strong sample-ratio mismatch can signal that the arms were not observed under a comparable assignment and measurement process. An outcome difference can be an artifact of that failure. ### Can the planner guarantee enough power? No. It is an approximation based on the entered baseline, target effect, independence, and model. Real baseline drift, clustering, attrition, multiplicity, and implementation problems change performance. ## Method summary - Arm intervals: Wilson score intervals - Difference interval: score-based independent-proportion interval derived from Wilson arm limits - P-value: pooled two-sided two-proportion z-test - SRM diagnostic: two-sided normal approximation to expected binomial allocation - Sample planning: equal-allocation normal approximation for two independent proportions - Default confidence: 95% - Default planning power: 80% ## Primary sources - [NIST: Confidence intervals for proportions](https://itl.nist.gov/div898/handbook/prc/section2/prc241.htm) - [NIST: Sample sizes required for testing proportions](https://itl.nist.gov/div898/handbook/prc/section2/prc242.htm) - [American Statistical Association: Guidance beyond statistical-significance labels](https://www.amstat.org/news-listing/2021/10/08/editorial-calls-time-on-statistically-significant-in-research) ## Related guidance - [A/B testing email service providers](https://emailbump.com/blog/ab-test-email-service-providers.md) - [Click-through rate definition](https://emailbump.com/glossary/click-through-rate.md) - [Open rate definition](https://emailbump.com/glossary/open-rate.md) - [Spam complaint definition](https://emailbump.com/glossary/spam-complaint.md) - [Campaign analytics documentation](https://emailbump.com/docs/analytics.md)