Should I use a one-tailed or two-tailed test for A/B testing?

The choice depends on your decision framework. If you will act differently based on a positive vs negative result (which is the case in most A/B tests), a one-tailed test is appropriate and requires 20-60% less traffic. If you need to share results for organizational learning or accountability, two-sided tests or TOST equivalence testing protect against confirmation bias.

How do you determine sample size for an A/B test?

Sample size depends on four factors: significance level (alpha), desired statistical power (typically 80% or higher), the minimum effect size you want to detect, and the variability of your metric. Power and sample size have a nonlinear relationship, so increasing power from 80% to 90% requires substantially more traffic. Effect size estimation is the hardest input because overly optimistic assumptions lead to underpowered tests.

What is the multiple comparisons problem in A/B testing?

When you test multiple metrics or variants simultaneously, the probability of at least one false positive increases. With 10 comparisons at a 5% significance level, there is a 40% chance of finding a spurious significant result. Corrections like Bonferroni (dividing alpha by the number of comparisons) or Benjamini-Hochberg (controlling the false discovery rate) help address this.

Can you use t-tests for non-binomial metrics like revenue per user?

Yes. The Central Limit Theorem ensures that the sampling distribution of means is approximately normal regardless of the underlying data distribution. Even highly skewed revenue data can be tested with standard t-tests, as long as you extract user-level data to estimate variance and compute the pooled standard error of the mean.

What is non-inferiority testing in A/B testing?

Non-inferiority testing determines whether a new treatment is not meaningfully worse than the current one, rather than proving it is better. This is useful when you want to simplify a feature, reduce costs, or make technical changes without harming key metrics. The core challenge is selecting the non-inferiority margin (delta), which directly influences conclusions.

What is the difference between non-inferiority and superiority testing?

Superiority tests aim to prove that a variant is better than the control. Non-inferiority tests aim to prove that a variant is not worse than the control by more than a pre-defined margin. Use superiority tests when you want to improve metrics, and non-inferiority tests when you need to demonstrate that a change doesn't harm performance significantly.

How do I choose the non-inferiority margin?

The non-inferiority margin should be based on business considerations, not statistical ones. Ask: what is the maximum degradation we can tolerate while still justifying this change? Consider factors like cost savings, compliance benefits, or operational efficiency. The margin must be set before running the experiment, never after seeing the data.

Can a test show both non-inferiority and superiority?

Yes. If the confidence interval lies entirely above both the non-inferiority margin and zero, you can conclude both non-inferiority and superiority. This is the ideal outcome: the change is at least as good as the control (non-inferior) and demonstrably better (superior).

Do non-inferiority tests require smaller sample sizes?

Often yes. Non-inferiority tests use one-sided hypotheses, which can reduce required sample sizes when the margin is sufficiently large. However, the actual savings depend on the margin size, baseline conversion rate, and power. Use our sample size calculator to compare requirements for your specific scenario.

When should I use a non-inferiority test instead of a superiority test?

Use non-inferiority tests when: (1) the business has already decided to make a change for non-metric reasons (compliance, cost reduction, tech migration), (2) you need to quantify the maximum potential downside, (3) proving superiority is unrealistic but demonstrating 'not worse' is sufficient, or (4) rolling back isn't feasible even if the variant performs slightly worse.

What happens if my confidence interval overlaps with the non-inferiority margin?

If the confidence interval overlaps with the margin, the result is inconclusive. You cannot rule out that the true difference is worse than what you're willing to accept. Options include: running a larger experiment to reduce uncertainty, accepting the risk if the overlap is small, or rejecting the change if the risk is too high.

Beyond Superiority: Maximising Insights with Non-Inferiority Tests

Non-inferiority tests are often overlooked in digital A/B testing. Learn how to use them to your advantage.

Andrea Corvi

Last updated: 28 February 2024

Non-inferiority tests, often overlooked even by seasoned experimenters, are valuable not just in drug testing but also within the digital world.

While much online literature focuses on hypotheses aiming to demonstrate the variant's superiority over control, this is not always the case in the digital product space.

My appreciation for non-inferiority test designs started when I realized that certain tests I conducted didn't follow the expected B>A outcome for full-scale deployment.

Indeed, some product changes follow a less strict path to production: even if a change isn't clearly superior, factors such as compliance, operational efficiency, or senior leadership's input can push for its implementation.

This is precisely where non-inferiority tests shine, offering a more fitting test design than simple superiority tests.

What is a Non-Inferiority Test

A non-inferiority experiment tries to show that the change(s) included in the variant are not unacceptably worse than the control.

The "unacceptably" term here introduces a very important parameter that needs to be set in a non-inferiority test design: the non-inferiority margin.

The non-inferiority margin represents the maximum difference that we're willing to tolerate between the control and variant groups while still considering the variant to be non-inferior.

Key Concept

The goal is to demonstrate that the variant is not worse than the control by more than a clinically or practically meaningful amount (the non-inferiority margin).

Scenario: Website Redesign

Consider a common practical scenario: the business has decided to migrate its website to a new framework (e.g. Vue.JS). Engineering work has started, and a few UX changes are being introduced alongside the migration. The primary KPI is sales.

The business assumes the new website will also bring more sales, but we know, as experimenters, that is unfortunately not always the case. (Just ask Marks & Spencer.)

A non-inferiority test design helps the business set the right expectations, define a clear go-live plan, and understand the potential impact before full-scale deployment.

Example: Setting the margin

It could be agreed beforehand that sales shouldn't dip by more than 2% compared to the previous design. This 2% non-inferiority margin represents the maximum acceptable reduction in sales attributed to the new website.

It's common to question why we would tolerate any loss rather than reverting to the previous design. However, sometimes rolling back isn't feasible, and the benefits of the new design outweigh the decrease in sales. The redesigned site might offer lower maintenance costs, streamline workflows, or reduce the time required for future product updates.

Relationship Between Non-Inferiority and Confidence Intervals

Confidence intervals play a crucial role in non-inferiority testing. A confidence interval provides a range of values likely to contain the true difference between the variant and control. How this interval relates to the non-inferiority margin determines the outcome.

Non-inferior

The entire CI lies above the negative non-inferiority margin. We can be confident that the variant is not unacceptably worse than the control.

Inconclusive

The CI overlaps with the non-inferiority margin. We cannot rule out that the true difference is worse than what we are willing to accept.

Non-inferior + Superior

The CI lies above both the non-inferiority margin and zero. The variant is not only non-inferior but demonstrably better than the control.

Visual Interpretation Guide

Non-Inferiority Region

The non-inferiority region is a range of values within which the difference between the variant and control is considered acceptable. This region is defined by the non-inferiority margin.

For example, if we set a non-inferiority margin of 2%, the non-inferiority region would be all values higher than -2%. If the true difference lies within this region, we can conclude that the variant is not more than 2% worse than the control.

No Non-Inferiority

Even if the measured difference is higher than the non-inferiority margin, if the confidence interval doesn't entirely lie above the margin, we cannot conclude non-inferiority.

Non-Inferiority Demonstrated

When the confidence interval lies entirely above the non-inferiority margin (but still overlaps with zero), we can conclude non-inferiority.

Non-Inferiority and Superiority

When the confidence interval lies entirely above both the non-inferiority margin and zero, we can conclude both non-inferiority and superiority.

Important

The non-inferiority margin must be defined before the experiment begins based on business or clinical considerations, not after seeing the data. Setting the margin after seeing results invalidates the test.

Benefits of Non-Inferiority Design

Flexibility

Non-inferiority tests use a one-sided hypothesis, which can result in smaller sample sizes compared to two-sided superiority tests when the margin is sufficiently large.

Cost-effectiveness

When the margin allows for a larger effective difference to detect, non-inferiority tests can reach conclusions faster. The key is setting a meaningful margin during planning.

Easier decision-making

Even if the variant does not demonstrate superiority, it may still be acceptable if it meets the non-inferiority criteria. Valuable when achieving superiority may be challenging or unnecessary.

How to Calculate Sample Size and Significance

ABTestResult.com provides powerful calculators to help you determine the minimum sample size required for a non-inferiority test and perform the test analysis.

For power analysis you will be able to set all the necessary parameters: non-inferiority margin, significance level, power, and baseline conversion rate. The calculator will then provide you with the minimum sample size required for the test to be valid.

In order to analyse the test, you will be able to input the test data and the calculator will provide you with the confidence interval and the statistical significance for the test.

References

Share this article

Help others learn about non-inferiority testing

If you found this article valuable, share it with your experimentation team and help spread better A/B testing practices.

Related Resources

Significance Calculator

Analyze your experiment results.

Sample Size Calculator

Plan experiments with proper power analysis.

MDE Calculator

Determine your minimum detectable effect.

Guardrail Metrics

Protect experiments from hidden harm.

Practical Significance

Learn when a result actually matters.

Region-Beta Paradox

Why borderline results cause the most damage.

Frequently Asked Questions

Ready to Try Non-Inferiority Testing?

Use our calculators to plan and analyze your non-inferiority tests.