Resources

This page is a collection of resources that you can use to learn more about A/B testing: it includes references to books and articles that improved my understanding of the subject.

Make sure to check out their work and please share interesting articles or books that you think should be included in this list.

Books

Articles

Explore vs Exploit: Finding the Balance in CRO

David Sanchez
Framework

Use Experiment Briefs to Design Better Experiments

Bhavik Patel
Framework

Facts and Fictions About Multiple Comparisons

Gail M. Sullivan
Multiple comparisons

Maximising Insights with Non-Inferiority Tests

Andrea Corvi
Non Inferiority

Experimentation: From startup to multiple agile teams

Positive John
Framework

One-tailed vs Two-tailed Tests of Significance in A/B Testing

Georgi Georgiev
Test type

Use Two-Sided Tests or Two One-Sided Tests (TOST) in A/B Testing

Ron Kohavi
Test type

Underpowered A/B Tests – Confusions, Myths, and Reality

Georgi Georgiev
Sample Size & Power

Statistical Significance for Non-Binomial Metrics

Georgi Georgiev
Non-Binomial

Interaction effects in online experimentation

Lukas Vermeer
Interaction Effects

Is non-inferiority on par with superiority?

Keith Goldfeld
Non Inferiority

Power and Sample Size Determination

Lisa Sullivan, PhD
Sample Size & Power

Sections

Whether you're here to learn, do pre-test analysis or post-test analysis, the cards below link to the most relevant page.
test analysis
Test Analysis
Use the power of statistics to analyse your a/b test. In a few clicks, you'll be able to find out about statistical significance, confidence intervals, sample ratio mismatch and much more...
sample sizing
Sample Sizing
Pre-test analysis: find out how many samples you need to run your experiment. It allows for different power and confidence levels, multiple variations (with Šidák correction) and multiple test setups.
utilities
Utilities
In this section you'll find a few useful standalone calculators: standard deviation from group of observations, normality checker for test data, sample ratio mismatch...
resources
Resources
Interested to know more about the theory behind a/b testing? Here you can find interesting reads from some of the most authoritative sources and documentation to upskill yourself on this topic.