Calculate the
Test Result
It's simple: fill the fields and click on the 🚀 button.

Test Analysis

This is the page where you can analyse your a/b test. Use the intuitive cards and input fields to choose the type of test you want to run, the type of primary metric of your experiment, fill the data and click on the rocket button to calculate the results. The multiple options supported, should cover most of the use cases. Whether it is a simple one-tailed superiority test or a non-inferiority test, with a few clicks you'll be able to understand the results of your test. If you'd like to understand how it works, can also use the randomise button to generate random values for the input fields and test the calculator. In case you need help, hover on the tooltip or click on the help icons.

Inputs
Confidence Level:
70%
75%
80%
85%
90%
95%
99%
Variants
Not counting control
Success Metrics
Number of
Testing for:
Superiority
Superiority or Inferiority
Non-inferiority
Before and After
Survey
Metric Type:
Conversion Rate
Average
Parameters
Users
Conversions
Rate
Std Dev
OR
Bulk Input
OFF
CSV Input
Control
Users
Conversions
Rate
Std Dev
OR
Bulk Input
Variant
Margin
Expressed as a percentage of the baseline metric
Not Worse
Variant metric to be not worse than:
Not Worse
Variant metric to be not worse than:
Non-Inferiority
First Measurements (Pre)
Size: waiting for input
Second Measurements (Post)
Size: waiting for input
OFF
Force T-Test
Paired Test
Recap Cards
Observations
per Measurement
Mean
1st Measurement
Mean
2nd Measurement
Normality Test
Distribution
Significance
Statistical
Mean of differences
between two samples (Δ=2nd-1st)
Confidence Interval
of the Difference
Summary
Normality Check Plot
Note

While in-sample data (pre and post samples) may exhibit non-normality (can be checked in the histogram on the right), significance calculations focus on sample statistics (mean, median...). The sample statistics (the mean of the differences between the two samples) is what needs to be normally distributed (can be checked in the histogram on the left). The plot of the sample statistics has been generated running random simulations of the experiment using resampling with replacement of the same size of the original sample. The number of resamplings depends on the size of the observations. For <100k observations 5k resamplings, 1k otherwise.

If you want to learn more about the topic, there are articles linked in the resources section.

Recap Cards
Total Users
Control+Variant
Control
Variant
Split Ratio
Difference
Significance
Statistical
Summary
Normality Check