Utilities
This page hosts a collection of additional calculators designed to assist you in your experimentation process. Whether you need to calculate the standard deviation, verify the normality of your data, or check for sample ratio mismatch, you can get the results you need in just a few clicks.
Calculators
The standard deviation is a measure of how spread out the numbers in a data set are and it's need for continuous metrics calculations. This calculator helps calculating the standard deviation σ of a set of measurements: to get started, either start manually inputting the measures or you can copy/paste the values from Excel/Google Sheets directly in the input field.
The formula for the sample size in a power analysis for a single variation test is:
n = ( Z p o w e r + Z c o n f i d e n c e _ l e v e l ) 2 ⋅ 2 σ 2 / d 2
where:
- Z p o w e r is the z-score that corresponds to the desired power level.
- Z α is the z-score that corresponds to the desired confidence level.
- σ is the standard deviation of the population
- d is the desired effect size
If we assume a power of 80% and a confidence level of 95%:
n = ( 0.84 + 1.96 ) 2 ⋅ 2 σ 2 / d 2
If we then solve this formula for d (the minimum detectable effect), we get:
d = ( 0.84 + 1.96 ) 2 ⋅ σ / n
For simplicity and ease of calculation, this formula is often approximated to d = 4 ⋅ σ / n .
This is how the formula for the minimum detectable effect given a power level, confidence level, standard deviation, and sample size is calculated.
The fundamental assumption of most of the statistical tests is that the data is normally distributed. This calculator helps checking if the data is normally distributed or not at the confidence level specified below. The Jarque-Bera test is utilized to assess whether the test statistic (mean of differences in the case of paired-samples or simply the average in the independent sample case) is drawn from a normal distribution. Thousands of random resamplings of the test statistic are generated for this purpose. The histograms are provided for visual inspection.
Tip: it works with a simple copy/paste of the values from Excel/Google Sheets in the input field.
The Sample Ratio Mismatch Calculator is a tool designed to evaluate the significance of sample ratio discrepancies in statistical analyses. The fundamental assumption when running experiments is that the groups are split evenly between the control and variation(s), unless differently planned.
Sample ratio mismatch occurs in A/B testing when the distribution of participants between different variations is not balanced, potentially leading to biased results. This imbalance can skew the comparison between groups, affecting the reliability and accuracy of statistical analysis.
How to check for SRM?
- Input the number of unique users in the control and variation(s) in the designated fields.
- Click on the button to run the analysis and the calculator will check for the significance of any observed mismatch.
Note: the calculator is based on the Chi-Square test and the expected ratio = 50:50