Schools of Thought: Bayesian vs Frequentist
The two major statistical philosophies and when to use each approach.
Two Competing Philosophies
There's a quiet war in statistics, and it affects every A/B test you'll ever run. Two schools of thought-Frequentist and Bayesian-offer fundamentally different ways to interpret probability and make decisions from data.
Don't worry: you don't need to pick a side or dive into complex math. But understanding the difference will help you interpret test results and choose the right tools.
The Fundamental Question
The Core Debate
Frequentist: "Probability is the long-run frequency of events. If I flipped a coin infinite times, 50% would be heads."
Bayesian: "Probability is my degree of belief. Based on what I know, I'm 50% confident the next flip is heads."
This philosophical difference leads to completely different tools and interpretations. In practice, most companies use Frequentist methods (p-values, confidence intervals), but Bayesian approaches are growing in popularity, especially at tech companies with sophisticated experimentation platforms.
The Two Schools
Frequentist
Core Belief
Probability = long-run frequency of events. Truth is fixed, data is random.
Key Question
"If there's no real effect, how likely is this data?"
Tools
- • P-values
- • Confidence intervals
- • Hypothesis testing
- • Fixed sample sizes
Pros
- ✓ Industry standard
- ✓ Well-understood
- ✓ No prior needed
Cons
- × Can't say "95% chance variant wins"
- × Must fix sample size
- × P-values are confusing
Bayesian
Core Belief
Probability = degree of belief. Data is fixed, truth is uncertain.
Key Question
"Given this data, what's the probability variant B is better?"
Tools
- • Prior beliefs
- • Posterior distributions
- • Credible intervals
- • Sequential testing
Pros
- ✓ Intuitive: "X% chance to win"
- ✓ Can peek anytime
- ✓ Incorporates prior knowledge
Cons
- × Requires priors (subjective)
- × More computationally intensive
- × Less standardized
Which Should You Use?
Choose Frequentist If:
- • Your company uses it (most do)
- • You need regulatory approval
- • You can plan sample sizes upfront
- • You want industry-standard tooling
Choose Bayesian If:
- • You need sequential testing (peek anytime)
- • You have strong prior knowledge
- • Stakeholders want "% chance to win"
- • You have custom infrastructure
What This Means in Practice
Frequentist Interpretation
What you see: "p-value = 0.03, reject null hypothesis"
What it means: "If there's no real difference between variants, we'd see data this extreme only 3% of the time. That's unlikely, so we reject the idea of no difference."
What you CAN'T say: "There's a 97% chance variant B is better." (This is a Bayesian statement!)
Bayesian Interpretation
What you see: "95% probability variant B beats control"
What it means: "Given the data and my prior beliefs, I'm 95% confident variant B is better."
What you CAN say: "There's a 95% chance variant B is better." (Much more intuitive!)
The Practical Reality
Most companies use Frequentist methods because:
- •They're the industry standard (regulatory agencies, academia, most tooling)
- •No prior knowledge required (just data)
- •Well-understood with decades of research
Bayesian methods are gaining ground because:
- •The outputs are more intuitive ("95% chance to win" vs p-values)
- •You can peek at results anytime without penalty (sequential testing)
- •Modern computing makes them feasible
Our recommendation: Unless you have specific reasons to use Bayesian methods (like sequential testing needs), start with Frequentist approaches. They're simpler, more standardized, and what most teams expect. Once you master those, you can explore Bayesian methods if needed.
Key Takeaways
- ✓Frequentist: Probability = long-run frequency. Uses p-values and confidence intervals.
- ✓Bayesian: Probability = degree of belief. Outputs "% chance to win."
- ✓Most companies use Frequentist methods (industry standard, regulatory acceptance).
- ✓Bayesian methods are growing at tech companies (intuitive, sequential testing).
- ✓Start with Frequentist unless you have specific reasons to choose Bayesian.