History and Why We Test
From scurvy cures to modern tech: the evolution of controlled experiments.
A/B testing didn't start with tech companies or digital marketing. Its roots stretch back 275 years to a desperate attempt to save sailors from a deadly disease. The story of experimentation is one of curiosity, innovation, and the scientific method applied to real-world problems.
Understanding this history helps us appreciate why we test and what makes randomized controlled trials so powerful.
The Origins of Controlled Trials
1747: The Scurvy Trial
In 1747, Scottish physician Dr. James Lind was serving on a naval ship where scurvy-a disease caused by vitamin C deficiency-was killing sailors by the thousands. Lind suspected that diet might be the cause, but opinions varied wildly about the cure.
Rather than guessing, Lind designed what would become the first documented clinical trial:
- He selected 12 sailors with scurvy who had similar symptoms
- He divided them into 6 groups of 2
- Each group received a different treatment: cider, vinegar, seawater, citrus fruits (oranges and lemons), dilute sulfuric acid, or a purgative mixture
- He kept their diet and environment otherwise identical
Within 6 days, the sailors who ate citrus fruits recovered completely. The others showed little to no improvement. This was groundbreaking: Lind had demonstrated causality through controlled comparison, not just correlation or anecdote.
The Pioneers of Experimentation
Standing on the shoulders of giants: the history of scientific testing
Image: James Lind by Thomas Chalmers is licensed under Public Domain
Dr. James Lind
Conducted the famous scurvy trial on sailors, testing different treatments including citrus fruits (lemons). This was the first documented controlled trial in medical history.
Image: William Sealy Gosset is licensed under Public Domain
William Sealy Gosset
Working at Guinness Brewery, developed the t-test for small sample sizes. Published under the pseudonym "Student" because Guinness didn't want competitors knowing they used statistics.
Image: Ronald A. Fisher is licensed under Public Domain
Sir Ronald Fisher
Formalized the concept of statistical significance and p-values. Developed ANOVA and laid the foundation for modern hypothesis testing.
From Medicine to Digital
The same principles that helped cure scurvy and improve beer quality now power digital experimentation. Today's A/B tests use the exact same statistical foundations developed by these pioneers over the past 275 years.
1908: The Birth of Small Sample Statistics
Fast forward 160 years. William Sealy Gosset, a chemist at Guinness Brewery in Dublin, faced a practical problem: he needed to test the quality of barley and hops, but couldn't afford large sample sizes for every batch.
Existing statistical methods required hundreds of observations. Gosset developed a new test that worked with small samples-what we now call the Student's t-test. He published under the pseudonym "Student" because Guinness didn't want competitors knowing they used statistics to optimize their beer.
This innovation made rigorous testing accessible to businesses and researchers who couldn't collect massive datasets. It's still one of the most widely used statistical tests today.
1925: P-values and Modern Hypothesis Testing
Sir Ronald Fisher, working in agricultural research, formalized the concept of statistical significance and introduced the p-value. He also developed Analysis of Variance (ANOVA) and randomized experimental design.
Fisher's work laid the foundation for modern hypothesis testing. His 1925 book "Statistical Methods for Research Workers" became the bible for experimental scientists and established the frameworks we still use in A/B testing today.
Why Do Modern Organizations A/B Test?
From scurvy to beer to digital products, the why behind experimentation has remained consistent. Modern organizations invest in A/B testing for three fundamental reasons:
1. Data-Driven Decision-Making
Move from HiPPO (Highest Paid Person's Opinion) to evidence-based choices. Tests provide objective data that removes personal bias and politics from decisions.
"Without data, you're just another person with an opinion." - W. Edwards Deming
2. Risk Mitigation
Catch negative changes before they hurt your entire user base. Tests act as a safety net, protecting revenue and user experience.
Example: Microsoft once shipped a change that would have cost $500M/year in lost revenue. Their A/B test caught it at 1% traffic.
3. Faster Learning
Build institutional knowledge faster. Each test teaches you something about your users, building a learning advantage over competitors who rely on intuition.
Compounding effect: Companies that test consistently learn exponentially faster, making better decisions over time.
The Compounding Value of Experimentation
Here's what makes experimentation culture so powerful: the benefits compound over time.
Year 1
You catch a few big mistakes and validate some winning changes
Year 2
Your team gets better at forming testable hypotheses. You accumulate wins.
Year 3
You've built a knowledge base of what works for your users. Your product has pulled ahead of competitors.
Year 5
Testing is in your DNA. You make better decisions faster than anyone else in your space.
Companies like Amazon, Netflix, Microsoft, and Booking.com run thousands of tests per year. They're not doing this because they're obsessive - they're doing it because it's the most reliable path to sustained growth.
Key Takeaways
- ✓Experimentation has 275 years of history from Dr. James Lind's 1747 scurvy trial to modern digital A/B tests.
- ✓William Sealy Gosset developed the t-test while working at Guinness, making small-sample statistics practical for businesses.
- ✓Sir Ronald Fisher formalized p-values and modern hypothesis testing in the 1920s.
- ✓Organizations test for three reasons: data-driven decisions, risk mitigation, and faster learning.
- ✓The benefits of experimentation compound over time, creating a sustainable competitive advantage.