Best A/B Testing Books

The most technical book on this list. A ground-up treatment of statistics applied specifically to online A/B testing, covering p-values, confidence intervals, statistical power, sample size calculations, and the common mistakes practitioners make when interpreting test results. Unlike academic statistics textbooks, every concept is illustrated through real A/B testing scenarios rather than dice rolls or coin flips. The book also addresses often-ignored topics like external validity (whether your test results will hold over time), running multiple tests simultaneously, and using percentage change as a KPI.

Practitioners who want to deeply understand the statistical foundations behind the tools they use every day. Essential for anyone who needs to calculate sample sizes, interpret p-values correctly, or explain test results to stakeholders without oversimplifying. Particularly valuable for analysts and data scientists working in CRO or product experimentation.

1. 1

   Many standard statistical practices in A/B testing are borrowed from clinical trials without adapting them to the unique characteristics of online business, leading to systematic errors in how tests are designed and interpreted.

2. 2

   Underpowered tests are one of the most common and costly mistakes in experimentation. The book provides clear formulas and guidance for proper sample size calculation to avoid them.

3. 3

   External validity is routinely overlooked. A test result that holds for one week may not hold for three months due to novelty effects, seasonality, or shifting user populations.

Written by veterans from Microsoft, Google, and LinkedIn who collectively ran tens of thousands of experiments. This is the definitive reference for building and scaling experimentation programs. The book covers everything from foundational concepts (what makes a good metric, how to set up a control group) to advanced platform engineering (building experimentation infrastructure, handling interference between variants, measuring long-term effects). Structured in five parts across 23 chapters, it progresses from introductory material to deeply technical platform and analysis topics.

Teams building or scaling an experimentation platform. Product managers and data scientists who need a comprehensive reference for running trustworthy experiments at scale. Also valuable for leadership trying to shift their organization toward data-driven decision-making and away from what the authors call the HiPPO (Highest Paid Person's Opinion).

1. 1

   The Overall Evaluation Criterion (OEC) is critical. Teams need a metric that is measurable in a short time period, sensitive enough to detect differences, and predictive of long-term business goals. "Profit" is almost never a good OEC because short-term theatrics can inflate it while hurting the business long-term.

2. 2

   Twyman's Law: "Any figure that looks interesting or different is usually wrong." The book stresses the importance of validating surprising results through A/A tests and sanity checks before acting on them.

3. 3

   Most ideas fail. Even at companies like Microsoft and Google, the majority of experiments show no positive effect. This is normal and actually demonstrates that the experimentation program is testing genuinely uncertain hypotheses rather than rubber-stamping obvious changes.

A Harvard Business School professor's case for making experimentation the centerpiece of organizational decision-making. Unlike the more technical books on this list, Thomke focuses on the strategic and organizational dimensions of experimentation. He draws on case studies from Booking.com, Amazon, Microsoft, Netflix, and others to show how companies that run thousands of experiments per year gain a systematic competitive advantage. The book also draws an important line between disciplined experimentation (well-designed tests that produce learnings regardless of outcome) and undisciplined experimentation (throwing ideas at the wall without proper controls).

Executives, product leaders, and anyone trying to build a culture of experimentation within their organization. Less about the statistical how-to and more about the strategic why and the organizational changes needed to make experimentation work at scale.

1. 1

   There is a critical difference between a "failure" and a "mistake." A failure is a well-designed experiment that doesn't move the KPI but still generates learning. A mistake is a badly designed experiment that produces inconclusive results and wastes resources.

2. 2

   Intuition is unreliable for innovation. Even experienced managers get it wrong most of the time, which is precisely why controlled experiments are necessary rather than relying on the judgment of senior leaders.

3. 3

   Experimentation should extend beyond product and R&D into marketing, customer service, operations, and pricing. The companies that benefit most from experimentation apply it across all functions, not just engineering.

Two Harvard professors trace the evolution of randomized controlled trials from academic research into mainstream business and government practice. The book is structured around real-world case studies: eBay discovering it could cut $50 million from its advertising budget, experiments on Airbnb revealing racial discrimination on the platform, and the UK's Behavioral Insights Team using experiments to improve tax compliance. It is shorter and more accessible than the other books on this list, serving as a broad introduction to why experiments matter across industries rather than a technical manual.

Anyone new to experimentation who wants to understand the broader landscape before diving into technical details. Also relevant for policymakers and public sector professionals interested in using experiments for social impact, not just commercial optimization.

1. 1

   Experiments frequently reveal that what organizations assume to be effective (like large advertising budgets) may be producing little to no measurable impact. eBay's experiment showed that much of its paid search spending was wasted.

2. 2

   Experimentation is not just a business tool. Governments and nonprofits use randomized trials to test interventions in healthcare, education, and public policy, often with results that overturn conventional wisdom.

3. 3

   Ethical considerations in experimentation are real and cannot be dismissed. Running experiments on users without their knowledge raises genuine consent and fairness questions that organizations need to address proactively.

The Vice Chairman of Ogilvy UK argues that rational, data-driven thinking has severe blind spots. Drawing on 30 years in advertising and behavioral science, Sutherland demonstrates that human decision-making runs on "psycho-logic" rather than pure logic. The book is filled with counterintuitive examples: making products more expensive can increase sales, adding friction to a process can improve satisfaction, and perception often matters more than objective reality. It is the most unconventional book on this list and deliberately challenges the data-first orthodoxy that dominates experimentation culture.

Experimentation practitioners who want to generate better hypotheses. If your test ideas are always incremental (button color, headline copy), this book will push you toward testing fundamentally different approaches. Also valuable for marketers, product designers, and anyone involved in shaping user experience.

1. 1

   People satisfice rather than optimize. Most real-world decisions are about avoiding the worst option, not finding the best one. This has direct implications for how you frame and test product changes.

2. 2

   Costly signaling builds trust. When a business visibly invests in something that doesn't directly maximize short-term profit (generous return policies, premium packaging), consumers unconsciously register it as a signal of commitment and reliability.

3. 3

   Small changes to context and perception can produce disproportionately large effects on behavior. The best experiment hypotheses often come from understanding psychology, not from analyzing funnel data.

The foundational text on user-centered design, originally published in 1988 and revised in 2013. Cognitive scientist Don Norman introduces the concepts of affordances, signifiers, mapping, and feedback that now form the core vocabulary of product design. The book explains why some products are intuitive while others are frustrating, and traces most usability problems back to poor design rather than user error. While not specifically about A/B testing, it provides the conceptual framework for understanding why certain design changes succeed or fail in experiments.

Anyone running experiments on user interfaces, websites, or digital products. Understanding Norman's principles helps you diagnose why a variant might be underperforming and generate hypotheses rooted in cognitive science rather than guesswork.

1. 1

   When users struggle with a product, the problem is almost always the design, not the user. Norman argues for replacing the term "human error" with "system error" because good design should account for how humans actually behave, not how designers wish they would.

2. 2

   Discoverability and understanding are the two most important characteristics of good design. Users need to figure out what actions are possible and how to perform them without relying on external instruction.

3. 3

   The Gulf of Execution (how hard it is to figure out how to use something) and the Gulf of Evaluation (how hard it is to tell what happened after you did something) are the two primary sources of usability friction. Experiments that reduce either gap tend to show positive results.