The Jargon: Traffic, Split, and Test Types
Decode common A/B testing terminology: traffic %, split ratios, A/A tests, A/B tests, and MVT.
Speaking the Language
Like any field, A/B testing has its own vocabulary. Understanding this terminology isn't just about sounding smart in meetings-it's about precise communication that prevents costly mistakes.
Walk into any experimentation meeting and you'll hear terms like "traffic allocation," "50/50 split," and "multivariate test" thrown around. For newcomers, this jargon can be intimidating—but it doesn't have to be.
Understanding this terminology isn't about sounding smart in meetings. It's about precise communication that prevents costly mistakes. When someone says "let's run 10% traffic," does that mean 10% of users see the test, or 10% see the variant? The answer matters for your sample size calculations and timeline.
In this section, we'll decode the essential vocabulary you need to confidently discuss, plan, and execute A/B tests with your team.
Traffic % vs. Split: Two Different Things
The most common source of confusion for beginners is mixing up Traffic % and Split. They sound similar but control completely different things:
Understanding Traffic & Split
Two separate but related concepts in experiment configuration
Traffic %
Portion of users included
Example: 20% traffic means 20% of your total users are included in the test. The other 80% see the current experience.
Split
Distribution between variants
Example: 50/50 split means within the 20% in the test, half see Control (A) and half see Variant (B).
Putting It Together
Total users: 100,000 daily active users
Traffic %: 20% → 20,000 users enter the test
Split: 50/50 → 10,000 see Control (A), 10,000 see Variant (B)
Excluded: 80,000 users continue seeing the current experience
Why not 100% traffic? Starting with lower traffic (10-20%) reduces risk. If the variant has a bug or hurts metrics, only a small portion of users are affected. Once validated, you can ramp to 100%.
Why Traffic % Matters
Starting with lower traffic serves multiple purposes:
- Risk mitigation: Bugs or negative impacts affect fewer users
- Faster iteration: Kill bad tests quickly without widespread damage
- Resource efficiency: Sample size calculations might not require 100% traffic
- Gradual rollout: Validate at 10% → 25% → 50% → 100%
Types of Tests: A/A, A/B, A/B/n, MVT
A/A Test (Calibration Test)
Both groups see the exact same experience. Used to validate your testing system and establish baseline false positive rates.
Why run A/A tests? To ensure your system doesn't detect "differences" where none exist. If an A/A test shows significant results, your testing system has a problem.
A/B Test (Simple Test)
The classic: Control (A) vs. one Variant (B). This is what most people mean when they say "A/B test."
Example: Testing a new checkout flow (B) against the current flow (A) to see which converts better.
A/B/n Test (Multiple Variants)
One control (A) plus multiple variants (B, C, D...). Test several different approaches simultaneously.
Example: Testing 3 different button colors (B: blue, C: green, D: red) against the current orange button (A).
Caution: More variants = more traffic needed to reach statistical significance.
MVT (Multivariate Testing)
Tests combinations of multiple variables simultaneously. More complex than A/B/n because it tests interactions between changes.
Example: Testing 2 headlines × 3 button colors × 2 layouts = 12 combinations
Warning: MVT requires massive traffic. For 12 combinations at 80% power, you need 12× the sample size of a simple A/B test.
Which Should You Use?
Start with simple A/B tests. They're easier to analyze, require less traffic, and build good testing habits. A/B/n tests are fine when you have several distinct approaches. MVT is advanced-only use it when you have enormous traffic and need to understand interaction effects between variables.
Key Takeaways
- ✓Traffic % controls what portion of total users are included in the test.
- ✓Split controls how traffic is distributed between control and variants.
- ✓A/A tests validate your testing system by ensuring you don't detect false differences.
- ✓A/B tests are the workhorse of experimentation-simple, reliable, and easy to analyze.
- ✓MVT requires massive traffic and should only be used when testing interaction effects matters.