A/B Testing & Experimentation Interview Questions
Practice experimentation concepts tested in data science and product analytics interviews: experiment design, sample size and power, p-values, novelty effects, and common A/B pitfalls.
Frequently Asked Questions
Why is A/B testing its own interview topic?
Product data science and growth/analytics roles run on experimentation, so loops include a dedicated A/B testing case: how you'd design a test, choose a metric, size the sample, decide significance, and avoid pitfalls. It blends statistics with product judgment.
What are the most common A/B testing pitfalls interviewers probe?
Peeking (stopping early when significance appears, inflating false positives), running underpowered tests, the multiple-comparisons problem, novelty and primacy effects, sample-ratio mismatch, ignoring network/interference effects, and optimizing a proxy metric that hurts the true goal.
How do you choose a sample size for an experiment?
Sample size is driven by the minimum detectable effect (MDE), baseline conversion rate, desired statistical power (usually 80%), and significance level (usually 5%). Smaller effects and lower baseline rates require larger samples. Interviewers want you to reason about this tradeoff, not memorize the formula.
What metrics should an A/B test track?
A single primary (decision) metric tied to the hypothesis, supported by secondary metrics and guardrail metrics (e.g. latency, churn, revenue) that catch unintended harm. Overall Evaluation Criterion (OEC) thinking - balancing short-term wins against long-term health - is a senior signal.
When is an A/B test the wrong tool?
When you can't randomize, when effects take too long to manifest, when network effects violate the independence assumption (e.g. marketplaces, social features), or for rare events that would need an impractically large sample. Then you turn to quasi-experiments, switchback tests, or causal-inference methods.