Your A/B Test Is Ambiguous - Ship or Not?

What interviewers evaluate

• Do you ask whether the test was powered to detect the effect size at all?
• Do you check validity first (sample-ratio mismatch, peeking, novelty effects)?
• Do you treat post-hoc subgroups as hypotheses, not results?
• Do you frame shipping as a decision with asymmetric costs, not a stats ritual?
• Do you protect the integrity of the experimentation culture (no re-running until significant)?

A framework to structure your answer

1. Separate the questions - 'what does the data say?' and 'what should we do?' are different questions.

Premium unlocks all 6 framework steps, a strong worked answer, 3 common variants, 5 pitfalls to avoid and 3 likely follow-ups.