ab-test-designer

Domain Context This skill implements a proven product management framework. The approach combines best practices from industry leaders and is designed for practical application in day-to-day PM work. Input Requirements Context about your product, feature, or problem Relevant data, research, or constraints (recommended but optional) Clear articulation of what you're trying to achieve A/B Test Designer When to Use Testing a new feature or design variation Validating a hypothesis before full rollout Optimizing conversion rates or key metrics Choosing between multiple design approaches Need to make a data-driven decision on a change What This Skill Does Helps you design rigorous A/B tests with clear hypotheses, success metrics, sample size calculations, and analysis plans. Instructions Help me design an A/B test for [feature/change]. Include: Hypothesis Current situation and metrics Proposed change Expected impact and why Test Design Primary success metric Secondary metrics Sample size needed Test duration User segments to include/exclude Variants Control (A): current experience Variant (B): new experience Any additional variants (C, D, etc.) Risks and Controls Potential negative impacts Guardrail metrics When to stop the test early Analysis Plan Statistical significance threshold How to handle edge cases Decision criteria Feature context: [Add context about the change you want to test] Best Practices Start with a clear, falsifiable hypothesis Choose one primary metric to avoid multiple comparison issues Calculate sample size upfront based on expected effect size Run tests for full weekly cycles to account for day-of-week effects Set a minimum test duration (usually 1-2 weeks) Define success criteria before running the test Monitor guardrail metrics (revenue, errors, performance) Example Input: Testing new onboarding flow vs current 3-step process Output: Hypothesis (new 1-step flow will increase co...