A/B Test Sample Planning Worksheet
Gather the baseline, meaningful effect, assignment unit and planned analysis inputs needed to assess whether a controlled marketing test is feasible.
When to use this template
Use this before committing traffic to an A/B test, especially when a small software site may not have enough eligible visitors to answer the question soon.
Choose another approach when: You want this worksheet to calculate significance, assume every test needs the same sample, or choose a large effect solely to make the test finish faster.
Gather these inputs
- A defined primary metric and its observed baseline or explicit absence.
- The eligible assignment population and realistic arrival rate.
- An appropriate analysis method and someone able to review its assumptions.
Copy and adapt the template
Edit the text below and copy it into your own document. Edits stay in this browser field and are not submitted or saved by this page.
Worked example
Illustrative example, not a Crany customer result. Replace every example detail with facts you can verify.
A fictional waitlist site has little eligible traffic and no reliable baseline for confirmed requests. The founder cannot justify a precise test duration or claim that a few dozen visits will settle a small copy difference. The worksheet identifies missing instrumentation and suggests a comprehension study for the immediate clarity question. If a controlled comparison is later feasible, a reviewed calculation uses its actual metric and assignment design. The sheet deliberately supplies no invented sample-size output.
How to use it
- Define the decision-relevant effect and assignment population before calculation.
- Use real baseline and traffic inputs with an appropriate reviewed analysis method.
- Assess duration and feasibility honestly, selecting another method when needed.
Check before using it
- No arbitrary visitor count is presented as universally sufficient.
- The practical effect is not inflated just to reduce the sample.
- A qualitative observation is not mislabeled as a powered A/B test.
What to measure
Compare planned and actual eligible sample and document any feasibility change. This is an input worksheet, not a statistical calculator or a substitute for analysis expertise.
Research and sources
The references support planning affected samples, comparison design and realistic traffic needs. The original worksheet does not transfer any health-study outcome to software marketing.
- Microsoft Research: Patterns of Trustworthy Experimentation, Pre-Experiment Stage — checked 2026-10-23. Clear hypotheses, appropriate measurement, assignment units and data-quality checks before a controlled experiment.
- GOV.UK: A/B testing, comparative studies — checked 2026-10-23. Control/variation comparisons, random allocation and sample/planning needs; the article’s application context is digital health, not software-marketing performance claims.
About Crany
Crany is an AI marketing co-founder in development for software founders and vibe coders, designed to strategize, execute approved marketing and improve based on results. This free resource is usable independently of the product. Join the public waitlist.