Experiment Exposure QA Checklist
Verify that eligible visitors see the assigned variant and that exposure and outcome events represent the experience actually delivered.
When to use this template
Use this before and at the start of a controlled website experiment when an incorrect bucket, cache or event can invalidate the comparison.
Choose another approach when: You want to inspect private user sessions unnecessarily, assume a query parameter proves random assignment, or begin analysis before checking the delivered variants.
Gather these inputs
- The exact experiment configuration and eligibility rules.
- Safe test contexts for each intended variant.
- The actual exposure and outcome event definitions.
Copy and adapt the template
Edit the text below and copy it into your own document. Edits stay in this browser field and are not submitted or saved by this page.
Worked example
Illustrative example, not a Crany customer result. Replace every example detail with facts you can verify.
A fictional experiment configuration assigns a visitor to variant B, but a shared cached page still displays A. The exposure event records B anyway. QA identifies that the report would describe an experience the visitor never saw. The owner corrects the delivery/event mismatch and repeats the safe checks before collecting interpretable results. A screenshot of the settings screen would not have caught it. This example is a diagnostic illustration; it is not evidence that a particular provider has this defect.
How to use it
- Test explicit safe cases for eligibility and each variant.
- Compare visible delivery with assignment and actual event records.
- Resolve mismatches before analyzing outcomes and recheck the corrected scope.
Check before using it
- Configuration success is distinguished from delivered exposure.
- Outcome events use the same population logic as the experiment.
- No unnecessary private session content is captured.
What to measure
Track verified delivery cases and instrumentation defects resolved. A passing QA set covers those cases only; monitor assignment and data quality during the live test as well.
Research and sources
The references discuss assignment integrity and data-quality risks. This original QA checklist is not an experiment SDK or a statistical validation by itself.
- Microsoft Research: Patterns of Trustworthy Experimentation, Pre-Experiment Stage — checked 2026-10-23. Clear hypotheses, appropriate measurement, assignment units and data-quality checks before a controlled experiment.
- Spotify Engineering: New Experimentation Platform, Part 2 — checked 2026-10-23. Predeclared success and guardrail metrics, assignment validity, planned analysis and interactions between experiments; historical platform account.
About Crany
Crany is an AI marketing co-founder in development for software founders and vibe coders, designed to strategize, execute approved marketing and improve based on results. This free resource is usable independently of the product. Join the public waitlist.