Experiment Change Interference Log
Record concurrent campaign, product and measurement changes that could alter an experiment’s exposure or interpretation.
When to use this template
Use this when several people may edit the same page, audience, campaign or analytics configuration while a test is running.
Choose another approach when: You want to ban emergency fixes, claim a log eliminates confounding, or attribute every result movement to whichever change happened most recently.
Gather these inputs
- The experiment scope and planned variants.
- Known concurrent releases, campaigns and measurement owners.
- A way to record actual change times and affected populations.
Copy and adapt the template
Edit the text below and copy it into your own document. Edits stay in this browser field and are not submitted or saved by this page.
Worked example
Illustrative example, not a Crany customer result. Replace every example detail with facts you can verify.
A fictional headline test runs while a teammate launches a campaign to a very different audience. At the same time, a form fix changes which successful requests are recorded. The log timestamps both changes and flags the instrumentation change as potentially invalidating comparability. The necessary form repair is not postponed to preserve an experiment. The team reviews whether to restart or limit interpretation rather than claiming that a traffic surge proves the headline worked. Recording an interaction does not automatically correct it statistically.
How to use it
- List owners and changes that could overlap the experiment’s population or measurement.
- Record actual timing and plausible interaction without asserting causation.
- Make a reviewed action and annotate or limit the analysis as appropriate.
Check before using it
- Necessary user-protection fixes are not blocked by the experiment.
- Planned changes and completed changes are distinguished.
- A log entry is not treated as a mathematical adjustment.
What to measure
Review unresolved interactions before a ship/no-ship decision. The log improves coordination and transparency, but the experiment may still require redesign or a fresh run.
Research and sources
The sources discuss interacting experiments and trustworthiness. This original coordination log extends the observation to marketing releases without claiming to reproduce a large-company experimentation platform.
- Spotify Engineering: New Experimentation Platform, Part 2 — checked 2026-10-23. Predeclared success and guardrail metrics, assignment validity, planned analysis and interactions between experiments; historical platform account.
- Microsoft Research: Patterns of Trustworthy Experimentation, Pre-Experiment Stage — checked 2026-10-23. Clear hypotheses, appropriate measurement, assignment units and data-quality checks before a controlled experiment.
About Crany
Crany is an AI marketing co-founder in development for software founders and vibe coders, designed to strategize, execute approved marketing and improve based on results. This free resource is usable independently of the product. Join the public waitlist.