Online Randomized Controlled Experiments
Web sites can run randomized controlled experiments to create a feedback loop. Key differences between offline experimentation and online experiments include:
- Logging: user interactions can be logged reliably.
- Number of users: large sites, such as Amazon, Bing/Microsoft, and Google run experiments, each with over a million users.
- Number of concurrent experiments: large sites run tens to hundreds of overlapping, or concurrent, experiments.
- Robots, whether web crawlers from valid sources or malicious internet bots.
- Ability to ramp-up experiments from low percentages to higher percentages.
- Ability to use the pre-experiment period as an A/A test to reduce variance.
Read more about this topic: Randomized Experiment
Famous quotes containing the words controlled and/or experiments:
“Babies control and bring up their families as much as they are controlled by them; in fact ... the family brings up baby by being brought up by him.”
—Erik H. Erikson (1904–1994)
“A country survives its legislation. That truth should not comfort the conservative nor depress the radical. For it means that public policy can enlarge its scope and increase its audacity, can try big experiments without trembling too much over the result. This nation could enter upon the most radical experiments and could afford to fail in them.”
—Walter Lippmann (1889–1974)