Online Randomized Controlled Experiments
Web sites can run randomized controlled experiments to create a feedback loop. Key differences between offline experimentation and online experiments include:
- Logging: user interactions can be logged reliably.
- Number of users: large sites, such as Amazon, Bing/Microsoft, and Google run experiments, each with over a million users.
- Number of concurrent experiments: large sites run tens to hundreds of overlapping, or concurrent, experiments.
- Robots, whether web crawlers from valid sources or malicious internet bots.
- Ability to ramp-up experiments from low percentages to higher percentages.
- Ability to use the pre-experiment period as an A/A test to reduce variance.
Read more about this topic: Randomized Experiment
Famous quotes containing the words controlled and/or experiments:
“It does not follow, because our difficulties are stupendous, because there are some souls timorous enough to doubt the validity and effectiveness of our ideals and our system, that we must turn to a state controlled or state directed social or economic system in order to cure our troubles.”
—Herbert Hoover (1874–1964)
“A country survives its legislation. That truth should not comfort the conservative nor depress the radical. For it means that public policy can enlarge its scope and increase its audacity, can try big experiments without trembling too much over the result. This nation could enter upon the most radical experiments and could afford to fail in them.”
—Walter Lippmann (1889–1974)