Online Randomized Controlled Experiments
Web sites can run randomized controlled experiments to create a feedback loop. Key differences between offline experimentation and online experiments include:
- Logging: user interactions can be logged reliably.
- Number of users: large sites, such as Amazon, Bing/Microsoft, and Google run experiments, each with over a million users.
- Number of concurrent experiments: large sites run tens to hundreds of overlapping, or concurrent, experiments.
- Robots, whether web crawlers from valid sources or malicious internet bots.
- Ability to ramp-up experiments from low percentages to higher percentages.
- Ability to use the pre-experiment period as an A/A test to reduce variance.
Read more about this topic: Randomized Experiment
Famous quotes containing the words controlled and/or experiments:
“You enter a state of controlled passivity, you relax your grip and accept that even if your declared intention is to justify the ways of God to man, you might end up interesting your readers rather more in Satan.”
—Ian McEwan (b. 1938)
“A country survives its legislation. That truth should not comfort the conservative nor depress the radical. For it means that public policy can enlarge its scope and increase its audacity, can try big experiments without trembling too much over the result. This nation could enter upon the most radical experiments and could afford to fail in them.”
—Walter Lippmann (1889–1974)