Online Randomized Controlled Experiments
Web sites can run randomized controlled experiments to create a feedback loop. Key differences between offline experimentation and online experiments include:
- Logging: user interactions can be logged reliably.
- Number of users: large sites, such as Amazon, Bing/Microsoft, and Google run experiments, each with over a million users.
- Number of concurrent experiments: large sites run tens to hundreds of overlapping, or concurrent, experiments.
- Robots, whether web crawlers from valid sources or malicious internet bots.
- Ability to ramp-up experiments from low percentages to higher percentages.
- Ability to use the pre-experiment period as an A/A test to reduce variance.
Read more about this topic: Randomized Experiment
Famous quotes containing the words controlled and/or experiments:
“Science is a system of statements based on direct experience, and controlled by experimental verification. Verification in science is not, however, of single statements but of the entire system or a sub-system of such statements.”
—Rudolf Carnap (1891–1970)
“Science is a dynamic undertaking directed to lowering the degree of the empiricism involved in solving problems; or, if you prefer, science is a process of fabricating a web of interconnected concepts and conceptual schemes arising from experiments and observations and fruitful of further experiments and observations.”
—James Conant (1893–1978)