Randomized Experiment - Online Randomized Controlled Experiments

Online Randomized Controlled Experiments

Web sites can run randomized controlled experiments to create a feedback loop. Key differences between offline experimentation and online experiments include:

• Logging: user interactions can be logged reliably.
• Number of users: large sites, such as Amazon, Bing/Microsoft, and Google run experiments, each with over a million users.
• Number of concurrent experiments: large sites run tens to hundreds of overlapping, or concurrent, experiments.
• Robots, whether web crawlers from valid sources or malicious internet bots.
• Ability to ramp-up experiments from low percentages to higher percentages.
• Ability to use the pre-experiment period as an A/A test to reduce variance.