Multivariate Testing - Design of Experiments

Design of Experiments

Statistical testing relies on design of experiments. Several methods in use for multivariate testing include:

1. Discrete choice and what has mutated to become choice modeling is the complex technique that won Daniel McFadden the Nobel Prize in Economics in 2000. Choice modeling models how people make tradeoffs in the context of a purchase decision. By systematically varying the attributes or content elements, one can quantify their impact on outcome, such as a purchase decision. What is most important are the interaction effects uncovered, which neither the Taguchi methods nor Optimal design solve for.
2. Optimal design involves iterations and waves of testings. Optimal design allows marketers the ability not only to test the maximum number of creative permutations in the shortest period of time but also to take into account relationships, interactions, and constraints across content elements on a website. This allows one to find the optimal solution unencumbered by limitations.
3. Taguchi methods: with multiple variations of content in multiple locations on a website, a large number of combinations need to be statistically tested and medium/low traffic websites can take some time to get a large enough sample of visitors to decide which content gives the best performance. For example, if 3 different images are to be tested in 3 locations, there are 27 combinations to test. Taguchi methods (namely Taguchi orthogonal arrays) can be used in the design of experiments in order to reduce the variations but still give statistically valid results on individual content elements. Taguchi uses fractional factorial designs.