Remedies
Looking for patterns in data is legitimate. Applying a statistical test of significance (hypothesis testing) to the same data the pattern was learned from is wrong. One way to construct hypotheses while avoiding data dredging is to conduct randomized out-of-sample tests. The researcher collects a data set, then randomly partitions it into two subsets, A and B. Only one subset—say, subset A—is examined for creating hypotheses. Once a hypothesis is formulated, it must be tested on subset B, which was not used to construct the hypothesis. Only where B also supports such a hypothesis is it reasonable to believe the hypothesis might be valid.
Another remedy for data dredging is to record the number of all significance tests conducted during the experiment and simply multiply the final significance level by this number (the Bonferroni correction); however, this is a very conservative metric. Methods particularly useful in analysis of variance, and in constructing simultaneous confidence bands for regressions involving basis functions are the Scheffé's method and, if the researcher has in mind only pairwise comparisons, the Tukey method. The use of a false discovery rate is a more sophisticated approach that has become a popular method for control of multiple hypothesis tests.
When neither approach is practical, one can make a clear distinction between data analyses that are confirmatory and analyses that are exploratory. Statistical inference is appropriate only for the former.
Ultimately, the statistical significance of a test and the statistical confidence of a finding are joint properties of data and the method used to examine the data. Thus, if someone says that a certain event has probability of 20% ± 2% 19 times out of 20, this means that if the probability of the event is estimated by the same method used to obtain the 20% estimate, the result is between 18% and 22% with probability 0.95. No claim of statistical significance can be made by only looking, without due regard to the method used to assess the data.
Read more about this topic: Data Dredging
Famous quotes containing the word remedies:
“Our remedies oft in ourselves do lie,
Which we ascribe to heaven.”
—William Shakespeare (1564–1616)
“Fear, coercion, punishment, are the masculine remedies for moral weakness, but statistics show their failure for centuries. Why not change the system and try the education of the moral and intellectual faculties, cheerful surroundings, inspiring influences? Everything in our present system tends to lower the physical vitality, the self-respect, the moral tone, and to harden instead of reforming the criminal.”
—Elizabeth Cady Stanton (1815–1902)
“The said doctor can easily practise upon a page, and, if he does well, he can use his remedies on my son.”
—Catherine De’ Medici (1519–1589)