Selection Bias and Meta-analysis
Selective reporting has been offered by critics as an explanation for the positive results reported by parapsychologists. Selective reporting is sometimes referred to as a "file drawer" problem, which arises when only positive study results are made public, while studies with negative or null results are not made public. Selective reporting has a compounded effect on meta-analysis, which is a statistical technique that aggregates the results of many studies in order to generate sufficient statistical power to demonstrate a result that the individual studies themselves could not demonstrate at a statistically significant level. For example, a recent meta-analysis combined 380 studies on psychokinesis, including data from the PEAR lab. It concluded that, although there is a statistically significant overall effect, it is not consistent and relatively few negative studies would cancel it out. Consequently, biased publication of positive results could be the cause.
The popularity of meta-analysis in parapsychology has been criticized by numerous researchers, and is often seen as troublesome even within parapsychology itself. Critics have said that parapsychologists misuse meta-analysis to create the incorrect impression that statistically significant results have been obtained that indicate the existence of psi phenomena. Physicist Robert Park states that parapsychology's reported positive results are problematic because most such findings are invariably at the margin of statistical significance and that might be explained by a number of confounding effects; Park states that such marginal results are a typical symptom of pathological science as described by Irving Langmuir.
Researcher J. E. Kennedy has said that concerns over the use of meta-analysis in science and medicine apply as well to problems present in parapsychological meta-analysis. As a post-hoc analysis, critics emphasize the opportunity the method presents to produce biased outcomes via the selection of cases chosen for study, methods employed, and other key criteria. Critics say that analogous problems with meta-analysis have been documented in medicine, where it has been shown different investigators performing meta-analyses of the same set of studies have reached contradictory conclusions.
Read more about this topic: Psychic Powers, Criticism and Controversy
Famous quotes containing the words selection and/or bias:
“The books for young people say a great deal about the selection of Friends; it is because they really have nothing to say about Friends. They mean associates and confidants merely.”
—Henry David Thoreau (1817–1862)
“The solar system has no anxiety about its reputation, and the credit of truth and honesty is as safe; nor have I any fear that a skeptical bias can be given by leaning hard on the sides of fate, of practical power, or of trade, which the doctrine of Faith cannot down-weigh.”
—Ralph Waldo Emerson (1803–1882)