Pitfalls and Remedies
| Error source | Potential issue | Solution | Subsection |
|---|---|---|---|
| Non-zero tolerance ε | The inexactness introduces a bias in the computed posterior distribution. | Theoretical/practical studies of the sensitivity of the posterior distribution to the tolerance. Noisy ABC. | #Approximation of the posterior |
| Non-sufficient summary statistics | The information loss causes inflated credible intervals. | Automatic selection/semi-automatic identification of sufficient statistics. Model validation checks (e.g., Templeton 2009). | #Choice and sufficiency of summary statistics |
| Small number of models/Mis-specified models | The investigated models are not representative/lack predictive power. | Careful selection of models. Evaluation of the predictive power. | #Small number of models |
| Priors and parameter ranges | Conclusions may be sensitive to the choice of priors. Model choice may be meaningless. | Check sensitivity of Bayes factors to the choice of priors. Some theoretical results regarding choice of priors are available. Use alternative methods for model validation. | #Prior distribution and parameter ranges |
| Curse-of-dimensionality | Low parameter acceptance rates. Model errors cannot be distinguished from an insufficient exploration of the parameter space. Risk of overfitting. | Methods for model reduction if applicable. Methods to speed up the parameter exploration. Quality controls to detect overfitting. | #Curse-of-dimensionality |
| Model ranking with summary statistics | The computation of Bayes factors on summary statistics may not be related to the Bayes factors on the original data, which may therefore render the results meaningless. | Only use summary statistics that fulfill the necessary and sufficient conditions to produce a consistent Bayesian model choice. Use alternative methods for model validation. | #Bayes factor with ABC and summary statistics |
| Implementation | Low protection to common assumptions in the simulation and the inference process. | Sanity checks of results. Standardization of software. | #Indispensable quality controls |
As for all statistical methods, a number of assumptions and approximations are inherently required for the application of ABC-based methods to real modeling problems. For example, setting the tolerance parameter to zero ensures an exact result, but typically makes computations prohibitively expensive. Thus, values of larger than zero are used in practice, which introduces a bias. Likewise, sufficient statistics are typically not available and instead, other summary statistics are used, which introduces an additional bias due to the loss of information. Additional sources of bias- for example, in the context of model selection—may be more subtle.
At the same time, some of the criticisms that have been directed at the ABC methods, in particular within the field of phylogeography, are not specific to ABC and apply to all Bayesian methods or even all statistical methods (e.g., the choice of prior distribution and parameter ranges). However, because of the ability of ABC-methods to handle much more complex models, some of these general pitfalls are of particular relevance in the context of ABC analyses.
This section discusses these potential risks and reviews possible ways to address them (Table 2).
Read more about this topic: Approximate Bayesian Computation
Famous quotes containing the words pitfalls and/or remedies:
“Because relationships are a primary source of self-esteem for girls and women, daughters need to know they will not lose our love if they speak up for what they want to tell us how they feel about things. . . . Teaching girls to make specific requests, rather than being indirect and agreeable, will help them avoid the pitfalls of having to be manipulative and calculating to get what they want.”
—Jeanne Elium (20th century)
“Fear, coercion, punishment, are the masculine remedies for moral weakness, but statistics show their failure for centuries. Why not change the system and try the education of the moral and intellectual faculties, cheerful surroundings, inspiring influences? Everything in our present system tends to lower the physical vitality, the self-respect, the moral tone, and to harden instead of reforming the criminal.”
—Elizabeth Cady Stanton (1815–1902)