Scenario Analysis - Traditional Critique

Traditional Critique

While there is utility in weighting hypotheses and branching potential outcomes from them, reliance on scenario analysis without reporting some parameters of measurement accuracy (standard errors, confidence intervals of estimates, metadata, standardization and coding, weighting for non-response, error in reportage, sample design, case counts, etc.) is a poor second to traditional prediction. Especially in “complex” problems, factors and assumptions do not correlate in lockstep fashion. Once a specific sensitivity is undefined, it may call the entire study into question.

It is faulty logic to think, when arbitrating results, that a better hypothesis will obviate the need for empiricism. In this respect, scenario analysis tries to defer statistical laws (e.g., Chebyshev's inequality Law), because the decision rules occur outside a constrained setting. Outcomes are not permitted to “just happen”; rather, they are forced to conform to arbitrary hypotheses ex post, and therefore there is no footing on which to place expected values. In truth, there are no ex ante expected values, only hypotheses, and one is left wondering about the roles of modeling and data decision. In short, comparisons of "scenarios" with outcomes are biased by not deferring to the data; this may be convenient, but it is indefensible.

“Scenario analysis” is no substitute for complete and factual exposure of survey error in economic studies. In traditional prediction, given the data used to model the problem, with a reasoned specification and technique, an analyst can state, within a certain percentage of statistical error, the likelihood of a coefficient being within a certain numerical bound. This exactitude need not come at the expense of very disaggregated statements of hypotheses. R Software, specifically the module “WhatIf,” (in the context, see also Matchit and Zelig) has been developed for causal inference, and to evaluate counterfactuals. These programs have fairly sophisticated treatments for determining model dependence, in order to state with precision how sensitive the results are to models not based on empirical evidence.