Project Follow Through - Analytical Methods

Analytical Methods

Due to the number of intervention sites and range of instruments the analysis was complex and extensive. According to Watkins (1997, p. 32), there were over 2,000 comparisons between Follow Through and Non-Follow Through groups alone. In 1968, Stanford Research Institute (SRI) was awarded the contract for the Follow Through evaluation. However, due to a variety of factors—including, perhaps, SRIs underestimation of the complexity involved in such a comprehensive analysis—Abt Associates, Inc. later inherited the evaluative duties in the summer of 1972. The summary of results, entitled Education as Experimentation: A Planned Variation Model (Stebbins, St. Pierre, Proper, Anderson, & Cerva) was published in 1977.

The empirical goal of the Follow Through evaluation was to determine which models were effective in raising student achievement in the three domains as evidenced by positive effects using the selected instruments. Within models, the evaluators compared performance on the various instruments between Follow Through (FT) and non-Follow Through (NFT) comparison groups at each site. Within groups, the evaluators averaged students’ scores on each measure (or outcome variable) in order to yield a “group” score. Thus, the group scores of FT students were compared to the group scores of NFT students. These scores were then adjusted using a statistical technique called analysis of covariance (ANCOVA; explained below). The difference between the FT and NFT students was then used to measure the effects of a given model (Watkins, 1997, pp. 32–33). Sites where models met the criterion for “educational effectiveness” were assigned a value of 1; negative effects were assigned -1; and null effects—“insignificant educationally, statistically, or both” (Wisler, et al., 1978, p. 176) —were assigned a zero.

An important—and later controversial—statistical technique was employed by the evaluators in order to improve the integrity of the results. Because there were differences between treatment and comparison groups (e.g. the average score on an outcome measure for a NFT group might have been higher than the corresponding average score for a FT group), the evaluators employed a method known as analysis of covariance (ANCOVA) in order to adjust for these and other differences. According to Elmore (1977, pp. 329–330), adjusted results using the ANCOVA technique should be interpreted cautiously for two reasons. First, ANCOVA “is not a substitute for random assignment, but it has become a conventionally accepted technique for handling initial group differences in quasi-experimental data” (p. 329). Second, the larger the initial differences between treatment and control groups, the weaker the strength of the results (p. 329).