Wizard of Oz Experiment - Origin

Origin

John F. (“Jeff”) Kelley coined the terms “Wizard of OZ” and “OZ Paradigm” for this purpose circa 1980 to describe the method he developed during his dissertation work at The Johns Hopkins University. (His dissertation advisor was the late Professor Alphonse Chapanis, the “Godfather of Human Factors and Engineering Psychology”). Amusingly enough, in addition to some one-way mirrors and such, there literally was a blackout curtain separating Jeff, as the “Wizard”, from view by the participant during the study.

The “Experimenter-in-the-Loop” technique had been pioneered at Chapanis’ Communications Research Lab at Johns Hopkins as early as 1975 (J.F. Kelley arrived in 1978). W. Randolph Ford used the experimenter-in-the-loop technique with his innovative CHECKBOOK program wherein he obtained language samples in a naturalistic setting. In Ford’s method, a preliminary version of the natural language processing system would be placed in front of the user. When the user entered a syntax that was not recognized, they would receive a “could you rephrase that?” prompt from the software. After the session, the algorithms for processing the newly obtained samples would be created or enhanced and another session would take place. This approach led to the eventual development of his natural language processing technique, "Multi-Stage Pattern Reduction". Dr. Ford's recollection was that Dr. Kelley did in fact coin the phrase "Wizard of Oz Paradigm" but that the technique had been employed in at least two separate studies before Dr. Kelley had started conducting studies at the Johns Hopkins Telecommunications Lab.

In that employment the experimenter (the “Wizard”) sat at a terminal in an adjacent room separated by a one-way mirror so the subject could be observed. Every input from the user was processed correctly by a combination of software processing and real-time experimenter intervention. As the process was repeated in subsequent sessions, more and more software components were added so that the experimenter had less and less to do during each session until asymptote was reached on phrase/word dictionary growth and the experimenter could “go get a cup of coffee” during the session (which at this point was a cross-validation of the final system’s unattended performance).

A final point: Dr. Kelley's recollection of the coinage of the term is backed up by that of the late Professor Al Chapanis. In their 1985 University of Michigan technical report, Green and Wei-Haas state the following: The first appearance of the "Wizard of Oz" name in print was in Jeff Kelley's thesis (Kelley, 1983a, 1983b, 1984a). It is thought the name was coined in response to a question at a graduate seminar at Hopkins (Chapanis, 1984; Kelley, 1984b). "What happens if the subject sees the experimenter ?" Kelley answered, "Well, that's just like what happened to Dorothy in the Wizard of Oz." And so the name stuck. (cited by permission)

There is also a passing reference to planned use of the "Wizard of Oz experiments" in a 1982 proceedings paper by Ford and Smith.

One fact, presented in Kelley's dissertation, about the etymology of the term in this context: Dr. Kelley did originally have a definition for the “OZ” acronym (aside from the obvious parallels with the 1900 book The Wonderful Wizard of Oz by L. Frank Baum). “Offline Zero” was a reference to the fact that an experimenter (the “Wizard”) was interpreting the users’ inputs in real time during the simulation phase.

Similar experimental setups had occasionally been used earlier, but without the "wizard of oz" name. Design researcher Nigel Cross conducted studies in the 1960s and 70s with "simulated" computer-aided design systems where the purported simulator was actually a human operator. As he explained, "All that the user perceives of the system is this remote-access console, and the remainder is a black box to him. ... one may as well fill the black box with people as with machinery. Doing so provides a comparatively cheap simulator, with the remarkable advantages of the human operator's flexibility, memory, and intelligence, and which can be reprogrammed to give a wide range of computer roles merely by changing the rules of operation. It sometimes lacks the real computer's speed and accuracy, but a team of experts working simultaneously can compensate to a sufficient degree to provide an acceptable simulation."