Wizard of Oz Experiment - Significance

Significance

The Wizard of OZ method (unlike the eponymous “wizard” in the film) is very powerful. In its original application, Dr. Kelley was able to create a simple keyboard-input natural language recognition system that far exceeded the recognition rates of any of the far more complex systems of the day.

The thinking current among many computer scientists and linguists at the time was that, in order for a computer to be able to “understand” natural language enough to be able to assist in useful tasks, the software would have to be attached to a formidable “dictionary” having a large number of categories for each word. The categories would enable a very complex parsing algorithm to unravel the ambiguities inherent in naturally produced language. The daunting task of creating such a dictionary led many to believe that computers simply would never truly “understand” language until they could be “raised” and “experience life” as humans, since humans seem to apply a life’s worth of experiences to the interpretation of language.

The key enabling factor for the first use of the OZ method was that the system was designed to work in a single context (calendar-keeping), which constrained the complexity of language encountered from users to the extent where a simple language processing model was sufficient to meet the goals of the application. The processing model was a two-pass keyword/keyphrase matching approach, based loosely on the algorithms employed in Weizenbaum’s famous Eliza program. By inducing participants to generate language samples in the context of solving an actual task (using a computer that they believed actually understood what they were typing), the variety and complexity of the lexical structures gathered was greatly reduced and simple keyword matching algorithms could be developed to address the actual language collected.

This first use of OZ was in the context of an Iterative design approach. In the early development sessions, the experimenter simulated the system in toto, performing all the database queries and composing all the responses to the participants by hand. As the process matured, the experimenter was able to replace human interventions, piece by piece, with newly-created developed code (which, at each phase, was designed to accurately process all the inputs that were generated in preceding steps). By the end of the process, the experimenter was able to observe the sessions in a “hands-off” mode (and measure the recognition rates of the completed program).

OZ was important because it addressed the obvious criticism:

Who can afford to use an iterative method to build a separate natural language system (dictionaries, syntax) for each new context? Wouldn’t you be forever adding new structures and algorithms to handle each new batch of inputs?

The answer turned out to be:

By using an empirical approach like OZ, anyone can afford to do this; Dr. Kelley’s dictionary and syntax growth reached asymptote (achieving from 86% to 97% recognition rates, depending on the measurements employed) after only 16 experimental trials and the resulting program, with dictionaries, was less than 300k of code.

In the 23 years that followed initial publication, the OZ method has been employed in a wide variety of settings, notably in the prototyping and usability testing of proposed user interface designs in advance of having actual application software in place.