Frederick Jelinek - Research and Legacy

Research and Legacy

Information theory was a fashionable scientific approach in the mid '50s. However, pioneer Claude Shannon mused in 1956 that this trendiness was dangerous: "Our fellow scientists in many different fields, attracted by the fanfare and by the new avenues opened to scientific analysis, are using these ideas in their own problems. It will be all too easy for our somewhat artificial prosperity to collapse overnight when it is realized that the use of a few exciting words like information, entropy, redundancy, do not solve all our problems." Indeed over the next decade, a combination of factors would shut down application of information theory to natural language processing (NLP) problems, in particular machine translation. One was the 1957 publication of Noam Chomsky's Syntactic Structures, which stated that "probabilistic models give no insight into the basic problems of syntactic structure". This accorded well with the philosophy of the artificial intelligence research of the time, which promoted rule-based approaches. The other factor was to be the 1966 ALPAC report, which recommended that the government stop funding research in machine translation. ALPAC chairman John Pierce later characterised that field as filled with "mad inventors or untrustworthy engineers". He argued that the underlying linguistic problems must be solved before attempts at NLP could be reasonably made. Combined, these elements essentially halted research in the field.

Jelinek had begun to develop an interest in linguistics after the immigration of his wife, who initially enrolled in the linguistics program of the MIT thanks to Roman Jakobson's help. Jelinek often accompanied her to Chomsky's lecture, and even went so far as to discuss the possibility of changing orientation with his adviser. Fano was "really upset", and with the failure of his project with Hockett at Cornell, he did not return to this avenue of research until starting work at IBM. The scope of research at IBM was considerably different from that of most other teams: "While Fred was leading IBM’s effort to solve the general dictation problem during the decade or so following 1972, most other U.S. companies and academic researchers were working on very limited problems or were staying out of the field entirely."

"He was not a pioneer of speech recognition, he was the pioneer of speech recognition."

It was only natural for Jelinek to see speech recognition as an information theory problem: a noisy channel (in this case the acoustic signal)—and yet this was a daring, or even anathema approach to observers. The concept of perplexity was introduced in their first model, New Raleigh Grammar, itself published (1976) in the "now famous paper in the Proceedings of the IEEE called "Continuous Speech Recognition by Statistical Methods"'. The basic noisy channel approach "reduced the speech recognition problem to one of producing two statistical models." Whereas New Raleigh Grammar was a hidden Markov model, Tangora (their next model) was broader and involved n-grams, specifically trigrams. Even though "it was obvious to everyone that this model was hopelessly impoverished", it would remain unimproved upon until another paper of Jelinek himself presented in 1999 (see under "selected publication"). The same trigram approach was applied to phones in single words. Although the identification of parts of speech turned out not to be very useful for speech recognition, tagging methods developed during these projects are now used in various NLP applications.

The incremental research techniques developed at IBM eventually became dominant in the field after DARPA, in the mid-80s, returned to NLP research and imposed that methodology to participating teams, shared common goals, data, and precise evaluation metrics. The Continuous Speech Recognition Group's research, which required large amounts of data to train the algorithms, eventually led to the creation of the Linguistic Data Consortium. In the 80s, although the broader problem of speech recognition remained unsolved, they sought to apply the methods developed to other problems, and came up with two: machine translation and stock value prediction. In fact, a group of IBM searchers eventually went to work for Renaissance Technologies. Jelinek comments: "The performance of the Renaissance fund is legendary, but I have no idea whether any methods we pioneered at IBM have ever been used. My former colleagues will not tell me: theirs is a very hush-hush operation!" Methods very similar to those developed for achieving speech recognition are at the base of most machine translation systems today.

Observers have noted that Pierce's paradigm, according to which engineering achievements in this area would be built on scientific progress, has been inverted, with the achievements in engineering being at the base of a number of scientific findings.

Jelinek's works won "best paper" awards on several occasions, and he received a number of company awards while he worked at IBM. He received the Society Award (for "outstanding technical contributions and leadership") from the IEEE Signal Processing Society for 1997, and the ESCA Medal for Scientific Achievement in 1999. He was a recipient of a IEEE Third Millenium Medal in 2000, the ELRA's first (2004) Antonio Zampolli Prize, the 2005 James L. Flanagan Speech and Audio Processing Award, and the 2009 Lifetime Achievement Award from the Association for Computational Linguistics. He received a honoris causa Ph.D. from Charles University in 2001, was elected to the National Academy of Engineering in 2006 and made one of twelve inaugural fellows of the International Speech Communication Association in 2008.