History
In 1967, Kucera and Francis published their classic work Computational Analysis of Present-Day American English, which provided basic statistics on what is known today simply as the Brown Corpus. The Brown Corpus was a carefully compiled selection of current American English, totaling about a million words drawn from a wide variety of sources. Kucera and Francis subjected it to a variety of computational analyses, from which they compiled a rich and variegated opus, combining elements of linguistics, psychology, statistics, and sociology. It has been very widely used in computational linguistics, and was for many years among the most-cited resources in the field.
Shortly after publication of the first lexicostatistical analysis, Boston publisher Houghton-Mifflin approached Kucera to supply a million word, three-line citation base for its new American Heritage Dictionary. This ground-breaking new dictionary, which first appeared in 1969, was the first dictionary to be compiled using corpus linguistics for word frequency and other information.
The initial Brown Corpus had only the words themselves, plus a location identifier for each. Over the following several years part-of-speech tags were applied. The Greene and Rubin tagging program (see under part of speech tagging) helped considerably in this, but the high error rate meant that extensive manual proofreading was required.
The tagged Brown Corpus used a selection of about 80 parts of speech, as well as special indicators for compound forms, contractions, foreign words and a few other phenomena, and formed the basis for many later corpora such as the Lancaster-Oslo-Bergen Corpus. The tagged corpus enabled far more sophisticated statistical analysis, much of it carried out by graduate student Andrew Mackie. Some of the analysis appears in Frequency Analysis of English Usage: Lexicon and Grammar, by Winthrop Nelson Francis and Henry Kucera, Houghton Mifflin (January, 1983) ISBN 0-395-32250-2.
One interesting result is that even for quite large samples, graphing words in order of decreasing frequency of occurrence shows a hyperbola: the frequency of the n-th most frequent word is roughly proportional to 1/n. Thus "the" constitutes nearly 7% of the Brown Corpus, "to" and "of" more than another 3% each; while about half the total vocabulary of about 50,000 words are hapax legomena: words that occur only once in the corpus. This simple rank-vs.-frequency relationship was noted for an extraordinary variety of phenomena by George Kingsley Zipf (for example, see his The Psychobiology of Language), and is known as Zipf's law.
Although the Brown Corpus pioneered the field of corpus linguistics, by now typical corpora (such as the Corpus of Contemporary American English, the British National Corpus or the International Corpus of English) tend to be much larger, on the order of 100 million words.
Read more about this topic: Brown Corpus
Famous quotes containing the word history:
“No one can understand Paris and its history who does not understand that its fierceness is the balance and justification of its frivolity. It is called a city of pleasure; but it may also very specially be called a city of pain. The crown of roses is also a crown of thorns. Its people are too prone to hurt others, but quite ready also to hurt themselves. They are martyrs for religion, they are martyrs for irreligion; they are even martyrs for immorality.”
—Gilbert Keith Chesterton (1874–1936)
“Let it suffice that in the light of these two facts, namely, that the mind is One, and that nature is its correlative, history is to be read and written.”
—Ralph Waldo Emerson (1803–1882)
“The history of men’s opposition to women’s emancipation is more interesting perhaps than the story of that emancipation itself.”
—Virginia Woolf (1882–1941)