History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“In history an additional result is commonly produced by human actions beyond that which they aim at and obtain—that which they immediately recognize and desire. They gratify their own interest; but something further is thereby accomplished, latent in the actions in question, though not present to their consciousness, and not included in their design.”
—Georg Wilhelm Friedrich Hegel (1770–1831)
“The history of the Victorian Age will never be written: we know too much about it.”
—Lytton Strachey (1880–1932)
“The history of all previous societies has been the history of class struggles.”
—Karl Marx (1818–1883)