History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“Bias, point of view, fury—are they ... so dangerous and must they be ironed out of history, the hills flattened and the contours leveled? The professors talk ... about passion and point of view in history as a Calvinist talks about sin in the bedroom.”
—Catherine Drinker Bowen (1897–1973)
“The history of medicine is the history of the unusual.”
—Robert M. Fresco, and Jack Arnold. Prof. Gerald Deemer (Leo G. Carroll)
“The custard is setting; meanwhile
I not only have my own history to worry about
But am forced to fret over insufficient details related to large
Unfinished concepts that can never bring themselves to the point
Of being, with or without my help, if any were forthcoming.”
—John Ashbery (b. 1927)