History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“In the history of the United States, there is no continuity at all. You can cut through it anywhere and nothing on this side of the cut has anything to do with anything on the other side.”
—Henry Brooks Adams (1838–1918)
“A great proportion of the inhabitants of the Cape are always thus abroad about their teaming on some ocean highway or other, and the history of one of their ordinary trips would cast the Argonautic expedition into the shade.”
—Henry David Thoreau (1817–1862)
“The history of our era is the nauseating and repulsive history of the crucifixion of the procreative body for the glorification of the spirit.”
—D.H. (David Herbert)