History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“One classic American landscape haunts all of American literature. It is a picture of Eden, perceived at the instant of history when corruption has just begun to set in. The serpent has shown his scaly head in the undergrowth. The apple gleams on the tree. The old drama of the Fall is ready to start all over again.”
—Jonathan Raban (b. 1942)
“... in America ... children are instructed in the virtues of the system they live under, as though history had achieved a happy ending in American civics.”
—Mary McCarthy (1912–1989)
“Regarding History as the slaughter-bench at which the happiness of peoples, the wisdom of States, and the virtue of individuals have been victimized—the question involuntarily arises—to what principle, to what final aim these enormous sacrifices have been offered.”
—Georg Wilhelm Friedrich Hegel (1770–1831)