History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
““And now this is the way in which the history of your former life has reached my ears!” As he said this he held out in his hand the fatal letter.”
—Anthony Trollope (1815–1882)
“It is the true office of history to represent the events themselves, together with the counsels, and to leave the observations and conclusions thereupon to the liberty and faculty of every man’s judgement.”
—Francis Bacon (1561–1626)
“Certainly there is not the fight recorded in Concord history, at least, if in the history of America, that will bear a moment’s comparison with this, whether for the numbers engaged in it, or for the patriotism and heroism displayed.”
—Henry David Thoreau (1817–1862)