History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“Boys forget what their country means by just reading “the land of the free” in history books. Then they get to be men, they forget even more. Liberty’s too precious a thing to be buried in books.”
—Sidney Buchman (1902–1975)
“Every generation rewrites the past. In easy times history is more or less of an ornamental art, but in times of danger we are driven to the written record by a pressing need to find answers to the riddles of today.... In times of change and danger when there is a quicksand of fear under men’s reasoning, a sense of continuity with generations gone before can stretch like a lifeline across the scary present and get us past that idiot delusion of the exceptional Now that blocks good thinking.”
—John Dos Passos (1896–1970)
“You that would judge me do not judge alone
This book or that, come to this hallowed place
Where my friends’ portraits hang and look thereon;
Ireland’s history in their lineaments trace;
Think where man’s glory most begins and ends
And say my glory was I had such friends.”
—William Butler Yeats (1865–1939)