History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“We aspire to be something more than stupid and timid chattels, pretending to read history and our Bibles, but desecrating every house and every day we breathe in.”
—Henry David Thoreau (1817–1862)
“A man will not need to study history to find out what is best for his own culture.”
—Henry David Thoreau (1817–1862)
“In history the great moment is, when the savage is just ceasing to be a savage, with all his hairy Pelasgic strength directed on his opening sense of beauty;—and you have Pericles and Phidias,—and not yet passed over into the Corinthian civility. Everything good in nature and in the world is in that moment of transition, when the swarthy juices still flow plentifully from nature, but their astrigency or acridity is got out by ethics and humanity.”
—Ralph Waldo Emerson (1803–1882)