History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“If usually the “present age” is no very long time, still, at our pleasure, or in the service of some such unity of meaning as the history of civilization, or the study of geology, may suggest, we may conceive the present as extending over many centuries, or over a hundred thousand years.”
—Josiah Royce (1855–1916)
“I believe that history might be, and ought to be, taught in a new fashion so as to make the meaning of it as a process of evolution intelligible to the young.”
—Thomas Henry Huxley (1825–95)
“In all history no class has been enfranchised without some selfish motive underlying. If to-day we could prove to Republicans or Democrats that every woman would vote for their party, we should be enfranchised.”
—Carrie Chapman Catt (1859–1947)