Politeness
A spider trap causes a web crawler to enter something like an infinite loop, which wastes the spider's resources, lowers its productivity, and, in the case of a poorly written crawler, can crash the program. Polite spiders alternate requests between different hosts, and don't request documents from the same server more than once every several seconds, meaning that a "polite" web crawler is affected to a much lesser degree than an "impolite" crawler.
In addition, sites with spider traps usually have a robots.txt telling bots not to go to the trap, so a legitimate "polite" bot would not fall into the trap, whereas an "impolite" bot which disregards the robots.txt settings would be affected by the trap.
Read more about this topic: Spider Trap
Famous quotes containing the word politeness:
“Nobody thanks a witty man for politeness when he accommodates himself to a society in which it is not polite to display wit.”
—Friedrich Nietzsche (1844–1900)
“It is wise to apply the oil of refined politeness to the mechanism of friendship.”
—Colette [Sidonie Gabrielle Colette] (1873–1954)
“I never went near the Wellesley College chapel in my four years there, but I am still amazed at the amount of Christian charity that school stuck us all with, a kind of glazed politeness in the face of boredom and stupidity. Tolerance, in the worst sense of the word.... How marvelous it would have been to go to a women’s college that encouraged impoliteness, that rewarded aggression, that encouraged argument.”
—Nora Ephron (b. 1941)