Politeness
A spider trap causes a web crawler to enter something like an infinite loop, which wastes the spider's resources, lowers its productivity, and, in the case of a poorly written crawler, can crash the program. Polite spiders alternate requests between different hosts, and don't request documents from the same server more than once every several seconds, meaning that a "polite" web crawler is affected to a much lesser degree than an "impolite" crawler.
In addition, sites with spider traps usually have a robots.txt telling bots not to go to the trap, so a legitimate "polite" bot would not fall into the trap, whereas an "impolite" bot which disregards the robots.txt settings would be affected by the trap.
Read more about this topic: Spider Trap
Famous quotes containing the word politeness:
“It is wise to apply the oil of refined politeness to the mechanism of friendship.”
—Colette [Sidonie Gabrielle Colette] (1873–1954)
“People are less self-conscious in the intimacy of family life and during the anxiety of a great sorrow. The dazzling varnish of an extreme politeness is then less in evidence, and the true qualities of the heart regain their proper proportions.”
—Stendhal [Marie Henri Beyle] (1783–1842)
“Nobody thanks a witty man for politeness when he accommodates himself to a society in which it is not polite to display wit.”
—Friedrich Nietzsche (1844–1900)