Politeness
A spider trap causes a web crawler to enter something like an infinite loop, which wastes the spider's resources, lowers its productivity, and, in the case of a poorly written crawler, can crash the program. Polite spiders alternate requests between different hosts, and don't request documents from the same server more than once every several seconds, meaning that a "polite" web crawler is affected to a much lesser degree than an "impolite" crawler.
In addition, sites with spider traps usually have a robots.txt telling bots not to go to the trap, so a legitimate "polite" bot would not fall into the trap, whereas an "impolite" bot which disregards the robots.txt settings would be affected by the trap.
Read more about this topic: Spider Trap
Famous quotes containing the word politeness:
“The highest perfection of politeness is only a beautiful edifice, built, from the base to the dome, of ungraceful and gilded forms of charitable and unselfish lying.”
—Mark Twain [Samuel Langhorne Clemens] (1835–1910)
“People are less self-conscious in the intimacy of family life and during the anxiety of a great sorrow. The dazzling varnish of an extreme politeness is then less in evidence, and the true qualities of the heart regain their proper proportions.”
—Stendhal [Marie Henri Beyle] (1783–1842)
“It is wise to apply the oil of refined politeness to the mechanism of friendship.”
—Colette [Sidonie Gabrielle Colette] (1873–1954)