Technical Measures To Stop Bots
The administrator of a website can use various measures to stop or slow a bot. Some techniques include:
- If the application is well behaved, adding entries to robots.txt will be adhered to. Google and other well-behaved bots can be stopped this way.
- Blocking an IP address. This will also block all browsing from that address.
- Disabling any web service API that the website's system might expose.
- Bots sometimes declare who they are and can be blocked on that basis; 'googlebot' is an example. Some bots make no distinction between themselves and a human browser.
- Bots can be blocked by excess traffic monitoring.
- Bots can sometimes be blocked with tools to verify that it is a real person accessing the site, like a CAPTCHA. Bots are sometimes coded to explicitly break specific Captcha patterns.
- Commercial anti-bot services: Companies offer anti-bot and anti-scraping services for websites. A few web application firewalls have limited bot detection capabilities as well.
- Locating bots with a honeypot or other method to identify the IP addresses of automated crawlers.
- Using CSS sprites to display such data as phone numbers or email addresses, at the cost of accessibility to screen reader users.
Read more about this topic: Web Harvesting
Famous quotes containing the words technical, measures and/or stop:
“Woman is the future of man. That means that the world which was once formed in man’s image will now be transformed to the image of woman. The more technical and mechanical, cold and metallic it becomes, the more it will need the kind of warmth that only the woman can give it. If we want to save the world, we must adapt to the woman, let ourselves be led by the woman, let ourselves be penetrated by the Ewigweiblich, the eternally feminine!”
—Milan Kundera (b. 1929)
“There are other measures of self-respect for a man, than the number of clean shirts he puts on every day.”
—Ralph Waldo Emerson (1803–1882)
“Let us stop being afraid. Of our own thoughts, our own minds. Of madness, our own or others’. Stop being afraid of the mind itself, its astonishing functions and fandangos, its complications and simplifications, the wonderful operation of its machinery—more wonderful because it is not machinery at all or predictable.”
—Kate Millett (b. 1934)