Technical Measures To Stop Bots
The administrator of a website can use various measures to stop or slow a bot. Some techniques include:
- If the application is well behaved, adding entries to robots.txt will be adhered to. Google and other well-behaved bots can be stopped this way.
- Blocking an IP address. This will also block all browsing from that address.
- Disabling any web service API that the website's system might expose.
- Bots sometimes declare who they are and can be blocked on that basis; 'googlebot' is an example. Some bots make no distinction between themselves and a human browser.
- Bots can be blocked by excess traffic monitoring.
- Bots can sometimes be blocked with tools to verify that it is a real person accessing the site, like a CAPTCHA. Bots are sometimes coded to explicitly break specific Captcha patterns.
- Commercial anti-bot services: Several companies, such as Distil, SiteBlackBox and Sentor, offer anti-bot and anti-scraping services for websites. A few Web Application Firewalls have limited bot detection capabilities as well.
- Locating bots with a honeypot or other method to identify the IP addresses of automated crawlers.
- Using CSS sprites to display such data as phone numbers or email addresses, at the cost of accessibility to screen reader users.
Read more about this topic: Web Scraping
Famous quotes containing the words technical, measures and/or stop:
“A technical objection is the first refuge of a scoundrel.”
—Heywood Broun (1888–1939)
“Him the Almighty Power
Hurld headlong flaming from th’ Ethereal Skie
With hideous ruine and combustion down
To bottomless perdition, there to dwell
In Adamantine Chains and penal Fire,
Who durst defie th’ Omnipotent to Arms.
Nine times the Space that measures Day and Night
To mortal men, he with his horrid crew
Lay vanquisht, rowling in the fiery Gulfe”
—John Milton (1608–1674)
“Happiness is always a by-product. It is probably a matter of temperament, and for anything I know it may be glandular. But it is not something that can be demanded from life, and if you are not happy you had better stop worrying about it and see what treasures you can pluck from your own brand of unhappiness.”
—Robertson Davies (b. 1913)