Technical Measures To Stop Bots
The administrator of a website can use various measures to stop or slow a bot. Some techniques include:
- If the application is well behaved, adding entries to robots.txt will be adhered to. Google and other well-behaved bots can be stopped this way.
- Blocking an IP address. This will also block all browsing from that address.
- Disabling any web service API that the website's system might expose.
- Bots sometimes declare who they are and can be blocked on that basis; 'googlebot' is an example. Some bots make no distinction between themselves and a human browser.
- Bots can be blocked by excess traffic monitoring.
- Bots can sometimes be blocked with tools to verify that it is a real person accessing the site, like a CAPTCHA. Bots are sometimes coded to explicitly break specific Captcha patterns.
- Commercial anti-bot services: Several companies, such as Distil, SiteBlackBox and Sentor, offer anti-bot and anti-scraping services for websites. A few Web Application Firewalls have limited bot detection capabilities as well.
- Locating bots with a honeypot or other method to identify the IP addresses of automated crawlers.
- Using CSS sprites to display such data as phone numbers or email addresses, at the cost of accessibility to screen reader users.
Read more about this topic: Web Scraping
Famous quotes containing the words technical, measures and/or stop:
“In effect, to follow, not to force the public inclination; to give a direction, a form, a technical dress, and a specific sanction, to the general sense of the community, is the true end of legislature.”
—Edmund Burke (1729–1797)
“Him the Almighty Power
Hurld headlong flaming from th’ Ethereal Skie
With hideous ruine and combustion down
To bottomless perdition, there to dwell
In Adamantine Chains and penal Fire,
Who durst defie th’ Omnipotent to Arms.
Nine times the Space that measures Day and Night
To mortal men, he with his horrid crew
Lay vanquisht, rowling in the fiery Gulfe”
—John Milton (1608–1674)
“Despite everybody who has been born and has died, the world has just gone on. I mean, look at Napoleon—but we went right on. Look at Harpo Marx—the world went around, it didn’t stop for a second. It’s sad but true. John Kennedy, right?”
—Bob Dylan [Robert Allen Zimmerman] (b. 1941)