About The Standard
If a site owner wishes to give instructions to web robots they must place a text file called robots.txt in the root of the web site hierarchy (e.g. www.example.com/robots.txt). This text file should contain the instructions in a specific format (see examples below). Robots that choose to follow the instructions try to fetch this file and read the instructions before fetching any other file from the web site. If this file doesn't exist web robots assume that the web owner wishes to provide no specific instructions.
A robots.txt file on a website will function as a request that specified robots ignore specified files or directories when crawling a site. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data. Links to pages listed in robots.txt can still appear in search results if they are linked to from a page that is crawled.
For websites with multiple subdomains, each subdomain must have its own robots.txt file. If example.com had a robots.txt file but a.example.com did not, the rules that would apply for example.com would not apply to a.example.com.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word standard:
“Any honest examination of the national life proves how far we are from the standard of human freedom with which we began. The recovery of this standard demands of everyone who loves this country a hard look at himself, for the greatest achievments must begin somewhere, and they always begin with the person. If we are not capable of this examination, we may yet become one of the most distinguished and monumental failures in the history of nations.”
—James Baldwin (1924–1987)