Robots Exclusion Standard - Examples

Examples

This example tells all robots to visit all files because the wildcard * specifies all robots:

User-agent: * Disallow:

This example tells all robots to stay out of a website:

User-agent: * Disallow: /

The next is an example that tells all robots not to enter four directories of a website:

User-agent: * Disallow: /cgi-bin/ Disallow: /images/ Disallow: /tmp/ Disallow: /private/

Example that tells a specific robot not to enter one specific directory:

User-agent: BadBot # replace 'BadBot' with the actual user-agent of the bot Disallow: /private/

Example that tells all robots not to enter one specific file:

User-agent: * Disallow: /directory/file.html

Note that all other files in the specified directory will be processed.

Example demonstrating how comments can be used:

# Comments appear after the "#" symbol at the start of a line, or after a directive User-agent: * # match all bots Disallow: / # keep them out

It is also possible to list multiple robots with their own rules. The actual robot string is defined by the crawler. A few sites, such as Google, support several user-agent strings that allow you to turn off a subset of their services by using specific user-agent strings.

Example demonstrating multiple user-agents:

User-agent: googlebot # all services Disallow: /private # disallow this directory User-agent: googlebot-news # only the news service Disallow: / # on everything User-agent: * # all robots Disallow: /something # on this folder

Read more about this topic:  Robots Exclusion Standard

Famous quotes containing the word examples:

    “Histories are more full of examples of the fidelity of dogs than of friends.”
    —Alexander Pope (1688–1744)

    “There are many examples of women that have excelled in learning, and even in war, but this is no reason we should bring ‘em all up to Latin and Greek or else military discipline, instead of needle-work and housewifry.”
    —Bernard Mandeville (1670–1733)

    “It is hardly to be believed how spiritual reflections when mixed with a little physics can hold people’s attention and give them a livelier idea of God than do the often ill-applied examples of his wrath.”
    —G.C. (Georg Christoph)