Crawling The Deep Web
A vast amount of Web pages lie in the deep or invisible Web. These pages are typically only accessible by submitting queries to a database, and regular crawlers are unable to find these pages if there are no links that point to them. Google's Sitemaps protocol and mod oai are intended to allow discovery of these deep-Web resources.
Deep Web crawling also multiplies the number of Web links to be crawled. Some crawlers only take some of the -shaped URLs. In some cases, such as the Googlebot, Web crawling is done on all text contained inside the hypertext content, tags, or text.
Strategic approaches may be taken to target deep-Web content. With a technique called screen scraping, specialized software may be customized to automatically and repeatedly query a given Web form with the intention of aggregating the resulting data. Such software can be used to span multiple Web forms across multiple Websites. Data extracted from the results of one Web form submission can be taken and applied as input to another Web form thus establishing continuity across the Deep Web in a way not possible with traditional web crawlers.
Read more about this topic: Web Crawler
Famous quotes containing the words crawling, deep and/or web:
“As I stand over the insect crawling amid the pine needles on the forest floor, and endeavoring to conceal itself from my sight, and ask myself why it will cherish those humble thoughts, and hide its head from me who might, perhaps, be its benefactor, and impart to its race some cheering information, I am reminded of the greater Benefactor and Intelligence that stands over me the human insect.”
—Henry David Thoreau (1817–1862)
“He began therefore to invest the fortress of my heart by a circumvallation of distant bows and respectful looks; he then entrenched his forces in the deep caution of never uttering an unguarded word or syllable. His designs being yet covered, he played off from several quarters a large battery of compliments. But here he found a repulse from the enemy by an absolute rejection of such fulsome praise, and this forced him back again close into his former trenches.”
—Sarah Fielding (1710–1768)
“Thou blind man’s mark, thou fool’s self-chosen snare,
Fond Fancy’s scum and dregs of scattered thought,
Band of all evils, cradle of causeless care,
Thou web of will whose end is never wrought;
Desire! desire, I have too dearly bought
With price of mangled mind thy worthless ware;”
—Sir Philip Sidney (1554–1586)