Crawling The Deep Web
A vast amount of Web pages lie in the deep or invisible Web. These pages are typically only accessible by submitting queries to a database, and regular crawlers are unable to find these pages if there are no links that point to them. Google's Sitemaps protocol and mod oai are intended to allow discovery of these deep-Web resources.
Deep Web crawling also multiplies the number of Web links to be crawled. Some crawlers only take some of the -shaped URLs. In some cases, such as the Googlebot, Web crawling is done on all text contained inside the hypertext content, tags, or text.
Strategic approaches may be taken to target deep-Web content. With a technique called screen scraping, specialized software may be customized to automatically and repeatedly query a given Web form with the intention of aggregating the resulting data. Such software can be used to span multiple Web forms across multiple Websites. Data extracted from the results of one Web form submission can be taken and applied as input to another Web form thus establishing continuity across the Deep Web in a way not possible with traditional web crawlers.
Read more about this topic: Web Crawler
Famous quotes containing the words crawling, deep and/or web:
“You know, Frank, I’m beginning to get a new perspective on this crawling little animal known as man. Why a dog or a cat or a bird is cleverer than any human. They sense me immediately. But these shrewd detectives of yours—. Take away one of man’s senses and you render him helpless.”
—Lester Cole (1904–1985)
“We are not very much to blame for our bad marriages. We live amid hallucinations; and this especial trap is laid to trip our feet with, and all are tripped up first and last. But the mighty Mother who had been so sly with us, as if she felt that she owed us some indemnity, insinuates into the Pandora-box of marriage some deep and serious benefits, and some great joys.”
—Ralph Waldo Emerson (1803–1882)
“However, our fates at least are social. Our courses do not diverge; but as the web of destiny is woven it is fulled, and we are cast more and more into the centre. Men naturally, though feebly, seek this alliance, and their actions faintly foretell it.”
—Henry David Thoreau (1817–1862)