Crawling The Deep Web
A vast amount of Web pages lie in the deep or invisible Web. These pages are typically only accessible by submitting queries to a database, and regular crawlers are unable to find these pages if there are no links that point to them. Google's Sitemaps protocol and mod oai are intended to allow discovery of these deep-Web resources.
Deep Web crawling also multiplies the number of Web links to be crawled. Some crawlers only take some of the -shaped URLs. In some cases, such as the Googlebot, Web crawling is done on all text contained inside the hypertext content, tags, or text.
Strategic approaches may be taken to target deep-Web content. With a technique called screen scraping, specialized software may be customized to automatically and repeatedly query a given Web form with the intention of aggregating the resulting data. Such software can be used to span multiple Web forms across multiple Websites. Data extracted from the results of one Web form submission can be taken and applied as input to another Web form thus establishing continuity across the Deep Web in a way not possible with traditional web crawlers.
Read more about this topic: Web Crawler
Famous quotes containing the words crawling, deep and/or web:
“But see, amid the mimic rout
A crawling shape intrude!”
—Edgar Allan Poe (1809–1849)
“The child knows only that he engages in play because it is enjoyable. He isn’t aware of his need to play—a need which has its source in the pressure of unsolved problems. Nor does he know that his pleasure in playing comes from a deep sense of well-being that is the direct result of feeling in control of things, in contrast to the rest of his life, which is managed by his parents or other adults.”
—Bruno Bettelheim (20th century)
“Science is a dynamic undertaking directed to lowering the degree of the empiricism involved in solving problems; or, if you prefer, science is a process of fabricating a web of interconnected concepts and conceptual schemes arising from experiments and observations and fruitful of further experiments and observations.”
—James Conant (1893–1978)