Query Selection in Deep Web Crawling: Help your crawler efficiently retrieve data from the largest data sources in the Web

The deep web is the content that is dynamically generated from data sources such as databases or file system. Unlike surface web where web pages are collected by following the hyperlinks embedded inside collected pages, data from a deep web data source is guarded by a search interface and only can be retrieved by queries. The amount of data in deep web exceeds by far that of the surface web. This calls for deep web crawlers to excavate the data so that they can be used, indexed, and searched upon in an integrated environment. Crawling deep web is the process of collecting data from search interfaces by issuing queries. One of the major challenges in crawling deep web is the selection of the queries so that most of the data can be retrieved at a low cost. This work first comprehensively introduces the state-of-art work in query selection techniques for crawling, then in-depth analyzes the remaining problems, such as cold start problem and return limit problem, and finally presents a novel technique to address them.