Producer-Consumer型モジュールで構成された並列分散Webクローラの開発 A Parallel Distributed Web Crawler Consisting of Producer-Consumer Modules

この論文にアクセスする

この論文をさがす

著者

抄録

Webクローラは,クローリング済みURLの検出やWebサーバに対する連続アクセス防止といった処理を実行しながらデータ収集を行う必要がある.Web空間に存在する大量のURLに対して高速な収集を実現するために並列分散クローリングが求められるが,省資源でのクローリングを行うためにも,処理の時間計算量と空間計算量の削減に加え,計算機間の負荷分散も必要である.本論文で提案するWebクローラは,クローリング処理をProducer-Consumer型のモジュール群で実行することにより,これまでの被クロールWebサイト単位での負荷分散でなく,Webクローラを構成するモジュール単位での負荷分散を実現する.つまり,Webクローラを構成する各モジュールが必要とする計算機資源に応じた分散処理が可能になり,計算機間での計算負荷やメモリ使用量の偏りを改善することができる.また,ホスト名やURLを管理するモジュールは時間計算量と空間計算量に優れたデータ構造を利用して構成されており,大規模なクローリングが省資源で可能になる.Web crawlers must collect Web data while performing tasks such as detecting crawled URLs and preventing consecutive accesses to a particular Web server. Parallel-distributed crawling is carried out at a high speed for the enormous number of URLs existing on the Web. However, in order to crawl efficiently, a crawler must realize load balancing between computers in addition to reducing time and space complexities in the crawling process. The Web crawler proposed in this paper crawls the Web using producer-consumer modules, which compose the crawler, and it realizes load balancing per module and not per crawled Web site. In other words, it realizes load balancing that is appropriate to certain computer resources necessary for the modules that compose the Web crawler; in this way, it improves biases in computation loads and memory utilization between computers. Moreover, the crawler is able to crawl the Web on a large scale while conserving resources, because the modules that manage host names or URLs are implemented by data structures that are temporally and spatially efficient.

Web crawlers must collect Web data while performing tasks such as detecting crawled URLs and preventing consecutive accesses to a particular Web server. Parallel-distributed crawling is carried out at a high speed for the enormous number of URLs existing on the Web. However, in order to crawl efficiently, a crawler must realize load balancing between computers in addition to reducing time and space complexities in the crawling process. The Web crawler proposed in this paper crawls the Web using producer-consumer modules, which compose the crawler, and it realizes load balancing per module and not per crawled Web site. In other words, it realizes load balancing that is appropriate to certain computer resources necessary for the modules that compose the Web crawler; in this way, it improves biases in computation loads and memory utilization between computers. Moreover, the crawler is able to crawl the Web on a large scale while conserving resources, because the modules that manage host names or URLs are implemented by data structures that are temporally and spatially efficient.

収録刊行物

  • 情報処理学会論文誌データベース(TOD)  

    情報処理学会論文誌データベース(TOD) 6(2), 85-97, 2013-03-29 

    一般社団法人情報処理学会

各種コード

  • NII論文ID(NAID)
    110009579515
  • NII書誌ID(NCID)
    AA11464847
  • 本文言語コード
    JPN
  • 資料種別
    Article
  • ISSN
    1882-7799
  • NDL 記事登録ID
    024421708
  • NDL 請求記号
    YH247-812
  • データ提供元
    NDL  IPSJ 
ページトップへ