A Generator of Hadoop MapReduce Programs that Manipulate One-dimensional Arrays

• Miyazaki Reina

  Graduate School of Engineering, Kochi University of Technology

• Matsuzaki Kiminori

  School of Information, Kochi University of Technology

• Sato Shigeyuki

  School of Information, Kochi University of Technology

抄録

MapReduce is a framework for large-scale data processing proposed by Google, and its open-source implementation, Hadoop MapReduce, is now widely used. Several language systems have been proposed to make developing MapReduce programs easier, for instance, Sawzall, FlumeJava, Pig, Hive, and Crunch. These language systems mainly target applications that can be naturally solved by using a MapReduce-like programming model. In this study, we propose a new MapReduce-program generator that accepts programs manipulating one-dimensional arrays. By using the proposed generator, users only need to write sequential programs to generate Hadoop MapReduce programs automatically. We applied some program optimization techniques to the generation of Hadoop MapReduce programs. In this paper, we also report our experiment results that compare programs generated by the proposed generator with hand-written MapReduce programs.

収録刊行物

• Journal of Information Processing

  Journal of Information Processing 25 (0), 841-851, 2017

  一般社団法人 情報処理学会

キーワード

• large-scale data processing
• MapReduce
• Hadoop MapReduce
• program generator

詳細情報

• CRID: 1390282680272625792
• NII論文ID: 130005990949
• NII論文ID: 170000148795
• NII書誌ID: AA11464814
• ISSN: 18827802
• ISSN: 18826652
• DOI: 10.2197/ipsjjip.25.841
• Web Site: http://id.nii.ac.jp/1001/00182661/
• Web Site: https://www.jstage.jst.go.jp/article/ipsjjip/25/0/25_841/_pdf
• 本文言語コード: en
• データソース種別:
  • JaLC
  • IRDB
  • Crossref
  • CiNii Articles
  • KAKEN
• 抄録ライセンスフラグ: 使用不可