An Adaptively Speculative Execution Strategy Based on Real-Time Resource Awareness in a Multi-Job Heterogeneous Environment

Liu, Qi; Cai, Weidong; Liu, Qiang; Shen, Jian; Fu, Zhangjie; Liu, Xiaodong; Linge, Nigel

Abstract

MapReduce (MRV1), a popular programming model, proposed by Google, has been well used to process large datasets in Hadoop, an open source cloud platform. Its new version MapReduce 2.0 (MRV2) developed along with the emerging of Yarn has achieved obvious improvement over MRV1. However, MRV2 suffers from long finishing time on certain types of jobs. Speculative Execution (SE) has been presented as an approach to the problem above by backing up those delayed jobs from low-performance machines to higher ones. In this paper, an adaptive SE strategy (ASE) is presented in Hadoop-2.6.0. Experiment results have depicted that the ASE duplicates tasks according to real-time resources usage among work nodes in a cloud. In addition, the performance of MRV2 is largely improved using the ASE strategy on job execution time and resource consumption, whether in a multi-job environment.

Citation

Liu, Q., Cai, W., Liu, Q., Shen, J., Fu, Z., Liu, X., & Linge, N. (2017). An Adaptively Speculative Execution Strategy Based on Real-Time Resource Awareness in a Multi-Job Heterogeneous Environment. KSII transactions on internet and information systems, 11(2), https://doi.org/10.3837/tiis.2017.02.004

• Journal Article Type: Article
• Acceptance Date: Jan 9, 2017
• Online Publication Date: Feb 28, 2017
• Publication Date: Mar 10, 2017
• Deposit Date: Apr 4, 2017
• Publicly Available Date: Jul 31, 2017
• Journal: KSII Transactions on Internet and Information Systems
• Print ISSN: 1976-7277
• Peer Reviewed: Peer Reviewed
• Volume: 11
• Issue: 2
• DOI: https://doi.org/10.3837/tiis.2017.02.004
• Keywords: Computer Networks and Communications; Information Systems
• Public URL: http://researchrepository.napier.ac.uk/Output/824928
• Contract Date: Jul 31, 2017