Skip to main content

Research Repository

Advanced Search

An Adaptively Speculative Execution Strategy Based on Real-Time Resource Awareness in a Multi-Job Heterogeneous Environment

Liu, Qi; Cai, Weidong; Liu, Qiang; Shen, Jian; Fu, Zhangjie; Liu, Xiaodong; Linge, Nigel

Authors

Qi Liu

Weidong Cai

Qiang Liu

Jian Shen

Zhangjie Fu

Nigel Linge



Abstract

MapReduce (MRV1), a popular programming model, proposed by Google, has been well used to process large datasets in Hadoop, an open source cloud platform. Its new version MapReduce 2.0 (MRV2) developed along with the emerging of Yarn has achieved obvious improvement over MRV1. However, MRV2 suffers from long finishing time on certain types of jobs. Speculative Execution (SE) has been presented as an approach to the problem above by backing up those delayed jobs from low-performance machines to higher ones. In this paper, an adaptive SE strategy (ASE) is presented in Hadoop-2.6.0. Experiment results have depicted that the ASE duplicates tasks according to real-time resources usage among work nodes in a cloud. In addition, the performance of MRV2 is largely improved using the ASE strategy on job execution time and resource consumption, whether in a multi-job environment.

Citation

Liu, Q., Cai, W., Liu, Q., Shen, J., Fu, Z., Liu, X., & Linge, N. (2017). An Adaptively Speculative Execution Strategy Based on Real-Time Resource Awareness in a Multi-Job Heterogeneous Environment. KSII transactions on internet and information systems, 11(2), https://doi.org/10.3837/tiis.2017.02.004

Journal Article Type Article
Acceptance Date Jan 9, 2017
Online Publication Date Feb 28, 2017
Publication Date Mar 10, 2017
Deposit Date Apr 4, 2017
Publicly Available Date Mar 29, 2024
Journal KSII Transactions on Internet and Information Systems
Print ISSN 1976-7277
Peer Reviewed Peer Reviewed
Volume 11
Issue 2
DOI https://doi.org/10.3837/tiis.2017.02.004
Keywords Computer Networks and Communications; Information Systems
Public URL http://researchrepository.napier.ac.uk/Output/824928

Files

An Adaptively Speculative Execution Strategy Based on Real-Time Resource Awareness in a Multi-Job Heterogeneous Environment (704 Kb)
PDF







You might also like



Downloadable Citations