An Optimized Speculative Execution Strategy Based on Local Data Prediction in a Heterogeneous Hadoop Environment

Liu, Xiaodong; Liu, Qi

Abstract

Hadoop is a famous distributed computing framework that is applied to process large-scale data. "Straggling tasks" have a serious impact on Hadoop performance due to imbalance of slow tasks distribution. Speculative execution (SE) presents a way to deal with Straggling tasks by monitoring the real-time progress of running tasks and replicating potential "Stragglers" on another node to increase the opportunity of completing backup tasks ahead of original. Current proposed SE strategies meet their challenges such as misjudgment of "Straggling tasks", improper selection of backup nodes, etc., which result in inefficient performance of the SE and its Hadoop system. In this paper, we propose an optimized SE strategy based on local data prediction, which collects task execution information in real time and uses Locally Weighted Regression (LWR) to predict remaining time of each running tasks, and selects an appropriate backup task node according to the actual requirements. It also combines a cost-benefit model to maximize the effectiveness of SE. According to the results, the proposed SE strategy implemented in Hadoop-2.6.0 enhances the accuracy of selecting potential Straggler task candidates, and shows better performance in various situations in a heterogeneous Hadoop environment.

Citation

Liu, X., & Liu, Q. (2017). An Optimized Speculative Execution Strategy Based on Local Data Prediction in a Heterogeneous Hadoop Environment. In Proceedings of 2017 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC) (128-131). https://doi.org/10.1109/cse-euc.2017.208

• Conference Name: 22017 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC)
• Conference Location: Guangzhou, Guangdong, China
• Start Date: Jul 21, 2017
• End Date: Jul 24, 2017
• Acceptance Date: Jun 20, 2017
• Online Publication Date: Aug 18, 2017
• Publication Date: Aug 18, 2017
• Deposit Date: Nov 15, 2017
• Publicly Available Date: Nov 16, 2017
• Publisher: Institute of Electrical and Electronics Engineers
• Pages: 128-131
• Book Title: Proceedings of 2017 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC)
• Chapter Number: NA
• ISBN: 9781538632208
• DOI: https://doi.org/10.1109/cse-euc.2017.208
• Keywords: Hadoop, Speculative Execution, Straggling Task, LWR, Prediction Accuracy
• Public URL: http://researchrepository.napier.ac.uk/Output/1010633