Dan-dan Jin
An optimized Speculative Execution Strategy Based on Local Data Prediction in Heterogeneous Hadoop Environment
Jin, Dan-dan; Liu, Qi; Liu, Xiaodong; Linge, Nigel
Abstract
Hadoop is a famous parallel computing framework that is applied to process large-scale data, but there exists such a task in hadoop framework, which is called “Straggling task” and has a serious impact on Hadoop. Speculative execution (SE) is an effective way to deal with the “Straggling task” by monitoring the real-time rate of running tasks and back up the “Straggler” on another node to increase the opportunity of completing backup task ahead of original. There are many problems in the proposed SE strategies, such as “Straggling task” misjudgment, improper selection of backup nodes, which will result in inefficient implementation of SE. In this paper, we propose an optimized SE strategy based on local data prediction, it collects task execution information in real time and uses Local regression to predict remaining time of the current task, and selects the appropriate backup task node according to the actual requirements, at the same time, it uses the consumption and benefit model to maximizes the effectiveness of SE. Finally, the strategy is implemented in Hadoop-2.6.0, the experiment proves that the optimized strategy not only enhances the accuracy of selecting the “Straggler” task candidates, but also shows better performance in heterogeneous Hadoop environment.
Citation
Jin, D., Liu, Q., Liu, X., & Linge, N. (2019). An optimized Speculative Execution Strategy Based on Local Data Prediction in Heterogeneous Hadoop Environment. Journal of Computers, 30(3), 130-142. https://doi.org/10.3966/199115992019063003010
Journal Article Type | Article |
---|---|
Acceptance Date | Jul 17, 2018 |
Online Publication Date | Dec 31, 2018 |
Publication Date | Jun 30, 2019 |
Deposit Date | Oct 5, 2018 |
Publicly Available Date | Dec 31, 2018 |
Journal | Journal of Computers |
Print ISSN | 1991-1599 |
Publisher | Zhonghua Minguo Diannao Xuehui |
Peer Reviewed | Peer Reviewed |
Volume | 30 |
Issue | 3 |
Pages | 130-142 |
DOI | https://doi.org/10.3966/199115992019063003010 |
Keywords | Hadoop, Speculative execution, Straggling task, Local Regression, Prediction accuracy |
Public URL | http://researchrepository.napier.ac.uk/Output/1311601 |
Files
An optimized Speculative Execution Strategy Based on Local Data Prediction...
(648 Kb)
Document
An optimized Speculative Execution Strategy Based on...
(865 Kb)
PDF
You might also like
Intelligent Question Answering System Based on Knowledge Graph
(2022)
Conference Proceeding
Improving Domestic NILM Using An Attention- Enabled Seq2Point Learning Approach
(2022)
Conference Proceeding
An Intelligent Method for Upper Limb Posture Recognition Based on Limited MEMS Data
(2022)
Conference Proceeding
A Residual Concatenate FCN Approach to Beam Blockage Correction in Weather Radars
(2021)
Presentation / Conference
Downloadable Citations
About Edinburgh Napier Research Repository
Administrator e-mail: repository@napier.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search