Yinghang Jiang
An Optimized Resource Scheduling Strategy for Hadoop Speculative Execution Based on Non-cooperative Game Schemes
Jiang, Yinghang; Liu, Qi; Dannah, Williams; Jin, Dandan; Liu, Xiaodong; Sun, Mingxu
Abstract
Hadoop is a well-known parallel computing system for distributed computing and large-scale data processes. “Straggling” tasks, however, have a serious impact on task allocation and scheduling in a Hadoop system. Speculative Execution (SE) is an efficient method of processing “Straggling” Tasks by monitoring real-time running status of tasks and then selectively backing up “Stragglers” in another node to increase the chance to complete the entire mission early. Present speculative execution strategies meet challenges on misjudgement of “Straggling” tasks and improper selection of backup nodes, which leads to inefficient implementation of speculative executive processes. This paper has proposed an Optimized Resource Scheduling strategy for Speculative Execution (ORSE) by introducing non-cooperative game schemes. The ORSE transforms the resource scheduling of backup tasks into a multi-party non-cooperative game problem, where the tasks are regarded as game participants, whilst total task execution time of the entire cluster as the utility function. In that case, the most benefit strategy can be implemented in each computing node when the game reaches a Nash equilibrium point, i.e. the final resource scheduling scheme to be obtained. The strategy has been implemented in Hadoop-2.x. Experimental results depict that the ORSE can maintain the efficiency of speculative executive processes and improve fault-tolerant and computation performance under the circumstances of Normal Load, Busy Load and Busy Load with Skewed Data.
Citation
Jiang, Y., Liu, Q., Dannah, W., Jin, D., Liu, X., & Sun, M. (2020). An Optimized Resource Scheduling Strategy for Hadoop Speculative Execution Based on Non-cooperative Game Schemes. Computers, Materials & Continua, 62(2), 713-729. https://doi.org/10.32604/cmc.2020.04604
Journal Article Type | Article |
---|---|
Acceptance Date | Dec 3, 2018 |
Publication Date | 2020-02 |
Deposit Date | Dec 18, 2018 |
Publicly Available Date | Mar 5, 2020 |
Print ISSN | 1546-2218 |
Electronic ISSN | 1546-2226 |
Publisher | Tech Science Press |
Peer Reviewed | Peer Reviewed |
Volume | 62 |
Issue | 2 |
Pages | 713-729 |
DOI | https://doi.org/10.32604/cmc.2020.04604 |
Keywords | Distributed computing, speculative execution, resource scheduling, noncooperative game theory. |
Public URL | http://researchrepository.napier.ac.uk/Output/1447642 |
Files
An Optimized Resource Scheduling Strategy For Hadoop Speculative Execution Based On Non-cooperative Game Schemes (Publisher PDF)
(611 Kb)
PDF
Publisher Licence URL
http://creativecommons.org/licenses/by/4.0/
Copyright Statement
Published under a Creative Commons Attribution (CC BY) license.
You might also like
PFL-LDG: Privacy-preserving Federated Learning via Lightweight Device Grouping
(2023)
Conference Proceeding
Near-Data Prediction Based Speculative Optimization in a Distribution Environment
(2022)
Journal Article
Intelligent Question Answering System Based on Knowledge Graph
(2022)
Conference Proceeding
Improving Domestic NILM Using An Attention- Enabled Seq2Point Learning Approach
(2022)
Conference Proceeding
An Intelligent Method for Upper Limb Posture Recognition Based on Limited MEMS Data
(2022)
Conference Proceeding
Downloadable Citations
About Edinburgh Napier Research Repository
Administrator e-mail: repository@napier.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search