Skip to main content

Research Repository

Advanced Search

An Optimized Resource Scheduling Strategy for Hadoop Speculative Execution Based on Non-cooperative Game Schemes

Jiang, Yinghang; Liu, Qi; Dannah, Williams; Jin, Dandan; Liu, Xiaodong; Sun, Mingxu

Authors

Yinghang Jiang

Qi Liu

Williams Dannah

Dandan Jin

Mingxu Sun



Abstract

Hadoop is a well-known parallel computing system for distributed computing and large-scale data processes. “Straggling” tasks, however, have a serious impact on task allocation and scheduling in a Hadoop system. Speculative Execution (SE) is an efficient method of processing “Straggling” Tasks by monitoring real-time running status of tasks and then selectively backing up “Stragglers” in another node to increase the chance to complete the entire mission early. Present speculative execution strategies meet challenges on misjudgement of “Straggling” tasks and improper selection of backup nodes, which leads to inefficient implementation of speculative executive processes. This paper has proposed an Optimized Resource Scheduling strategy for Speculative Execution (ORSE) by introducing non-cooperative game schemes. The ORSE transforms the resource scheduling of backup tasks into a multi-party non-cooperative game problem, where the tasks are regarded as game participants, whilst total task execution time of the entire cluster as the utility function. In that case, the most benefit strategy can be implemented in each computing node when the game reaches a Nash equilibrium point, i.e. the final resource scheduling scheme to be obtained. The strategy has been implemented in Hadoop-2.x. Experimental results depict that the ORSE can maintain the efficiency of speculative executive processes and improve fault-tolerant and computation performance under the circumstances of Normal Load, Busy Load and Busy Load with Skewed Data.

Citation

Jiang, Y., Liu, Q., Dannah, W., Jin, D., Liu, X., & Sun, M. (2020). An Optimized Resource Scheduling Strategy for Hadoop Speculative Execution Based on Non-cooperative Game Schemes. Computers, Materials & Continua, 62(2), 713-729. https://doi.org/10.32604/cmc.2020.04604

Journal Article Type Article
Acceptance Date Dec 3, 2018
Publication Date 2020-02
Deposit Date Dec 18, 2018
Publicly Available Date Mar 5, 2020
Print ISSN 1546-2218
Electronic ISSN 1546-2226
Publisher Tech Science Press
Peer Reviewed Peer Reviewed
Volume 62
Issue 2
Pages 713-729
DOI https://doi.org/10.32604/cmc.2020.04604
Keywords Distributed computing, speculative execution, resource scheduling, noncooperative game theory.
Public URL http://researchrepository.napier.ac.uk/Output/1447642

Files


An Optimized Resource Scheduling Strategy For Hadoop Speculative Execution Based On Non-cooperative Game Schemes (Publisher PDF) (611 Kb)
PDF

Publisher Licence URL
http://creativecommons.org/licenses/by/4.0/

Copyright Statement
Published under a Creative Commons Attribution (CC BY) license.







You might also like



Downloadable Citations