Qi Liu
A speculative approach to spatial-temporal efficiency with multi-objective optimization in a heterogeneous cloud environment
Liu, Qi; Cai, Weidong; Shen, Jian; Fu, Zhangjie; Liu, Xiaodong; Linge, Nigel
Authors
Abstract
A heterogeneous cloud system, for example, a Hadoop 2.6.0 platform, provides distributed but cohesive services with rich features on large-scale management, reliability, and error tolerance. As big data processing is concerned, newly built cloud clusters meet the challenges of performance optimization focusing on faster task execution and more efficient usage of computing resources. Presently proposed approaches concentrate on temporal improvement, that is, shortening MapReduce time, but seldom focus on storage occupation; however, unbalanced cloud storage strategies could exhaust those nodes with heavy MapReduce cycles and further challenge the security and stability of the entire cluster. In this paper, an adaptive method is presented aiming at spatial–temporal efficiency in a heterogeneous cloud environment. A prediction model based on an optimized Kernel-based Extreme Learning Machine algorithm is proposed for faster forecast of job execution duration and space occupation, which consequently facilitates the process of task scheduling through a multi-objective algorithm called time and space optimized NSGA-II (TS-NSGA-II). Experiment results have shown that compared with the original load-balancing scheme, our approach can save approximate 47–55 s averagely on each task execution. Simultaneously, 1.254‰ of differences on hard disk occupation were made among all scheduled reducers, which achieves 26.6% improvement over the original scheme.
Journal Article Type | Article |
---|---|
Acceptance Date | Jun 12, 2016 |
Online Publication Date | Aug 14, 2016 |
Publication Date | Nov 21, 2016 |
Deposit Date | Jan 30, 2017 |
Publicly Available Date | Nov 14, 2017 |
Journal | Security and Communication Networks |
Print ISSN | 1939-0114 |
Publisher | Wiley |
Peer Reviewed | Peer Reviewed |
Volume | 9 |
Issue | 17 |
Pages | 4002-4012 |
DOI | https://doi.org/10.1002/sec.1582 |
Keywords | MapReduce; cloud storage; load balancing; multi-objective optimization; prediction model |
Public URL | http://researchrepository.napier.ac.uk/Output/451187 |
Contract Date | Nov 14, 2017 |
Files
A Speculative Approach to Spatial-Temporal Efficiency with Multi-Objective Optimisation in a Heterogeneous Cloud Environment
(291 Kb)
PDF
You might also like
An adaptive approach to better load balancing in a consumer-centric cloud environment
(2016)
Journal Article
A Survey of Speculative Execution Strategy in MapReduce
(2016)
Presentation / Conference Contribution
An Introduction of Non-intrusive Load Monitoring and Its Challenges in System Framework
(2016)
Presentation / Conference Contribution
A Method for Electric Load Data Verfication and Repair in home Environment
(2016)
Presentation / Conference Contribution
Downloadable Citations
About Edinburgh Napier Research Repository
Administrator e-mail: repository@napier.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search