Data Temperature Informed Streaming for Optimising Large-Scale Multi-Tiered Storage

Davies-Tagg, Dominic; Anjum, Ashiq; Zahir, Ali; Liu, Lu; Yaseen, Muhammad Usman; Antonopoulos, Nick

Abstract

Data temperature is a response to the ever-growing amount of data. These data have to be stored, but they have been observed that only a small portion of the data are accessed more frequently at any one time. This leads to the concept of hot and cold data. Cold data can be migrated away from high-performance nodes to free up performance for higher priority data. Existing studies classify hot and cold data primarily on the basis of data age and usage frequency. We present this as a limitation in the current implementation of data temperature. This is due to the fact that age automatically assumes that all new data have priority and that usage is purely reactive. We propose new variables and conditions that influence smarter decision-making on what are hot or cold data and allow greater user control over data location and their movement. We identify new metadata variables and user-defined variables to extend the current data temperature value. We further establish rules and conditions for limiting unnecessary movement of the data, which helps to prevent wasted input output (I/O) costs. We also propose a hybrid algorithm that combines existing variables and new variables and conditions into a single data temperature. The proposed system provides higher accuracy, increases performance, and gives greater user control for optimal positioning of data within multi-tiered storage solutions.

Citation

Davies-Tagg, D., Anjum, A., Zahir, A., Liu, L., Yaseen, M. U., & Antonopoulos, N. (2024). Data Temperature Informed Streaming for Optimising Large-Scale Multi-Tiered Storage. Big Data Mining and Analytics, 7(2), 371-398. https://doi.org/10.26599/bdma.2023.9020039

• Journal Article Type: Article
• Online Publication Date: Apr 22, 2024
• Publication Date: 2024-06
• Deposit Date: Aug 8, 2024
• Publicly Available Date: Aug 8, 2024
• Journal: Big Data Mining and Analytics
• Print ISSN: 2096-0654
• Electronic ISSN: 2097-406X
• Publisher: Institute of Electrical and Electronics Engineers (IEEE)
• Peer Reviewed: Peer Reviewed
• Volume: 7
• Issue: 2
• Pages: 371-398
• DOI: https://doi.org/10.26599/bdma.2023.9020039