Q-CSM: Q-Learning-based Cognitive Service Management in Heterogeneous IoT Networks

Duran, Kubra; Ozdem, Mehmet; Gursu, Kerem; Canberk, Berk

Abstract

The dramatic increase in the number of smart services and their diversity poses a significant challenge in Internet of Things (IoT) networks: heterogeneity. This causes significant quality of service (QoS) degradation in IoT networks. In addition, the constraints of IoT devices in terms of computational capability and energy resources add extra complexity to this. However, the current studies remain insufficient to solve this problem due to the lack of cognitive action recommendations. Therefore, we propose a Q-learning-based Cognitive Service Management framework called Q-CSM. In this framework, we first design an IoT Agent Manager to handle the heterogeneity in data formats. After that, we design a Q-learning-based recommendation engine to optimize the devices’ lifetime according to the predicted QoS behaviour of the changing IoT network scenarios. We apply the proposed cognitive management to a smart city scenario consisting of three specific services: wind turbines, solar panels, and transportation systems. We note that our proposed cognitive method achieves 38.7% faster response time to the dynamical IoT changes in topology. Furthermore, the proposed framework achieves 19.8% longer lifetime on average for constrained IoT devices thanks to its Q-learningbased cognitive decision capability. In addition, we explore the most successive learning rate value in the Q-learning run through the exploration and exploitation phases.

Citation

Duran, K., Ozdem, M., Gursu, K., & Canberk, B. (2024, November). Q-CSM: Q-Learning-based Cognitive Service Management in Heterogeneous IoT Networks. Presented at 2024 IEEE 10th World Forum on Internet of Things (WFIoT2024), Ottawa, Canada

• Presentation Conference Type: Conference Paper (published)
• Conference Name: 2024 IEEE 10th World Forum on Internet of Things (WFIoT2024)
• Start Date: Nov 10, 2024
• End Date: Nov 13, 2024
• Acceptance Date: Aug 31, 2024
• Online Publication Date: Dec 30, 2024
• Publication Date: 2024
• Deposit Date: Oct 11, 2024
• Publicly Available Date: Dec 30, 2024
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• Pages: 713-718
• Series ISSN: 2768-1734
• Book Title: 2024 IEEE 10th World Forum on Internet of Things (WF-IoT)
• ISBN: 9798350373028
• DOI: https://doi.org/10.1109/WF-IoT62078.2024.10811013
• Keywords: internet of things, heterogeneity, quality of service, reinforcement learning, cognitive management
• Public URL: http://researchrepository.napier.ac.uk/Output/3885918
• External URL: https://wfiot2024.iot.ieee.org/