Swarmchestrate - Application-level Swarm-based Orchestration Across the Cloud-to-Things Continuum

Project Description

The next-generation of intelligent cyber-physical and IoT systems powered by artificial intelligence (AI) are becoming an essential part of our environment. With the ever-growing number of intelligent smart devices collecting huge amounts of data at the edges of the network, the need for processing this data in a heterogeneous computing environment, that spans across large geographic distances, various providers, connectivity types and network zones, is becoming essential. Smart cities, smart manufacturing, augmented reality, industry 4.0, and self-driving cars are just some examples of the wide range of domains, where the applicability of such systems is highly impactful. However, the efficient management of these complex distributed applications raises significant challenges, as both their deployment on heterogeneous computing resources and their run-time reconfiguration should take into consideration several aspects, such as resource requirements and constraints, potential volatility of resources, energy optimisation or dynamically changing security requirements. Additionally, providing trust in such distributed systems, where resources are spanning multiple administrative domains and various providers, is hard to achieve.
The efficient and effective management and processing of the large amounts of data in this next generation of distributed systems has to deal with versatile requirements. Some data need to be processed locally due to regulations, privacy issues and/or performance constraints, while some other may require access to long-term sophisticated computational cloud resources. Applications running on such systems have a wide range of requirements, including the execution of low-latency analytics closer to the data source, privacy-sensitivity, context-awareness, time and location awareness, as well as the need of simultaneous access to geographically distributed arrays of sensors, remote localised heterogeneous computational resources, and to a large-scale on-the-fly allocated multi-cloud infrastructure. Therefore, a new generation of orchestration tools and solutions is needed to handle this complexity efficiently, and to take into consideration this dynamically changing set of complex requirements in an intelligent fashion.
All currently available orchestration tools, responsible for deploying and managing data processing applications in the Cloud-to-Edge computing continuum, are based on a certain level of centralisation. Such centralisation, while relatively easy to implement, carries several disadvantages. The central component can become a single point of failure, can be easily overloaded as the system scales and also provides a good target for security attacks. Additionally, such a centralised approach does not fit well to the highly distributed and dynamically changing nature of the computing environment. A centralised management approach cannot react fast enough for some changes in local environments (e.g. volatility of resources) and cannot support fast adaptation of resources and application requirements (e.g. due to the movement of certain computing elements).
An alternative is a decentralised self-adaptive system that can be aware of its surroundings and can organise and reorganise itself without any central control and management. While the implementation of such a system is complex, recent advances in various fields of computing, including Swarm computing, distributed AI, distributed ledger systems and decentralised identity management, now enable the efficient realisation of such an approach. The overall aim of the Swarmchestrate project is to combine the above-mentioned emerging technologies and create a completely decentralised autonomous and self-organised application management system that supports the processing of large amounts of data generated at the network edges with the help of hyper-distributed and complex applications in the Cloud-to-Edge continuum. The approach applied by Swarmchestrate is fundamentally new to application orchestration and suitable to manage hyper-distributed applications that span across large distances and the different layers of the dynamic compute continuum.
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101135012 and UKRI Guarantee scheme