State of Practice: LLMs in Software Engineering and Software Architecture

Jahic, Jasmin; Sami, Ashkan

Abstract

Large Language Models (LLMs) are finding their way into Software Engineering by assisting with tasks such as code generation. Furthermore, LLMs might have a potential to perform even more complex tasks, such as suggesting architectural design. However, there is a lack of empirical surveys on how software engineering companies use (and plan to use) LLMs and if LLMs truly can provide benefits to software architects. To understand the state of practice considering adoption of LLMs in software engineering, existing challenges, and future trends, we have surveyed 15 different software engineering companies. To understand the ability of LLMs to perform more complex tasks, we report on our experiments with LLM-assisted architectural design. We applied ChatGPT on 5 software projects and in total performed 50 different experiments. Our results capture the state of the practice of LLMs in software engineering and demonstrate how LLMs perform when assisting with (more complex task such as) architectural design. Engineers, architects, and project managers should profit from these results to guide their decision towards targeted adoption of LLMs in their business and engineering domains.

Citation

Jahic, J., & Sami, A. (2024, June). State of Practice: LLMs in Software Engineering and Software Architecture. Presented at 21st IEEE International Conference on Software Architecture (ICSA 2024): 3rd International Workshop on Software Architecture and Machine Learning, Hyderabad, India

• Presentation Conference Type: Conference Paper (published)
• Conference Name: 21st IEEE International Conference on Software Architecture (ICSA 2024): 3rd International Workshop on Software Architecture and Machine Learning
• Start Date: Jun 4, 2024
• End Date: Jun 8, 2024
• Online Publication Date: Aug 21, 2024
• Publication Date: 2024
• Deposit Date: Jul 13, 2024
• Publicly Available Date: Aug 21, 2024
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• Pages: 311-318
• Series ISSN: 2768-4288
• Book Title: 2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C)
• DOI: https://doi.org/10.1109/ICSA-C63560.2024.00059
• Keywords: Architecture; AI; Design Space Exploration; ChatGPT
• Publisher URL: https://conf.researchr.org/details/icsa-2024/icsa-2024-workshops/19/State-of-Practice-LLMs-in-Software-Engineering-and-Software-Architecture