Data-Limited Continuous Experimentation (dlCE): A Literature Review

Chren, Stanislav; Fagerholm, Fabian; Bjarnason, Elizabeth; Linåker, Johan; Rafi, Saima; Lehtelä, Bettina; Runeson, Per; Kauppinen, Marjo

Abstract

Continuous experimentation (CE) is a software development approach where product decisions are data-driven. Large global internet-facing companies such as Microsoft, Google, and Facebook apply the practice by leveraging their massive user bases to obtain high statistical significance in experimentation results. However, companies with smaller user bases, such as small- and medium-sized enterprises (SMEs) and early-stage software startups, struggle to adopt CE due to limitations in user data. Our goal is to increase understanding of situations in CE where data limitations occur, as observed in the research literature. We investigate data limitations and challenges related to them, including their characteristics, and solutions and practices that may address these. We conducted a rapid review of CE papers from a previous systematic literature review, and analysed these to identify scenarios that exhibit data limitations in CE. We present a framework that illustrates different dimensions of data-limited CE (DLC) and connects scenarios to challenges and their potential future solutions. Most challenges and potential solutions that we found are related to the amount of data. We also note examples of other limitations related to, e.g., evaluation metrics, that provide interesting avenues for further research.

Citation

Chren, S., Fagerholm, F., Bjarnason, E., Linåker, J., Rafi, S., Lehtelä, B., Runeson, P., & Kauppinen, M. (2024, November). Data-Limited Continuous Experimentation (dlCE): A Literature Review. Presented at ICSOB 2024, Utrecht, The Netherlands

• Presentation Conference Type: Conference Paper (published)
• Conference Name: ICSOB 2024
• Start Date: Nov 18, 2024
• End Date: Nov 20, 2024
• Online Publication Date: Mar 23, 2025
• Publication Date: 2025
• Deposit Date: Jun 23, 2025
• Publisher: Springer
• Peer Reviewed: Peer Reviewed
• Pages: 324-339
• Series Title: Lecture Notes in Business Information Processing
• Series Number: 539
• Book Title: Software Business: 15th International Conference, ICSOB 2024, Utrecht, The Netherlands, November 18–20, 2024, Proceedings
• ISBN: 978-3-031-85848-2
• DOI: https://doi.org/10.1007/978-3-031-85849-9_26