Root causing, detecting, and fixing flaky tests: State of the art and future roadmap
Department
Software Engineering and Game Development
Document Type
Article
Publication Date
5-1-2021
Abstract
A flaky test is a test that may lead to different results in different runs on a single code under test without any change in the test code. Test flakiness is a noxious phenomenon that slows down software deployment, and increases the expenditures in a broad spectrum of platforms such as software-defined networks and Internet of Things environments. Industrial institutes and labs have conducted a whole lot of research projects aiming at tackling this problem. Although this issue has been receiving more attention from academia in recent years, the academic research community is still behind the industry in this area. A systematic review and trend analysis on the existing approaches for detecting and root causing flaky tests can pave the way for future research on this topic. This can help academia keep pace with industrial advancements and even lead the research in this field. This article first presents a comprehensive review of recent achievements of the industry as well as academia regarding the detection and mitigation of flaky tests. In the next step, recent trends in this line of research are analyzed and a roadmap is established for future research.
Journal Title
Software - Practice and Experience
Journal ISSN
00380644
Volume
51
Issue
5
First Page
851
Last Page
867
Digital Object Identifier (DOI)
10.1002/spe.2929