Detecting and Measuring Flakiness in Test Suites
Software Engineering Seminar
Computer Science Department
Monday, September 10, 11am, ENGR 4201
George Mason University
Although ideally our test cases control all possible inputs that are used by the system under test, in practice, there are a variety of nondeterministic inputs to the SUT that are not controlled by tests. These uncontrolled inputs can lead to ``flaky'' tests --- tests whose outcome is non-deterministic. Unfortunately, flaky tests have become a fact of life for many software engineers: due to inherent non-determinism in the code being tested, effective tests (especially system/integration tests) are inherently flaky. Given that many tests may be flaky, we can no longer rely on a simple pass/fail result of our test suite: did our tests pass randomly? Did they fail randomly? To cope with this flakiness, testing tools need to be able to detect which tests are likely to become flaky, and collect sufficient metrics that each test outcome can be categorized instead as "likely pass," "likely fail," or "unknown" (flaky). In this talk, I will describe several of my recent and ongoing efforts to automatically detect flaky tests and to measure the impact of flakiness in software testing tools that rely on code coverage.
Jon is an Assistant Professor directing research in Software Engineering and Software Systems at George Mason University. His research makes it easier for developers to create reliable software by improving software testing. Jon’s recent work in accelerating software testing has been recognized with an ACM SIGSOFT Distinguished Paper Award (ICSE ’14 – Unit Test Virtualization with VMVM), and has been the basis for an industrial collaboration with Electric Cloud. His program analysis work has led to several widely adopted runtime systems for the JVM, including the Phosphor taint tracking system (OOPSLA ’14) and the CROCHET checkpoint/rollback tool (ECOOP ’18). Jon is committed to releasing software artifacts that accompany his research papers. His research interests bring him to publish at venues such as ICSE, FSE, ISSTA, OOPSLA, OSDI and EuroSys. Jon serves on a variety of program committees, and has been involved in the logistical organization of many recent OOPSLAs, most recently as publicity chair. He also co-organized the PL/SE mentoring workshop at SPLASH in 2017 and 2018. His other interests include photography and cycling.