*Detecting and Measuring Flakiness in Test Suites*

Software Engineering Seminar
Computer Science Department

Monday, September 10, 11am, ENGR 4201

*Jonathan Bell*

George Mason University


Although ideally our test cases control all possible inputs that are used
by the system under test, in practice, there are a variety of
nondeterministic inputs to the SUT that are not controlled by tests. These
uncontrolled inputs can lead to ``flaky'' tests --- tests whose outcome is
non-deterministic. Unfortunately, flaky tests have become a fact of life
for many software engineers: due to inherent non-determinism in the code
being tested, effective tests (especially system/integration tests) are
inherently flaky. Given that many tests may be flaky, we can no longer rely
on a simple pass/fail result of our test suite: did our tests pass
randomly? Did they fail randomly? To cope with this flakiness, testing
tools need to be able to detect which tests are likely to become flaky, and
collect sufficient metrics that each test outcome can be categorized
instead as "likely pass," "likely fail," or "unknown" (flaky). In this
talk, I will describe several of my recent and ongoing efforts to
automatically detect flaky tests and to measure the impact of flakiness in
software testing tools that rely on code coverage.


Jon is an Assistant Professor directing research in Software Engineering
and Software Systems at George Mason University. His research makes it
easier for developers to create reliable software by improving software
testing. Jon’s recent work in accelerating software testing has been
recognized with an ACM SIGSOFT Distinguished Paper Award (ICSE ’14 – Unit
Test Virtualization with VMVM), and has been the basis for an industrial
collaboration with Electric Cloud. His program analysis work has led to
several widely adopted runtime systems for the JVM, including the Phosphor
taint tracking system (OOPSLA ’14) and the CROCHET checkpoint/rollback tool
(ECOOP ’18). Jon is committed to releasing software artifacts that
accompany his research papers. His research interests bring him to publish
at venues such as ICSE, FSE, ISSTA, OOPSLA, OSDI and EuroSys. Jon serves on
a variety of program committees, and has been involved in the logistical
organization of many recent OOPSLAs, most recently as publicity chair. He
also co-organized the PL/SE mentoring workshop at SPLASH in 2017 and 2018.
His other interests include photography and cycling.


[log in to unmask]  |