Tackling the useless mutants problem

Abstract

Mutation testing is a fault-based testing criterion to assess and improve the quality of a test suite. Despite attracting much interest, the costs of using mutation testing are usually high, hindering its use in industry. Useless mutants (e.g., equivalent and duplicate) contribute to increase costs. The equivalent mutant problem has already been proven undecidable, and manually detecting equivalent mutants is an error-prone and timeconsuming task. The duplicate mutant, although eventually killed by some test, requires an unnecessary computational cost. This way, solutions, even partial, can help reducing these costs. In this work, we tackle the useless mutants problem from two perspectives. First, we propose improving the transformation rules embedded in the mutation operators to avoid useless mutants. We present i-rule, a common language to avoid the generation of equivalent (e-rule) and duplicate (d-rule) mutants. We also present a strategy to help mutation tool developers find out occurrence patterns that lead to new i-rules. We instantiate the strategy with 100 Java programs, which led us to find out 99 i-rules for three common mutation testing tools (MuJava, Major, and Pit). To evaluate the effectiveness of the i-rules on reducing costs, we implement 32 of them in the MuJava tool and execute with classes of well-known projects. The results show we reduced the number of mutants by almost 20% on average and saved time to generate the mutants, thus demonstrating the potential of our approach for reducing mutation costs. Second, we present an approach to suggest equivalent mutants by using automated behavioral testing. We perform static analysis to automatically generate tests directed for the entities impacted by the mutation. For each analyzed mutant, our approach can suggest the mutant as equivalent or non-equivalent. In the case of non-equivalent mutants, our approach provides the test cases capable of killing them. For the equivalent mutants suggested, we also provide a ranking of mutants with a strong or weak chance of the mutant being indeed equivalent. We implement our approach in a tool called Nimrod. To evaluate Nimrod, we execute it against a set of 1,542 mutants from eight open-source projects. The results indicate that the Nimrod is very effective in suggesting equivalent mutants. It reached more than 96% of accuracy in five out of eight studied subjects. Compared with manual analysis of the surviving mutants, Nimrod takes a third of the time to suggest equivalent and is 25 times faster to indicate non-equivalent.