Mutation Testing

End of January I attended the OOP2015 conference in Munich. Among the load of interesting sessions was one that left a mark. It was the workshop conducted by Filip van Laenen and Markus Schirp about Mutation Testing (slides here), which I'd like to summarize in this post.

What is Mutation Testing?

Mutation testing is a method to ensure the quality of your Test. With mutations testing you verify if your tests not only invokes your product code pretending to cover lines and branches but that the tests actually reflects the semantics of your code. While your tests guard your product code from bugs, mutation testing guards your tests suite from critical gaps.

How does it work?

Mutation testing is available for a set of languages. The implementation for Java is the PIT. PIT modifies the compiled byte code by applying a set of predefined rules, so call Mutators. Each change to the byte code is called a Mutant. Against the altered byte code your unit tests are executed, resulting in three outcomes:

• The tests fails. This proves, that your tests detects the changes to the byte code correctly and reports an error. This outcome is called a Killed Mutant.
• The test does not fail, but the line of code is covered by the test execution. This proves, that your test does not cover the full semantics of the product. This outcome is called a Survived Mutant (I call them Survivors).
• The product code is not covered by a test at all but contains a mutant. This is outcome is an Uncovered Mutant (I call them Lurker), which can also be indicated by a Code Coverage tool such as Cobertura or jacoco.

What's the use of it?

Every mutant not killed by a test is a blind spot of your test suite and may become an actual bug in the long run. With mutation testing you can determine, whether your Unit Test - the detail specification - covers all the semantics of your product code. For every uncovered mutant you may either decide if your specification (test) is incomplete or your product code contains unneeded semantics which should be removed. That way mutation testing leads to smaller and simpler code and further is an indicator, when you are done with testing.

Its no silver bullet

Mutation testing is no silver bullet as it does not replace the need for properly designed tests in the first place. Testing for all combinations of Mutations may become a quite resource consuming operation, requiring to reduce the scope of mutation testing in larger projects on the crucial parts.
Further, some survived mutants may have to be accepted. Survived mutants may easily occur and may not necessarily be an indicator that your coverage is insufficient. For a simple example, think about logging code or the equality in some comparison cases, like ( a > b ? a : b) which in Java is equal to (a >=b ? a : b).

How to use it?

To use it on a maven project, an official maven plugin is avaible (see link for a good documentation how to use it). There are options for command line or Ant as well. This will create a report in html or xml. You may narrow the scope of the mutation testing using include and exclude parameter. The report itself run on a regular basis is already a good source of information for conduction code reviews.
For using it with SonarQube, a plugin is available, but its current version 0.5 is only compatible with SonarQube prior to version 4.2. There is a fork on github, making it compatible with Sonar API version 4.3 and I am working on a version for SonarQube 5.0 with some more rules. Both have to be compiled and deployed manually to Sonar.

TL;DR

Mutation Testing alters your product code in a deterministic way and verifies if your test suite finds the induced bug. This leads to smaller and simpler source code and you can tell when your're done with testing.