Scribble 0.3.0

I am proud to announce a new version of the the Scribble testing library! The biggest changes are the new modularization and documentation. For every functional aspect there is now a separate module so that not a whole load of unused dependencies have to be included in your project if you only require just a single functional aspect. In addition to this, the entire project documentation is now kept in the source and be generated using maven's site support. This includes this wiki documentation as well, although the publishing process is not yet part of the release build jobs.
As new features for testing I introduce a http server as a TestRule that can be set up in various ways to server static content. It's still rather limited, but will be contiuously improved in future releases. Further features are the possibility to create temporary zip files, record system out and err via a TestRule and capture and restore System Properties - a simple rule that helps keeping the test environment clean, and finally a matcher for matching date strings against a data format.

For more information, have a look at the wiki or find the source code on GitHub. 

Task

• [SCRIB-55] - Modularize Scribble

Story

• [SCRIB-35] - Embedd static HTTP content as a rule
• [SCRIB-43] - Build documentation as part of the release
• [SCRIB-49] - Create zipped temp file from resources
• [SCRIB-50] - Date Format Matcher
• [SCRIB-52] - Rule for capturing System.out and System.err
• [SCRIB-53] - Rule for setting and restoring System Properties

Bug

• [SCRIB-39] - ConfigPropertyInjection#isMatching sets default value
• [SCRIB-51] - TemporaryFile not usable as ClassRule
• [SCRIB-57] - ApacheDS all prevents exclusion of modules
• [SCRIB-58] - Remove SLF4J Binding dependencies
• [SCRIB-59] - DirectoryServer/DirectoryService not working as ClassRule

Scribble Release 0.2.0

I am proud to announce a new version of the the Scribble testing library! The new version has support for an embedded ldap server which allows to write tests against an ldap server without having to rely on existing infrastructure. Further, the JCR support has been improved, now it's possible to pre-initialize a JCR repository with content from a descriptor file and to create a security-enabled in-memory repository. Some additional improvements have been made in the CDI injection support and the matchers have been extended for availability checks for URLs.

For more information, have a look at the wiki or find the source code on GitHub.

Release Notes - Scribble - Version 0.2.0

Bug

• [SCRIB-31] - Primitive types not support for ConfigProperty injection
• [SCRIB-32] - String to Number conversion of default values in ConfigProperty injection fails
• [SCRIB-41] - LDAP Rules are not properly applied
• [SCRIB-42] - ResourceAvailabilityMatcher is not compatible with URL
• [SCRIB-48] - Directory Rules can not be used as ClassRules

Story

• [SCRIB-1] - Builder support for LDAP Server and Service
• [SCRIB-2] - Make LDAP Port configurable
• [SCRIB-5] - Matchers for availability of an URL
• [SCRIB-10] - Support for prepared JCR Content
• [SCRIB-12] - Support security enabled content repositories
• [SCRIB-14] - Add Convenience method for admin login
• [SCRIB-33] - Convenience Methods for Directory creation
• [SCRIB-34] - Convenience Method for anonymous login
• [SCRIB-38] - Supply package-info.java

Mutation Testing

End of January I attended the OOP2015 conference in Munich. Among the load of interesting sessions was one that left a mark. It was the workshop conducted by Filip van Laenen and Markus Schirp about Mutation Testing (slides here), which I'd like to summarize in this post.

What is Mutation Testing?

Mutation testing is a method to ensure the quality of your Test. With mutations testing you verify if your tests not only invokes your product code pretending to cover lines and branches but that the tests actually reflects the semantics of your code. While your tests guard your product code from bugs, mutation testing guards your tests suite from critical gaps.

How does it work?

Mutation testing is available for a set of languages. The implementation for Java is the PIT. PIT modifies the compiled byte code by applying a set of predefined rules, so call Mutators. Each change to the byte code is called a Mutant. Against the altered byte code your unit tests are executed, resulting in three outcomes:

• The tests fails. This proves, that your tests detects the changes to the byte code correctly and reports an error. This outcome is called a Killed Mutant.
• The test does not fail, but the line of code is covered by the test execution. This proves, that your test does not cover the full semantics of the product. This outcome is called a Survived Mutant (I call them Survivors).
• The product code is not covered by a test at all but contains a mutant. This is outcome is an Uncovered Mutant (I call them Lurker), which can also be indicated by a Code Coverage tool such as Cobertura or jacoco.

What's the use of it?

Every mutant not killed by a test is a blind spot of your test suite and may become an actual bug in the long run. With mutation testing you can determine, whether your Unit Test - the detail specification - covers all the semantics of your product code. For every uncovered mutant you may either decide if your specification (test) is incomplete or your product code contains unneeded semantics which should be removed. That way mutation testing leads to smaller and simpler code and further is an indicator, when you are done with testing.

Its no silver bullet

Mutation testing is no silver bullet as it does not replace the need for properly designed tests in the first place. Testing for all combinations of Mutations may become a quite resource consuming operation, requiring to reduce the scope of mutation testing in larger projects on the crucial parts.
Further, some survived mutants may have to be accepted. Survived mutants may easily occur and may not necessarily be an indicator that your coverage is insufficient. For a simple example, think about logging code or the equality in some comparison cases, like ( a > b ? a : b) which in Java is equal to (a >=b ? a : b).

How to use it?

To use it on a maven project, an official maven plugin is avaible (see link for a good documentation how to use it). There are options for command line or Ant as well. This will create a report in html or xml. You may narrow the scope of the mutation testing using include and exclude parameter. The report itself run on a regular basis is already a good source of information for conduction code reviews.
For using it with SonarQube, a plugin is available, but its current version 0.5 is only compatible with SonarQube prior to version 4.2. There is a fork on github, making it compatible with Sonar API version 4.3 and I am working on a version for SonarQube 5.0 with some more rules. Both have to be compiled and deployed manually to Sonar.

TL;DR

Mutation Testing alters your product code in a deterministic way and verifies if your test suite finds the induced bug. This leads to smaller and simpler source code and you can tell when your're done with testing.