How to detect fake tests - Introduction to Mutation Testing

In the last posts (1, 2, 3) I showed various ways for producing fake tests. Of course, good developers won't fake their tests, and the chances to encounter a test suite purely made of fake tests in real life is rather low. Nevertheless, in certain environments it may occasionally happen that metrics are polished for various reasons. But it's more likely, that the quality of a test suites deteriorates over time because of various reasons, i.e. project pressure, sloppy moments during coding, wrong assumptions, etc. And typically we rely on metrics to determine whether our project is in good shape.

My intention for the last three posts was to show, how easy the common metrics - test count, line and condition coverage - can be tricked and are of very low value without the proper context. They are as good for determining the health of a software project as lines of codes are. They might be an weak indicator but nothing more.

The main question is, how could we determine the actual value of our tests and test suites? How would others do it? Firebrigades test their procedures and techniques on a real fire. Military is holding maneuvers, martial arts fighters test their skills in championships, NetFlix is letting the Chaos Monkey terminate instances to detect holes in the recovery procedures.

What is the main reason to have automated tests? To detects bugs that slipped into existing code unintentionally. It doesn't matter if you wrote the tests beforehand by practicing Uncle Bob style TDD or afterwards to create a safepoint for your code. The base assumption is, once you've written your code and your tests, it's free of errors. But it's called Software for a reason: it may change over time. The once written, error-free code will eventually be changed. To ensure, it is still functional, the test suites are run and if it's all green, nothing was broken. But how can you be sure of that?

The only thing to verify your test suite is capable of detecting bugs is to induce bugs in your code.

The technique of altering your code and re-run your test suite to verify the test suite detects the code change is called Mutation Testing. The concept is known for quite a while and was mostly subject to academic research with the tools being somewhat theoretical and less practical to use. But since the arrival of Pitest.org a practical, stable and well integrated tool has been around that should be in every developer's toolbox.

Pitest mutates bytecode and runs highly parallel making it the fastest mutation testing tool for the JVM. Pitest offers a set of Mutation Operators that modify bytecode according to a defined ruleset and thus creates a modified version of the code, a Mutation. The test suite is run again and if at least one test fails, the Mutation is killed. In the end, the Mutation Score is calculated from the number of killed mutations vs the total number of mutations.

Different to line or branch coverage, which can be determined with a single test suite execution, Pitest requires one test suite execution per mutation. With larger code-bases the execution time increases exponentially due to the sheer number of combinations of mutations. Although Pitest offers a variety of settings and options to limit execution time - i.e. delta execution, selection of mutation operators, exclusion of classes, to name a few - it requires some thorough planning how this technique should be incorporated into the CI/CD pipeline. The value it delivers, comes with a price.

In the next post of this series, I will provide examples of how to setup and run pitest with practical examples, so stay tuned.

How to fake tests (Part 3)

In this 3rd part of the series I want to show how assertions can be faked, so that not only lines and branches get covered but the test themselves also contain some assertions.
Faking Assertions only makes sense if a metric such as "assertions/test" is computed at all. Otherwise you may skip that part, because every proper code review would reveal your test as fake.
Test libraries such as Junit or TestNG contain various means for expressing assertions. In addition to this, some frameworks exist for that sole purpose, i.e. Hamcrest, Truth, to name a few. Basic approach for all is, to invoke the system under test (generating coverage information) and to verify outcomes against assertions.
But the outcomes doesn’t have to be related to what is declared as expected for the test to succeed. So all of the following assertions might do the trick

• assertTrue(true);
• assertNotNull(new Object()); (a real life example I’ve encountered during a code review)
• assertEquals("2","2");
• …

After having applied fake assertions and fake coverage, our testsuite satisfies the following criteria

• Big, lots of tests for all the methods
• 100% Line Coverage
• 100% Condition Coverage
• Tests contain assertions (maybe 1 assertion/method as a metric)

This would make every project manager happy, because the quality of the product is so good and there is proof for that....

Not!

 You’ve probably produced the most sophisticated test suite with best quality ratings with minimum effort to create that has no value at all (Achievement unlocked).

In the next post I'll show how all these fakes described in this and the earlier posts can be revealed as such -  and more important, how the effectiveness of a test suite can be determined and gaps in a sensible test suite be found. So stay tuned.

How to fake tests (Part 2)

In the last post I described how to write fake tests to statisfy number-of-tests KPI. Apparently this is not a good practice for software craftsmen. Unfortunately some organisation do value KPIs more than good craftsmenship and may be simply tricked by fake tests. So in today's post I'd like to show you how to fake line and condition coverage of tests. This is a call to action for decision makers who base their decisions on such numbers: don't trust them. And for developers: if encouter things like the following (or like in the last post): fix them. So let's start with line coverage.

Faking Line Coverage

Line Coverage is a metric that measures how many and which lines have been covered during execution. There are various tools to measure coverage.

• Jacoco – Measuring on ByteCode level which has the advantage that you can test your actual artifacts, but bytecode can be different to its source at times.
• ECobertura, Clover – Measuring on SourceCode level which is more precise than byte-code measuring but injects additional code before compilation, ending up in different artifacts than you want to deliver.

When running your tests with line coverage enabled, all lines touched are recorded to produce the metric. If you have 0% line coverage, you didn’t run any code. So let’s extend our test to get some coverage:

@Test
public void test() {
  subject.invokeSomeMethod();
}

Obviously this test is broken because in cannot fail – unless the code itself produces an exception. But with tests like these you may achieve quite easily a high line coverage and a stable test suite.
But typical programs are rarely linear and have some sort of loop or branch constructs. So it’s unlikely you achieve 100% line coverage. So we have to fake branch coverage, too.

 

Faking Condition Coverage

Lets assume our simple program consists of the following code

Object compute(Object input) {
  if("left".equals(input) {
    return "right";
  } 
  return "left";
}

It has one condition with two branches. With a single test, you may get 66% Line Coverage and 50% Condition Coverage. I’ve experiences several times that branch coverage is perceived as “better” or of “more value” because it’s harder to achieve. If “harder” means “more code” it’s certainly true, but branch coverage suffers the same basic problem as line coverage does: it’s just a measure for which code is executed and not how good your tests are. It also depends on the code base, what is harder to achieve. If the happy-flow you test covers only a minor part of the code, you may have 50% branch-coverage but only 10% line coverage. Given the above example, assume the “left”-branch contains 10 lines of code, but you only test for the “right”-branch.

But as we are developers who want to make happy managers, let’s fake branch coverage!
Given, we only test a single flow in a single test, we need two tests:

@Test
public void testLeft() {
 String output = compute("left");
}
@Test
public void testRight() {
 String output = compute("right");
}

This test will produce 100% branch- and line coverage and is very unlikely to fail, ever.
But again: it’s worthless, because we don’t check any output of the operation. So the operation may return anything without failing the test. But still in terms of KPI metrics we achieved:

• 2 tests for 1 method (great ratio!)
• 100% line coverage
• 100% condition coverage

What we missed to have is an assertion. Assertion postulate expected outcomes of an operation. If the actual outcome is different than expected, the test fails. Theoretically it would would be possible to count assertions per test in static code analysis. But I’ve never seen such metric although it’s value would be similar to line- or condition coverage. Nevertheless: we can fake it!

So in the next post, I'll show you how to fake assertions.

How to fake tests (Part 1)

In most projects, metrics play an important role to determine the status, health, quality etc. of the project. Not rarely the common metrics for quality have been

• Number of Unit Tests (Total, Failed, Successful)
• Line Coverage
• Branch Coverage

Usually those “KPI” (Key Performance Indicators) were used by “managers” to steer the project to success. The problem with these metrics is: they are totally useless if taken out of context - and the context is usually not that well defined in terms of metrics, but often requires knowledge and insight into the system that’s been measured.

This post is about to show how to game the system and life-hack those KPIs to fake good quality. It’s NOT a best practice but a heads up to those who make decision based on those metrics to look behind the values.

Faking Number of Unit Tests

Most (if not all?) frameworks count the number of tests executed, which failed and which succeeded. A high number of tests is usually perceived as a good indicator of quality. The increase of the amount of tests should correlate with the increase in lines of code (another false-friend KPI). But what is counted as a test?

Let’s look at the Junit which is the de-facto standard for developing and executing Java based unit tests, but other frameworks such as TestNG follow similar concepts.

In Junit 3 it was every parameterless public void method starting with “test” in a class extending TestCase. Since Junit 4 every method annotated with @Test counts as a Test.

That’s it. Just a name convention or an Annotation and you have your test, so let’s fake it!

@Test
public void test() {

}

This is pure gold: a stable and ever succeeding Unit Test!

Copy and paste or even generate those and you produce a test suite satisfying the criteria:

• Big, tons of tests, probable even more than you have LoCs
• Stable, none of these tests is failing. Ever.
• Fast, you have feedback about the success within seconds.

The downside: it’s worthless (surprise, surprise!). There are basically two primary reasons, why its worthless:

• It doesn’t run any code
• It doesn’t pose any assertion about the outcome

Good indicators to check the first one are line or condition coverage analysis. The latter is more difficult to check.

In the upcoming posts we'll have a look into both.

Scribble 0.3.0

I am proud to announce a new version of the the Scribble testing library! The biggest changes are the new modularization and documentation. For every functional aspect there is now a separate module so that not a whole load of unused dependencies have to be included in your project if you only require just a single functional aspect. In addition to this, the entire project documentation is now kept in the source and be generated using maven's site support. This includes this wiki documentation as well, although the publishing process is not yet part of the release build jobs.
As new features for testing I introduce a http server as a TestRule that can be set up in various ways to server static content. It's still rather limited, but will be contiuously improved in future releases. Further features are the possibility to create temporary zip files, record system out and err via a TestRule and capture and restore System Properties - a simple rule that helps keeping the test environment clean, and finally a matcher for matching date strings against a data format.

For more information, have a look at the wiki or find the source code on GitHub. 

Task

• [SCRIB-55] - Modularize Scribble

Story

• [SCRIB-35] - Embedd static HTTP content as a rule
• [SCRIB-43] - Build documentation as part of the release
• [SCRIB-49] - Create zipped temp file from resources
• [SCRIB-50] - Date Format Matcher
• [SCRIB-52] - Rule for capturing System.out and System.err
• [SCRIB-53] - Rule for setting and restoring System Properties

Bug

• [SCRIB-39] - ConfigPropertyInjection#isMatching sets default value
• [SCRIB-51] - TemporaryFile not usable as ClassRule
• [SCRIB-57] - ApacheDS all prevents exclusion of modules
• [SCRIB-58] - Remove SLF4J Binding dependencies
• [SCRIB-59] - DirectoryServer/DirectoryService not working as ClassRule

Scribble Release 0.2.0

I am proud to announce a new version of the the Scribble testing library! The new version has support for an embedded ldap server which allows to write tests against an ldap server without having to rely on existing infrastructure. Further, the JCR support has been improved, now it's possible to pre-initialize a JCR repository with content from a descriptor file and to create a security-enabled in-memory repository. Some additional improvements have been made in the CDI injection support and the matchers have been extended for availability checks for URLs.

For more information, have a look at the wiki or find the source code on GitHub.

Release Notes - Scribble - Version 0.2.0

Bug

• [SCRIB-31] - Primitive types not support for ConfigProperty injection
• [SCRIB-32] - String to Number conversion of default values in ConfigProperty injection fails
• [SCRIB-41] - LDAP Rules are not properly applied
• [SCRIB-42] - ResourceAvailabilityMatcher is not compatible with URL
• [SCRIB-48] - Directory Rules can not be used as ClassRules

Story

• [SCRIB-1] - Builder support for LDAP Server and Service
• [SCRIB-2] - Make LDAP Port configurable
• [SCRIB-5] - Matchers for availability of an URL
• [SCRIB-10] - Support for prepared JCR Content
• [SCRIB-12] - Support security enabled content repositories
• [SCRIB-14] - Add Convenience method for admin login
• [SCRIB-33] - Convenience Methods for Directory creation
• [SCRIB-34] - Convenience Method for anonymous login
• [SCRIB-38] - Supply package-info.java

Multi-Module Integration Test Coverage with Jacoco and Sonar

Yesterday I have struggled to capture IT coverage results in a multi-module project setup, which I eventually solved.

So lets assume, I have the following setup:

rootModule
+Module1
+Module2
| +SubModule2-1
|    +SubModule2-1-1
| +SubModule2-2
+ITModule
  +ITModule1

The ITModule contains only integration tests, where ITModule1 is a special scenario, that requires a single module. Module2 consists of nested submodules. There are several examples out there to use a path like ../target/jacoco-it.exec but that's obviously not working if you more than one nesting level.

To know how to solve it, you must understand, how sonar is doing the analysis. When analysing the coverage information sonar checks the code of each module against the coverage file that is specified in the sonar.jacoco.itReportPath property which defaults to target/jacoco-it.exec. So when analyzing Module1 it check for coverage info in Module1/target/jacoco-it.exec. But as the coverage data is captured in the ITModule, respectively ITModule1, I have to point sonar to the file generated in the IT module.
So the best location to gather the coverage data is to the use rootModule, i.e. rootModule/target/jacoco-it.exec and append the results of all IT tests to that file.

I use the following plugin configuration that uses separate files for unit-test coverage (don't forget the append flag otherwise overall coverage will be incorrect) and the central file for IT covergage.

<plugin>
  <groupId>org.jacoco</groupId>
  <artifactId>jacoco-maven-plugin</artifactId>
  <version>0.7.4.201502262128</version>
  <executions>
    <execution>
      <id>prepare-agent</id>
      <goals>
        <goal>prepare-agent</goal>
      </goals>
      <configuration>
        <destFile>target/jacoco.exec</destFile>
        <append>true</append>
        <propertyName>surefireArgLine</propertyName>
      </configuration>
    </execution>
    <execution>
      <id>prepare-it-agent</id>
      <phase>pre-integration-test</phase>
      <goals>
        <goal>prepare-agent</goal>
      </goals>
      <configuration>
        <destFile>${session.executionRootDirectory}/target/jacoco-it.exec</destFile>
        <append>true</append>
        <propertyName>failsafeArgLine</propertyName>
      </configuration>
    </execution>
  </executions>
 </plugin>

The ${session.executionRootDirectory} property is the root of execution, when I build the entire project, it will point to the rootModule. So this is the best path to use, when you have multi-module with more than one level of nesting.

For the analysis, I need to point sonar to use that file when analyzing IT coverage. So I have to set the sonar.jacoco.itReportPath to that file. Unfortunately, this does not work with the session.executionRootDirectory property and I have to set the absolute path to the file manually. I do not recommend to specify the absolute path in the pom.xml as this path is specific to the build environment. So either set the path in Sonar or as System property of your build environment. I set it directly in the Sonar Project Settings (Java > Jacoco), for example /opt/buildroot/myProject/target/jacoco-it.exec. Now sonar will check that file for the IT coverage analysis of each module.

Scribble 0.1.3

 While working on the next release of Inkstand, I had to fix some bugs in the Scribble test framework's injection support which just got released.

Release Notes - Scribble - Version 0.1.3

Bug

• [SCRIB-26] - Check for null injection target
• [SCRIB-27] - TcpPort has no proper toString() representation
• [SCRIB-30] - Field candidates are not collected for null-values

Story

• [SCRIB-29] - Injection does not fail if no target is found

Scribble 0.1.2

Today I released version 0.1.2 of the Scribble testing framwork. Beside some bugifxes, the main improvement was the added support for initializing the JCR ContentRepository test rules with nodetype definitions from a CND file.

 Bug

    [SCRIB-16] - @Inject Annotations is not considered
    [SCRIB-19] - InjectableHolders are not recognized properly when injecting
    [SCRIB-23] - Sonar Issue: The use of XPath.evaluate() is vulnerable to XPath injection
    [SCRIB-24] - Sonar Issue: The usage of /DocumentBuilder.parse(...) is vulnerable to XML External Entity attacks
    [SCRIB-25] - Sonar Issue: Use a cryptographically strong random number generator (RNG) like "java.security.SecureRandom" in place of this PRNG

Story

    [SCRIB-11] - Convenience Methods for InMemory and StandaloneRepository creation
    [SCRIB-20] - Initialize Repository with CND node types

Task

    [SCRIB-17] - Set up Build for master branch
    [SCRIB-18] - Set up Test Quality Assesment
    [SCRIB-21] - Update Apache DS Dependency to 2.0.0-M20
    [SCRIB-22] - Fix "Copyright and license headers should be defined" Rule configuration

User Acceptance Testing with Selenium and Cucumber

In the last implementation project I participated in we applied the Behavior Driven Development approach where the user stories are defined in Given-When-Then style. In this article I want to describe how I combined Cucumber with Selenium in order to automate our User-Acceptance Tests using Behavior Driven Development.

For running automated BDD tests there are some free frameworks available (a brief comparison of BDD frameworks). We decided to go for Cucumber because it served all our requirements, has a good documentation with good examples and is pretty easy to get it up and running

Cucumber JUnit Test

The following listing shows a simple example that is practically the archetype for all our Cucumber tests.

@RunWith(Cucumber.class)
@CucumberOptions(
  features = { "classpath:features/example/" },
  glue = { "my.project.uat." },
  tags = { "@Example" })
public class ExampleCucumberTest {
  //empty
}

The annotations of the example in detail are:

• @RunWith declares the TestRunner for this test, which is the Cucumber class. The test won't run without it.
• @CucumberOptions define various options for this tests. The options are optional but are quite helpful in controlling the behavior of the test
• features: declares a path were the BDD feature files (text files) are found. The example points to a location in the classpath. All feature files (.feature extension) below that location are considered. Multiple locations can be defined as an array.
• glue: defines the packages where Steps and Hooks are located. Steps and Hooks contain the actual code for the tests. Multiple packages can be defined as an array.
• tags: defines which stories should be executed. If you omit this option, all Stories are executed, otherwise only those that have one of the tags set will be run.

Of course there are more options available (see Cucumber apidoc)  but these are the options I use most. As soon as you have your first story written, you can start right away with this simple test.

BDD Feature 

The following exaple story is taken from the Cucumber documentation

@Example
Feature: Search courses
  Courses should be searchable by topic
  Search results should provide the course code

Scenario: Search by topic
    Given there are 240 courses which do not have the topic "biology"
    And there are 2 courses, A001 and B205, that each have "biology" as one of the topics
    When I search for "biology"
    Then I should see the following courses:
      | Course code |
      | A001        |
      | B205        |

With that feature in the right location you can run the above test with JUnit. Of course it will not be successful. Actually, with the default settings it will ignore all the steps unless you use the @CucumberOption(strict=true) which is recommended when you run the test as part of a quality gate.
The Cucumber documentation provides some good descriptions on Features and their syntax. You can define Backgrounds that are executed for each scenario of the feature (similar to JUnit 4 @BeforeClass) or Scenario Outlines to run the feature against a set of data. It is even possible to write your BDDs in different languages. Therefore you have to start the feature with the line following line and have all the keywords in the according language.

#language: de
Funktionalität: ...

But you have to be careful with the encoding of the Feature files and special characters. It's best to use UTF-8 as default encoding. A complete list of the keywords in other languages can be found in the Cucumber apidoc.

Cucumber Steps

When the test for the feature is run and the steps or part of it are not yet implemented, it will produce an output like this:

You can implement missing steps with the snippets below:

@Given("^there are (\\d+) courses which do not have the topic \"([^\"]*)\"$")
public void there_are_courses_which_do_not_have_the_topic(int arg1, String arg2) throws Throwable {
    // Express the Regexp above with the code you wish you had
    throw new PendingException();
}

@Given("^there are (\\d+) courses, A(\\d+) and B(\\d+), that each have \"([^\"]*)\" as one of the topics$")
public void there_are_courses_A_and_B_that_each_have_as_one_of_the_topics(int arg1, int arg2, int arg3, String arg4) throws Throwable {
    // Express the Regexp above with the code you wish you had
    throw new PendingException();
}
...

The test prints out skeletons for unimplemented steps. What you do now is to create a new step which is a simple Java class and put in one of the packages defined in the glue. Copy the skeletons into the class and implement it.
So the steps are basically what is executed for each line. It is possible to pass parameters to the steps that are extracted and converted so that steps can be reused with different values. Its also possible to define entire tables as input.

Hooks

Hooks are basically the same as steps but fulfill a similar role like the JUnit @Before and @After annotated methods, the even use the similar annotations (actually, the are named the same but are in a different package). You can trigger certain hooks using tags like shown in the following example:

@WithFirefox
Scenario: Response times with 10 users
...

and the according hook in Java will be

@Before("@WithFirefox")
public class BrowserHook {
 ...
  public void setupScenario_Firefox() {
   ...
  }
 ...

Dependencies between Hooks and Steps

In order to reuse existing code or to access the state of a particular Hook or Step instance you can create a dependency between the classes by defining a constructor that accepts a particular Hook or Step. The Cucumber JUnit runner will create instances of the according classes and inject them in classes that are dependent.

public class MyBrowserSteps {

  private BrowserHook browserHook;

  public MyBrowserSteps(final BrowserHooks browserHook) {
    this.browserHook = browserHook;
  }

The same applies to Steps so you can make one set of steps dependent on other steps.

Selenium Steps

So far I only described how to write any test with Cucumber, but for User Acceptance Testing you might want to test the actual solution. For web application that is the deployed application that is accessed by a browser. For browser automation the Selenium framework is widely known and framework of choice for most cases. It provides a recording tool (a plugin to Firefox) to record user interactions with the browser. It provides a model to access elements of the website using Java and various methods of locating elements in the browser.
For automated user-acceptance tests with Selenium and Cucumber the basic approach would be to

1. Record actions with the Selenium Recorder
2. Copy them to Steps classes that match your BDD
3. Define assertions in Then step implementations

A step implemented with Selenium will look like in the following example

@When("^I push the button$")
public void i_push_the_button() throws Throwable {
  driver.findElement(By.cssSelector("div.v-select-button")).click();
}

In order to reuse steps for different browsers, I used the BrowserHook shown in on of the example on above and set-up the browser using a specific tag for each browser. The driver is first initialized upon the first call to getDriver(). In the step itself I retrieve the driver from the BrowserHook that got injected. The BrowserHook may be implemented like this

public final class BrowserHooks {
  private enum DriverType {
    headless,
    firefox,
    ie,
    chrome, ;
  }
  
  private WebDriver driver;

  public WebDriver getDriver() {
    if (driver == null) {
      switch (driverType) {
 case ie:
   driver = new InternetExplorerDriver();
   break;
        case firefox:
          driver = new FirefoxDriver();
          break;
        ...
    }
    return driver;
  }

  @Before("@WithFirefox")
  public void setupScenario_Firefox() {
    driverType = DriverType.firefox;
  }
  
  @Before("@WithIE")
  public void setupScenario_InternetExplorer() {
    driverType = DriverType.ie;
  }
  ...
}

And the step definition that uses it may look like

public class MyBrowserSteps {

  private BrowserHook browserHook;

  public MyBrowserSteps(final BrowserHooks browserHook) {
    this.browserHook = browserHook;
  }

  @When("^I push the button$")
  public void i_push_the_button() throws Throwable {
    this.browserHook.getDriver().findElement(By.cssSelector("div.v-select-button")).click();
  }
...

Aggregate Steps

One of the big advantages of a BDD framework like Cucumber is, that you can define steps that aggregate multiple steps. A good example for this is the Login Story. Although this is a point of typical discussions whether "Login User" is a valid Use Case or User Story (with regards to its business value) the requirement to allow a user to login does exists and its parameters need to be defined (whether it is via Single Sign On, Smartcard, Username/Password, Two-Factor or whatever else).
So let's assume you define a login user story such as

 Given the login screen is being displayed
 When I enter my username "xxx" and my password "yyy"
  And I push the login button
 Then I see the main screen of the application
  And I see my name being displayed in the user info box

Now you don't want to describe all these steps over and over again because the rest of the application under test requires a logged in user. So you could begin the other stories with

• When the user "xxx" is logged in

Or even better by using a hook/tag before the story like

• @Authenticated

Now, what you do in your code is to define a dependency to the steps class that contains the login step definitions and invoke each of them in the correct order either in a hook definition or in a step definition. The advantage of a hook is that you could combine it with other hooks to set up the test user or even persona.

public class LoginSteps {

  private BrowserHook browserHook;

  public LoginSteps (final BrowserHooks browserHook) {
    this.browserHook = browserHook;
  }


  @Given("^the login screen is being displayed$")
  public the_login_screen_is_being_displayed() {
    this.browserHook.getDriver().get(baseURL);
  }
  @When("^I enter my username \"([^\"]*)\" and my password \"([^\"]*)\"$")
  public void I_enter_my_username_and_my_password(String arg1, String arg2) throws Throwable {
    //with Selenium, put in the values in the login form
  }
  @When("^I push the login button$")
  public void I_push_the_login_button() throws Throwable {
    // with Selenium, locate the submit/login button and click it
  }
  ...
}

public class LoginHook {

  private LoginSteps loginSteps;
  private String testUser;
  private String testPassword;

  public LoginHook (final LoginSteps loginSteps) {
    this.loginSteps= loginSteps;
   
  }

  @Before(value="@PersonaXY", order=1)
  public void selectPersonaXY() {
     this.testUser = ...;
     this.testPassword = ...;
  }

  @Before(value="@Authenticated", order=2)
  public void login() {
    this.loginSteps.the_login_screen_is_being_displayed();
    this.loginSteps.I_enter_my_username_and_my_password(testUser, testUserPassword);
    this.loginSteps.I_push_the_login_button();
    ...
  }
}

And how it is used in a story

 @Authenticated @PersonaXY
 Given I see the meaningful screen
 When I do something purposeful
 Then I get a sensible result

 Conclusion

In this article I gave a brief introduction into Cucumber and how to write testcases with it. I showed how to define steps with Selenium to create meaningful, browser-based user acceptance testing and I showed how to combine thereby reuse steps and hook to create a rich user acceptance testing suite.

JUnit Testing with Jackrabbit

Writing unit tests for your code is not only best practices, it's essential for writing quality code. In order to write good unit tests, you should use mocking of code not under test. But what if you're using a technology or an API that would require quite a lot of complicated mocks?
In this article I'd like to describe how you write unit tests for code that accesses a JCR repository.

At first I really tried to mock the JCR API using Mockito, but stopped my attempt at the point where I had to mock the behavior of the Query Object Model. It became apparent, that writing mocks would outweigh the effort to write the actual production code by far. So I had to search for an alternative and found one.

The reference implementation of JCR is the Apache Jackrabbit project. This implementation comes with a set of JCR Repository implementations, one of these is the TransientRepository. The TransientRepository starts the repository on first login and shuts it down on the last session being closed. The repository is created in memory which works pretty fast and makes it the best solution for unit testing. But nevertheless, a directory structure is created for the repository and unless not specified a config file is created as well.

For writing unit tests against this repository, we need the following:

• a temporary directory to locate the directory structure of the repository
• a configuration file (unless you want one created on every startup)
• the repository instance
• a CND content model description to initialize the repository data model (optional) 
• an admin session to perform administrator operations
• a cleanup operation to remove the directory structure
• the maven dependencies to satisfy all dependencies

Let's start with the Maven dependencies. You need the JCR Spec, the Jackrabbit core implementation and the Jackrabbit commons for setting up the repository.

<properties>
  <!-- JCR Spec -->
  <javax.jcr.version>2.0</javax.jcr.version>
  <!-- JCR Impl -->
  <apache.jackrabbit.version>2.6.5</apache.jackrabbit.version>
</properties>
...
<dependencies>
<!-- The JCR API -->
  <dependency>
    <groupId>javax.jcr</groupId>
    <artifactId>jcr</artifactId>
    <version>${javax.jcr.version}</version>
  </dependency>
  <!-- Jackrabbit content repository -->
  <dependency>
    <groupId>org.apache.jackrabbit</groupId>
    <artifactId>jackrabbit-core</artifactId>
    <version>${apache.jackrabbit.version}</version>
    <scope>test</scope>
  </dependency>
  <!-- Jackrabbit Tools like the CND importer -->
  <dependency>
    <groupId>org.apache.jackrabbit</groupId>
    <artifactId>jackrabbit-jcr-commons</artifactId>
    <version>${apache.jackrabbit.version}</version>
    <scope>test</scope>
  </dependency>
</dependencies> 

Now let's create the directory for the repository. I recommend to locate it in a temporary folder so multiple test runs don't affect each other if cleanup failed. We use the Java TempDirectory facility for that:

//prefix for the repository folder
import java.nio.file.Files;
import java.nio.file.Path;
...
private static final String TEST_REPOSITORY_LOCATION = "test-jcr_";
...
final Path repositoryPath = 
      Files.createTempDirectory(TEST_REPOSITORY_LOCATION);

Next, you require a configuration file. If you already have a configuration file available in the classpath, i.e. in src/test/resource, you should load it first:

final InputStream configStream = 
  YourTestCase.class.getResourceAsStream("/repository.xml");

Knowing the location and the configuration, we can create the repository:

import org.apache.jackrabbit.core.config.RepositoryConfig;
import org.apache.jackrabbit.core.TransientRepository;
...
final Path repositoryLocation = 
      repositoryPath.toAbsolutePath();
final RepositoryConfig config = 
      RepositoryConfig.create(configStream, repositoryLocation.toString());
final TransientRepository repository = 
  new TransientRepository(config);

If you ommit the config parameter, the repository is created in the working directory including the repository.xml file, which is good for a start, if you have no such file.

Now that we have the repository, we want to login to create a session (admin) in order to populate the repository. Therefore we create the credentials (default admin user is admin/admin) and perform a login:

final Credentials creds = 
  new SimpleCredentials("admin", "admin".toCharArray());
final Session session = repository.login(creds);

With the repository running and an open session we can initialize the repository with our content model if require some extensions beyond the standard JCR/Jackrabbit content model. In the next step I import a model defined in the Compact Node Definition (CND) Format, described in JCR 2.0

import org.apache.jackrabbit.commons.cnd.CndImporter;
...
private static final String JCR_MODEL_CND = "/jcr_model.cnd.txt";
...
final URL cndFile = YourTestCase.class.getResource(JCR_MODEL_CND);
final Reader cndReader = new InputStreamReader(cndFile.openStream());
CndImporter.registerNodeTypes(cndReader, session, true);

All the code examples above should be performed in the @BeforeClass annotated method so that the repository is only created once for the entire test class. Otherwise a lot of overhead will be generated. Nevertheless, in the @Before and @After annotated methods, you should create your node structures and erase them again (addNode() etc).

Finally, after you have performed you test, you should cleanup the test environment again. Because a directory was created for the transient repository, we have to remove the directory again, otherwise the temp folder will grow over time.
There are three options for cleaning it up.

1. Cleaning up in @AfterClass annotated method
2. Cleaning up using File::deleteOnExit()
3. Cleaning up using shutdown hook

I prefer combining 1 and 3 for fail-safe deletion. For option 1 we require a method to destroy the repository and cleaning up the directory. For deleting the directory I use Apache Commons FileUtil as it allows deletion of directory structures containing files and subdirectories. The method could look like this:

import org.apache.commons.io.FileUtils;
...
@AfterClass
public static void destroyRepository(){
  repository.shutdown();
  String repositoryLocation = repository.getHomeDir();
  try {
    FileUtils.deleteDirectory(new File(repositoryLocation));
  } catch (final IOException e) {
   ...
  }
  repository = null;
}

As fail-safe operation I prefer to add an additional shutdown hook that is executed when the JVM shuts down. This will delete the repository even when the @AfterClass method is not invoked by JUnit. I do not use the deleteOnExit() method of File as it requires the directory to be empty while I could call any code in the shutdown hook using my own cleanup implementation.


A shutdown hook can easily be added to the runtime by specifying a Thread to be executed on VM shutdown. We simply add a call to the destroy methode to the run() method.

Runtime.getRuntime().addShutdownHook(new Thread("Repository Cleanup") {
  @Override
  public void run() {
    destroyRepository();
  }
});

Now you should have everything to set-up you Test JCR Repositoy and tear-down the test environment. Happy Testing!