How to fake tests (Part 1)

In most projects, metrics play an important role to determine the status, health, quality etc. of the project. Not rarely the common metrics for quality have been

• Number of Unit Tests (Total, Failed, Successful)
• Line Coverage
• Branch Coverage

Usually those “KPI” (Key Performance Indicators) were used by “managers” to steer the project to success. The problem with these metrics is: they are totally useless if taken out of context - and the context is usually not that well defined in terms of metrics, but often requires knowledge and insight into the system that’s been measured.

This post is about to show how to game the system and life-hack those KPIs to fake good quality. It’s NOT a best practice but a heads up to those who make decision based on those metrics to look behind the values.

Faking Number of Unit Tests

Most (if not all?) frameworks count the number of tests executed, which failed and which succeeded. A high number of tests is usually perceived as a good indicator of quality. The increase of the amount of tests should correlate with the increase in lines of code (another false-friend KPI). But what is counted as a test?

Let’s look at the Junit which is the de-facto standard for developing and executing Java based unit tests, but other frameworks such as TestNG follow similar concepts.

In Junit 3 it was every parameterless public void method starting with “test” in a class extending TestCase. Since Junit 4 every method annotated with @Test counts as a Test.

That’s it. Just a name convention or an Annotation and you have your test, so let’s fake it!

@Test
public void test() {

}

This is pure gold: a stable and ever succeeding Unit Test!

Copy and paste or even generate those and you produce a test suite satisfying the criteria:

• Big, tons of tests, probable even more than you have LoCs
• Stable, none of these tests is failing. Ever.
• Fast, you have feedback about the success within seconds.

The downside: it’s worthless (surprise, surprise!). There are basically two primary reasons, why its worthless:

• It doesn’t run any code
• It doesn’t pose any assertion about the outcome

Good indicators to check the first one are line or condition coverage analysis. The latter is more difficult to check.

In the upcoming posts we'll have a look into both.