More about the coverage – let’s get it right and fast automatically


I was writing once about superb TCases software which allows to put pair-wise testing into practice. I said it was a giant leap towards the right coverage. In this short article I would like to go much further. This is not going to be another leap, it is going to be a flight 🙂

The problem

When dealing with a problem in QA practice it has often combinatorial nature. There are many possible combinations of either data, actions or other “inputs” which constitute the testing space. The testing space always tends to be infinite or at least extremely large as not only must one take into consideration the fact all the inputs have to be used in test runtime but also all the relations between them. Such a relations can only be tested when specific combinations of inputs will be used. Thus, the main problem to solve is how to choose those combinations. Well, this one is already solved here.

When testing combinations are known next problem arises quickly. There are always many combinations to test. There is huge work to do to create automated tests out of generated combinations not to mention manual testing. And there is even more work to do when system under test changes as the tests have to be adjusted accordingly. Let’s solve this problem.

The solution

Automation is the process which started quite long time ago. We are not testing manually, we have set of automated tests to run. We can use them on all the testing levels: small, medium and large tests are automated and used in development process continuously. However before we can run the test we have to write it. It takes time as it is still manual process nowadays. It is now time to move on and start creating test cases automatically!

It looks more less like this:


old process

It is important to move to such a process:


new process

When speaking about combinatorial testing problem we have TCases at hand which utilizes pair-wise testing concept and generates the set of testcases to be created. After that, such a raw testcases need to be translated into executable test cases:


automatic test cases design time


There are domain specific languages in use at present. I wrote shortly about it here already. I am going to use internal domain language that is a language crafted from Java itself.

Let’s assume, we want to test the behaviour of Notepad++ application in terms of new document settings with regards to format, encoding and default language. These settings are available at: Settings->Preferences->New Document.

I am going to use all the available formats and 5 languages and encodings for the sake of simplicity (in reality, one needs to use all of them in the testing space of course).

We have 3 variables: format, encoding and default language. Each variable contains information related to testing space that is which values are possible and at the same time it contains information related to domain language (Has attributes). Domain language contains two kinds of information: the one related to GIVEN and WHEN clauses (command) and the one related to THEN clause (expected).

The testing space looks like this (input.xml):

So, for example when “format_isChecked” variable will have FORMAT_WINDOWS value, it will mean:

translated later into:

All the hashes and asterisks are used as placeholders only. It is much easier to write dsl related info when using external domain language like Cucumber. Dealing with internal one like in this example (Java) is more problematic because of braces and dots.

Anyway, each variable value will be translated into GIVEN/WHEN and THEN sentences and each variable value which belong to the same testcase will be concatenated with each other to create a single dsl test case.

To achieve this, we need to create output xml file with raw testcase which are not executable yet (they are manually executable though). I use 2-tuple coverage:

After TCases is run we get (output.xml):

The coverage is pair-wise (2-tuple) only but we already have 35 test cases! We are not afraid of high amount of required test cases anymore.

Now, we just need the last step and generate executable test cases. Here we go with the generator:

After generator is run with output.xml file we get this:

Which is the set of executable test cases. Now, just paste it into JUnit file: