Background

I was writing once about TCases software which allows to put pair-wise testing into practice. I said it was a giant leap towards the right coverage. In this short article I would like to go much further. This is not going to be another leap, it is going to be a flight 🙂

The problem

When dealing with a problem in QA practice it has often combinatorial nature. There are many possible combinations of either data, actions or other “inputs” which constitute the testing space. The testing space always tends to be infinite or at least extremely large as not only must one take into consideration the fact all the inputs have to be used in test runtime but also all the relations between them. Such a relations can only be tested when specific combinations of inputs will be used. Thus, the main problem to solve is how to choose those combinations. Well, this one is already solved here.

When testing combinations are known next problem arises quickly. There are always many combinations to test. There is huge work to do to create automated tests out of generated combinations not to mention manual testing. And there is even more work to do when system under test changes as the tests have to be adjusted accordingly. Let’s solve this problem.

The solution

Automation is the process which started quite long time ago. We are not testing manually, we have set of automated tests to run. We can use them on all the testing levels: small, medium and large tests are automated and used in development process continuously. However before we can run the test we have to write it. It takes time as it is still manual process nowadays. It is now time to move on and start creating test cases automatically!

It looks more less like this:

It is important to move to such a process:

When speaking about combinatorial testing problem we have TCases at hand which utilizes pair-wise testing concept and generates the set of testcases to be created. After that, such a raw testcases need to be translated into executable test cases:

Example

There are domain specific languages in use at present. I wrote shortly about it here already. I am going to use internal domain language that is a language crafted from Java itself.

Let’s assume, we want to test the behaviour of Notepad++ application in terms of new document settings with regards to format, encoding and default language. These settings are available at: Settings->Preferences->New Document.

I am going to use all the available formats and 5 languages and encodings for the sake of simplicity (in reality, one needs to use all of them in the testing space of course).

We have 3 variables: format, encoding and default language. Each variable contains information related to testing space that is which values are possible and at the same time it contains information related to domain language (Has attributes). Domain language contains two kinds of information: the one related to GIVEN and WHEN clauses (command) and the one related to THEN clause (expected).

The testing space looks like this (input.xml):

So, for example when “format_isChecked” variable will have FORMAT_WINDOWS value, it will mean:

translated later into:

All the hashes and asterisks are used as placeholders only. It is much easier to write dsl related info when using external domain language like Cucumber. Dealing with internal one like in this example (Java) is more problematic because of braces and dots.

Anyway, each variable value will be translated into GIVEN/WHEN and THEN sentences and each variable value which belong to the same testcase will be concatenated with each other to create a single dsl test case.

To achieve this, we need to create output xml file with raw testcase which are not executable yet (they are manually executable though). I use 2-tuple coverage:

After TCases is run we get (output.xml):

The coverage is pair-wise (2-tuple) only but we already have 35 test cases! We are not afraid of high amount of required test cases anymore.

Now, we just need the last step and generate executable test cases. Here we go with the generator:

After generator is run with output.xml file we get this:

Which is the set of executable test cases. Now, just paste it into JUnit file:

 

Things you might want to see

If you want to see how the runtime looks like here it is:

And if you are interested in the code, you can get it from HERE. Use automatic_tc_generation branch.

As you can see we do not have to pay any attention to test cases anymore. We just focus on testing space that is variables which influence the behaviour of the system. The rest happens automatically.

Sum up

I showed here practical example for extremely heavy tests as you can see on the movie. I also used internal domain language which is not perfect for testcase generation. But this is valid approach for all testing levels. Component tests, integration tests, system tests – all of them can take advantage. If you have any testing framework which can have tests expressed in some kind of nice language, it is easy to write the generator. Once generator is in place, you can have hundreds of test cases to deliver optimal coverage.

This is a very important approach in my opinion and it really makes the difference when you do QA job. When system changes for some reason, you just need to concentrate on testing space to adjust it to new variables. Then, you just hit generate button. Documentation of the coverage is in place as well: it is clear what coverage is used. It is clear how much it can be increased or decreased.

Don’t write test cases, generate them – it means stop jumping and start to fly.