Another approach to generate test cases

I described a way to generate test cases some time in the past HERE. Recently, I tried another approach which I put in Github as a real example: https://github.com/grzegorzgrzegorz/testcase-generation. This is also using TCases to generate non-executable test cases from model. More about TCases HERE

The concept of the approach is:

  • create model file
  • generate non-executable test cases (by TCases)
  • prepare variable related classes for deserialization
  • generate test cases by using the code which under the hood deserializes non-executable test cases in order to generate executable code

For example: there is an application which converts text into valid sentence, that is makes first letter capitalized and adds dot at the end. If there is variable called Capitals in the model, there has to be class Capitals prepared. This requires some work, but it allows to separate executable code from model. In the code snippet below there is one of the non executable test cases which has Capitals variable set to “NotFirstLetterCapitalized”:

Information about how to prepare string with regards to capitalization and how to assert it are put into Capitals class:

The generated, executable test case looks like this:

To sum up: the main advantage of this approach is the separation of DSL syntax and model data. There is much more info on Github: https://github.com/grzegorzgrzegorz/testcase-generation.

3 stages of pipeline tests

It is not so easy to develop pipeline at certain level of complication. I was describing pipeline testing framework in one of the earlier posts. It is great tool to catch many defects very fast. However, in my opinion it is capable of doing so for about 50% of them. The rest leaks to production. Why is this happening?

In the past I created a post dealing with the problem of 4 great variables. So here the problem hits us in practice: it is only possible to catch most of the defects related to CODE using pipeline testing framework but none related to ENVIRONMENT or CONFIGURATION or DATA.

What can we do? We need to add extra testing stages which will fill in the gaps.

1. stage – pipeline testing framework

It can catch most of the problems related to code: logic problems, syntax problems and so on. Multiple tests are possible to be written verifying logic paths, variable values, overall syntax correctness and expected communication with other jobs or libraries (names, input parameters etc.).

2. stage – jenkins validation

In this stage we send pipeline under test to jenkins for validation. Two steps are required:

More information on this is here: https://www.jenkins.io/doc/book/managing/cli/ and here: https://www.jenkins.io/doc/book/pipeline/development/.

The same code which passed 1. stage should be sent to jenkins application to run declarative-linter command. It is able to find code problems which pipeline testing framework cannot. These are defects related to Jenkins specific things like mandatory sections required by declarative pipelines. The best example for me is STEPS section which cannot be missing but 1. stage is not able to validate it properly.

3. stage – pipeline draft run

This stage is meant to catch the rest of the defects. While it means this should be able to detect configuration, environment and data problems it needs to be as similar to production run as possible. To achieve it, the following is needed in my opinion for draft run to be useful:

  • it should use the same files as production
  • it cannot influence external systems in any way
  • it should be possible to run it fast

Two first points there can be fixed by creating draft launcher job which sets configuration in a way no interaction with outside world is made (Jira communication, repository communication, user communication are all set to off). Draft jobs are created in Jenkins which are using the same files as production jobs but with modified configuration by draft launcher.

At this point, pipeline draft run is possible but it will take the same amount of time as production one. This is not desired behaviour for sure. We come to the third point here.

This point is hard to do but really important. To make the pipeline run fast, all time consuming operations need to be replaced with some dummy operations. It means for pipeline to run successfully, real data from previous run has to be used.

I am solving this problem in this way:

  • pipeline puts build artifacts to shared disk so that they are accessible by all Jenkins nodes
  • it is reusing them in consecutive stages/jobs
  • at the same time I can reach build artifacts location from the past run and copy it aside to draft workspace to use it in draft pipeline
  • during draft run job which is building application reconfigures its workspace (configuration parameter) to draft workspace just after cloning repository
  • building phase can be completely skipped and few unit tests can be run just to check reporting in Jenkins (configuration parameter)
  • etc.

Conclusion

Having all 3 stages in place gives me high confidence pipeline works properly. Moreover it is possible to develop it rapidly without testing it in production. I get the feedback about current code, configuration, environment and data. I can also change any of this variable values and retest quickly. Fast feedback is the essence of development. We all need it.

Do not be fanatic, be flexible.

I met professional fanatic. Who is fanatic?

  • he always knows everything (and he has good explanation for everything)
  • never modifies his way of thinking
  • has zero flexiblity no matter the circumstances

Imagine asteroid coming to hit the earth and cause ELE. Fanatics work on program which should save the earth. It will operate some high energy laser which will hit up asteroid and change its course. There are 12 hours before the impact.
Fanatics start their TDD approach. They are relaxed and self-confident as they know they create highest quality code. Somewhere between 2. and 3. version of the feature branch, after 20 tiny tickets and total amount of about 35 reviews of even tiner mini-branches asteroid hits the earth and turns it into a desert with no life on the surface.
If fanatics have been on the moon, they would have been happy because just when asteoid was hitting the earth, they understood application domain finally and 3. version of the feature branch was meant to be production version. This wonderful feeling you are on perfect path is worth any cost. It doesn’t matter for them their work is useless from that point on.
Here comes the only bright side of ELE: there are no fanatics anymore.

How not to be a fanatic?

It is simple, when there is question which contains word “always” never answer yes:

  • should you always work using TDD? No ! Only when it makes sense. I totally agree with this interesting article: https://techleadjournal.dev/episodes/58/ stating TDD should be applied in the right moment. You need to make some code and only some tests after when you try to understand application domain. Working TDD style from the scratch slows you down by the multiple factor and is waste of time as your work goes to the junk anyway on this stage. Only after you know you are ready to proceed with given feature branch you should start TDD.
  • should you always have pyramid like test levels? No ! Only when it makes sense. If your application for example is some rest api thing which has few functions which do not interact with external systems, you can even have square like test levels shape: number of integration tests could be the same as unit tests. The only disadvantage of integration tests in such case is slow feedback as if they fail you do not immediately know where is the point of failure. But you have your unit tests set which will tell you that and large number of integration tests will save you from regression.
  • should you always create highest quality possible code? No ! Only when it makes sense. When you create a helper tool for analyzing/filtering/reporting some stuff, make a ticket, make a feature branch and a decent review. Check if code follows good practices, if it is clear and most importantly has good set of tests. And this is good enough! Do not spend hours on the review torturing the details – you really can stop at some point before reaching the perfection. It is a tool, like a car wrench. If it is not useful anymore you can just throw it away and create new oone. 

Do not waste your time on:

  • making multiple tickets for simple application
  • making multiple branches for simple application
  • making each commit be atomically consistent

Do not make simple task bureaucratic horror. I am not afraid to say it loud – in general you can just afford some technical debt. It just needs to be calculated well. Very often the cost of delivering late is much higher.
It is better to have something decent on schedule than have something perfect when it is not needed anymore.

Few recipes for declarative pipeline

It is often required to iterate over some set of names using sequential or parallel stages.
The good example is when we have list of modules – possibly retrieved automatically – which we want to test, sequentially or in parallel.

1. Sequential example.

Let’s look at sequential example:

First we just build the application without executing any of the tests.

In my opinion tests should be always separated from the build process – even if they are unit tests. The task seems to be easy but it requires some attention.
Firstly, we need to use scripted pipeline approach. Thus it is possible to use for loop. Each of the stage gets generated test name.
Secondly, we need catchError step. If not used, the pipeline would abort on first unsuccessful iteration, while we want all iterations to execute no matter the status is.
Thirdly, after each iteration we need to preserve the surefire output result – testng in this example – so that it can be archived properly.
All of the stages of this pipeline are executed using the same node and the same workspace path.

2. Parallel example.

Making the pipeline parallel I call flattening. Let’s see the example:

 

The key to understand more complex pipelines is to understand where each stage is executed in terms of workspace path and machine. In the example here all the stages which are generated are executed in parallel as parallel keyword is used. It means there will be different workspaces used for each of them (they are executed on the same machine): workspace_path, workspace_path@1, workspace_path@2 etc.
(it is also possible to configure the stages, so that they are executed on different Jenkins nodes).
Thus the pipeline needs to first stash the build artifacts after BUILD stage and then unstash them in generated stage. Stashed files are stored in Jenkins master to allow accessibility to all of the stages no matter their location.
In this example we do not need catchError step as even when any of the stage fails, the rest will still be executed. Testng report file will not be overwritten (all of the reports reside in different workspaces) but it is good practice to rename it so that Jenkins can handle it properly when sending them to Jenkins master for report to be generated. In this example all the reports are gathered in one place and reported in one go while log files are archived after each stage finishes.

3. Summary.

It was just a few simple recipes of the pipelines. There is of course much more: we can split the pipeline not only into stages but also into jobs. BUILD stage could be separate job in the above example. We can do more tasks in parallel than just tests like multiple checkouts, multiple builds. The parallelism can be related to either stages or jobs.
The more things are flattened the more attention we need to draw to the resources we have: number of machines, number of Jenkins nodes on each of them in comparison to available memory and processors, disk read/write speed.
It is required so that the pipeline speed increases with flattening process and not the opposite.