In the previous article I mentioned about attention we need to pay to the resources we have when flattening the tests. Still, the available resources end at some point.

Let’s imagine we have a pipeline with quick/small tests launched after each commit. As new features are added, they get longer and longer. We want to be sure our software is working so apart from unit tests we add integration/medium tests as well as heavy/large part doing end to end stuff.

We try to flatten the tests and start large ones as fast as possible but they are so slow it doesn’t help. We try to run large tests phase every day once in the evening but then we can only learn the next day something is wrong with the software which is not what we want.

 

What can we do? How can we quickly get feedback about the application under test while having all of the tests executed? Quickly – means something between 20 or 30 minutes and not hours of waiting…

We can find a way to the solution by not analyzing the pipeline anymore but try to look at it from other point of view: the feedback point of view. What is the group of people who needs the feedback then? What will happen if the feedback for some area of the application will be delayed?
Most often there are teams in the company which deal with specific parts of the application: team 1 will do module A and B while team 2 will do C, D and E. Let’s imagine module B is the one involved in large tests. It is tested by many unit tests but then undergoes long phase of end to end tests which last for example 1 hour.
We need to separate large tests phase just as we did when running it once a day, but trigger them synchronously with small tests and making sure it will use the same build artifact as small and medium tests do. The triggering of the large tests phase has to be done only when they are not in progress to prevent the tests queue to grow indefinetely.

Thus, we get feedback as fast as we can for small and medium tests (every 30 minutes) which allows team 2 to keep the pace. Team 1 also benefits as it gets feedback about module A very often. It gets feedback about module B as fast as possible that is about every 75 minutes if testing cycle is active all the time.
If our pipeline is continuous deployment one, we need a trick to allow us to deploy only fully tested code as some of the commits will not be tested by large phase at all. This can be done by tagging the code by each of the test phase (small/medium and large) individually and letting only fully tagged commit id to be deployed.
The only price we pay is that we do not know what is the full test result of every single commit. We get the full feedback about green and yellow tests but red ones are only known sometimes. The runs with uncertain large tests result in diagram 4 are run numbers: 2,3,5,6 while 1 and 4 are fully tested and may be safely released or deployed. I am sure this is very small price for advantages we get.
This idea may be used to divide the pipeline into even more phases if there are more of them which do not fit into 30 minutes basic tests duration. More large tests or maybe some additional medium ones. All of them may be triggered when small tests start under condition they are not in progress and all of them may set individual tags to mark the commits they were testing to be able to always track commits which are safe for release. We have separate feedbacks for all of the phases done as fast as possible.

It is better to have all the tests be very fast and finish after 20 minutes but if this is impossible the above example approach may help to keep both the rich and quick feedback in place.