Statewise testing (GUI automation part IV)

Introduction

You must have heard about state transition tables. This is great test design technique to let us achieve the right coverage when testing with GUI application. We analyze the application from its states perspective to be able to adjust the coverage we can achieve according to resources we have available. In this part about GUI automation I will show how easy we can adapt the testing framework to use it.

Details

We have a framework at this point available which has a DSL layer and it looks very nice (as shown HERE). What we can test at this point (from test design technique point of view) is actually one test case decision table like this:

action TC 1
GIVEN().tab().isOpened(). X
WHEN().tab().isWritten(text). text = “text1”
AND().tab().contentIsCopied(). X
THEN().clipboardContent().shouldBe(equalTo(text)); X

However we need to be able to look on the application from state transition point of view, to be able to act basing on the state transition diagram. We would like to have an automation which can execute state transition diagram by walking the paths we choose.

Let’s look at the example of one – it focuses on the feature LTR/RTL which is text direction left to right and vice versa in Notepad++.

text direction feature in Notepad

text direction feature in Notepad

State transition diagram with text direction focus

State transition diagram with text direction focus

As you can see there is one main state which is one tab selection in Notepad++ (there is always some tab selected). There are also 2 substates which indicate if RTL or LTR is used. So it means when we start using the application we always start with 1 tab selected and LTR text direction.  Then, we can change the text direction or do any other transitions like createNewTab, selectTab or deleteTab. I do not want to write here all the state transition diagram details (I hope to create separate post on test design techniques some day) – let’s only say the basic coverage for this technique should be to execute each state and each transition at least once.
We would like to be able to transform this diagram into Java code and introduce one extra layer on top of we already have in the Sikuli testing framework and then run it.

Let’s do it!

Every of 3 states will be represented as separate class in Java:

Now we can navigate through the state transition diagram using IntelliJ suggestions and we can make any path we wish. As the result we can adjust the coverage to our needs.
The example test case based on the state transition diagram looks like this:

What we test here is we navigate through all the states and transitions at least once and at some point we execute assertTabIsWritable. By doing so, we test if the tab currently selected in the test flow is writable or not. Under the hood we use here Sikuli based testing framework which is described in previous posts.

As you can see we are using the existing framework to check if tab is writable. We are focusing only on this feature. We of course could be checking much more: like when executing createNewTab we could be checking if extra tab was created etc. but I skipped it for simplicity sake.

The video goes here:

You may also view the source code which is
HERE under GUI_automation_part4 branch.

Summary

This is just an example of the automation of tests which are basing on state transition diagram. The power of this approach is related to this test design technique. Firstly, if a requirement is described in this way we are ready for automatic testing. Secondly, in my opinion when such a diagrams which decribe specific funcionality exist in the project, it is possible to use them for example for purpose of retesting. If there is a new feature F1 in functionality A1 it is possible to have automated test for the A1 functionality by executing its state transition diagrams to check if nothing is spoiled there by F1.

The great GUI automation summary

So this would be all I wanted to present about GUI automation. This framework in my opinion is extremely flexible. This could be used for any application – not only GUI. This is just a matter of substitution of Sikuli with other thing like some web service self made framework for testing web service application. One could simplify it from programmatic point of view, one could add tests which would be testing framework itself (unit tests), or else improve the DSL layer. Still even such simple version is usable and shows the potential.
I hope that now the tank comparison from the 1. part (HERE) is more clear 🙂

Start speaking your own language (GUI automation III)

Introduction

At this point we KNOW HOW TO AUTOMATE and we know HOW TO AUTOMATE TESTING. Still although the way the tests are created seems to be very flexible, their content remains unclear especially for non developers. They look more like a code and they may require extra documentation to clear them out. What should we do to make it easy to understand for every party in the project, and speed up writing the tests at the same time? The advantage is at this point we have a custom framework so we can improve it any way we wish. Let’s move towards behaviour driven development then.

Behaviour driven development

You can read more about it HERE. In general it means focusing on behaviour of the system by describing it with help of domain specific language (dsl) that is a language which is on one hand easy to understand for everyone because of its similiarity to English and on the other hand is tightly related to the application itself. The main advantages in practice are: improved overall communication (we don’t have to ask what business analyst had on his mind, nor what is the specific test doing), automation speed up (the behaviour described in domain specific language can be directly executable, which means just after it is created it can be run by QA engineer or tester), documentation improvement (we do not have to create aditional documentation as the described behaviour of the system becomes documentation itself).
As you can read in Wikipedia the popular template for the dsl are GIVEN, WHEN, THEN words. Thinking about it from a perspective of automation engineer one can say GIVEN may be considered as a setup, WHEN is a body of a test and THEN is a assertion. Let’s use it in practice!

Notepad specific language

For interacting with Notepad we need specific language. There is no such a language yet so let’s create Notepad Specific Language – NSL.
We need words which are related to Notepad GUI entities like document, tab, bookmark and actions like open, write, close. Let’s look at the example below:

So if we follow the pattern from Wikipedia, we can say:

It looks much better now, doesn’t it ?

Test report can also speak NSL

After the test is run we need to review results. Often happens we need to know what actions were taken at specific point in time. Let’s look at the test ouput we get when appropriate DSL logging is present.

As we can see not only the testcase itself becomes very clear but also the logging done in NSL level when analyzed with application logs allows mapping each action done on system test level to the sequence of events which take place on single service and class levels (component level) at the given point in time. Of course this is easy now to do the reverse thing and tweak logging a little bit to allow copy paste of log output to allow instant replay of the logged actions. So if we just change the logging messages a bit so that all of them are Java methods, we will be able to copy paste them and execute instantly. Consider this:

 

More about test report

This is nice thing we can see the test run log in NSL, but this is even better to have the screenshots. This is good test proof and very good way to see what went wrong in case test produce error instead of pass or fail, especially when it is hard to tell from application logs what is the reason. We are using custom framework so we can have screenshots anyway we like. Let’s do it then.

In this example screenshot is made in BDD layer of THEN class just before hamcrest based assertion is made. Screenshot information is added to the log messages. Screenshot itself is made by Sikuli method capture which is using java.awt.Robot under the hood. The class is one of the 3 main mechanisms which drive Sikuli as you can learn HERE.

As you can see in the example, there is parameter passed to the method which allows for capturing not only the full screen but also any region which is used by Sikuli. As for previous posts I share the code I was telling you about HERE under GUI_automation_part3 branch.

The movie goes here:

Summary

There are new nice things I presented you here. BDD aproach constitutes efficient solution for quality control. However, we can do more, we can do better. As usual… I will cover this in my next part about GUI automation.

How to automate testing of GUI application ? (GUI automation II)

Let’s start testing

In the last ARTICLE I showed how to automate actions for Notepad++. It is time to apply this to get the real value for quality control of the application: test automation.
For this purpose we need something which will allow us to follow automation good practices namely create small independent tests with setups, teardowns and reports.

Details

Let’s look at JUnit with hamcrest matchers. This is universal solution used very often by developers for their unit testing.
The example test looks like this:

We have few interesting things here: the TestCases class has 1 test in this example. I call such a class a test set as it can contains many tests. We have a test set then with 1 test, with setup (@BeforeClass) and teardown (@AfterClass). Those are applied only once so Notepad++ will be launched at the beginning of all tests and will be closed after all tests finish.

JUnit also has a possibility to use @Before and @After decorators to allow some piece of code to fire before and after every test. Hence I try to avoid it to keep the tests as clear as possible: please take a look at finally clause. It allows to create very clear teardown without the decorator.
Hamcrest matchers are also very useful addition. I prefer them to standard JUnit assertions for both writing and logging clarity. I marked them yellow in the code above. The meaning is clear and when executing the test the following information is displayed when result is pass:

and the following for fail:

If you are interested, the full code is available HERE under GUI_automation_part2 branch.

Look and feel

I use IntelliJ Community edition to run the tests. Please take a look:


What next ?

This is definately not all we can do in area of testing automation. We have Java in use here so there is much, much more. I will cover it in my next articles.