Category / Test Automation

Let’s use computer to execute tests for us.

by Grzegorz

JSoup defence for Selenium

Selenium problems

It may happen, Selenium causes problems. There can be at least 2 major things:

  • Selenium test case lasts very long

Selenium request is really expensive. Especially when executing the tests through hub-node (selenium grid) infrastructure. The request is sent from the build machine to selenium hub, then to selenium node, then on selenium node to the browser. The response travels all this way back. When the performance problem shows up during test case execution, the cause is often related to the fact Selenium sends too many requests. We often are not aware how often requests are sent.

  • the page which is to be automated is getting refreshed often

Sometimes we need to assert the page which is automatically refreshed in specific time interval. This problem causes notorious StaleElementReference exceptions. When refresh event happens after Selenium grabs WebElement but before it invokes a method on it the exception surfaces.

 The real life problem

Recently I was dealing with such a problems and was trying to think of a solution.

In my case I was iterating through the table to assert the specific cell in each row.

So the code was more less like this:

 

The performance was very poor and staleness problem was present in almost each run.

Notice, java is sending requests to web browser at all marked lines. Surprisingly, it is the case for every iteration in the loop as well!

When page refreshes during driver chain method, you will get stale element reference exception. The same thing will happen when page gets refreshed anywhere during loop execution. The list of web elements which is used during the test cannot be refreshed until loop is completed!

How to solve such a problem? The solution is either to try to catch the exception so that processing starts at the beginning of the refresh interval or to decrease number of Selenium requests to minimum and move processing to memory as much as possible. The first solution turned out to be impossible as the loop was lasting 3 times longer than page refresh rate…

Here comes the cavalry

JSoup (https://jsoup.org) is the ultimate solution for all such a problems. Not only it is great library extremely easy to use with great documentation and intuitive methods but also it allows to extremely smooth code refactor because of the fantastic feature it supports: CSS selectors.

Just take a look:

The table is extracted using Selenium and then the processing is passed to JSoup completely for the looping time:

  • JSoup creates document of the html table, which is kind of snapshot of the data present at the time document was created which assures data consistency
  • the document is then queried using CSS selectors – completely offline from Selenium point of view and entirely in memory
  • the result is converted back to Selenium WebElement to continue Selenium methods

Now, the web browser interaction is reduced to only 2 places.

The solution is staleness proof and significantly improves execution performance: one just needs to catch StaleElementReference exception when Selenium is in play:

The only thing to consider in this specific example is to decide if we can accept the situation page was refreshed after grabbing the table but before sending getLocation request. Notice, it is perfectly save if there is no page refresh at all.

As for performance, even using local web browser and very small table the difference is noticable (on selenium grid the difference is really huge, believe me!):

Sum up

If there is a problem with multiple Selenium requests which cause performance issue or are making tests unreliable because of StaleElementReference exception – switch to offline processing with JSoup. Just remember, you need to understand the number of Selenium requests in your code, the exact cause of staleness and the impact offline processing brings to your test case consistency.

 

by Grzegorz

Automatic test case generation for state transition diagrams (approach 1.0)

Approach 1.0

This article is left here for historical reason. Please read newest version of the idea which is described HERE.

Increase automatic test case generation

I was writing about 2 things in the past: state transition based testing and automatic testcase generation. This is actually about 2 complementary test design techniques: state transition diagrams and decision tables respectively (I do not want to write about details of these techniques now – this is a subject for separate post I hope to write in the future). In the latter post I showed how to automate test case generation for decision tables, the goal for today is to show how to start automation when diagram is the starting point.

Combinatorial nature of a problem can be expressed as decision table and can be translated into xml for TCases application to process it and produce output which contains optimal set of test cases (automatic testcase generation). However, the most general way to analyze application under test is the state transition diagram. I already showed how to use this technique in order to achieve the coverage but I showed only the manual approach. Still, we need automatic test case generation!

When diagram is in use, the trouble begins: how to process it automatically? How to generate set of test cases from a diagram? It was quite a while until I came up with some reasonable solution.

I recently thought I could try TCases for this purpose. Although this is meant to identify variables and its values, if transitions of the diagrams could be considered as variables and their dependencies were described in TCases xml input file, I could get valid set of transitions and each transition would be used at least once in basic coverage setting. 

Practical example

Create model

Let’s use the same problem as in state transition based testing. We want to test if Notepad is working when switching between tabs and changing text direction inside each of them as well as writing text in each of them. This is very simplified model but it is enough to ilustrate the concept. The state transition diagram looks like this:

TextDirectionSTDiagramDetailed

Now, it is required to translate it into XML representation which will be parsable for TCases (I was writing about TCases HERE). This is it:

INPUT is the state name, VAR is the transaction.

COMMAND in HAS elements contains domain language sentences which are executable after simple processing by domain language generator.

WHEN elements describe needed dependencies to allow only valid combinations of transactions.

EXPECTED in HAS elements shows we just assert if Notepad GUI is visible after each set of transactions is run.

There is one problem with this file: in line 16 we need to give all the sequence of transactions needed to reach SELECT2TAB as TAB_1_IS_SELECTED state has 2 outgoing transactions. This shows there is a disadvantage of modeling the diagram in this way if there are states using very many transactions.

Generate executable test case

After generator is run, the set of test cases is produced. Generator reference is

The link to the source code is shown at the end of this post if you are interested.

When using basic coverage which is 1-tuple coverage it will mean each transaction will be used at least once. Because each transaction is marked as TRUE or FALSE (decision about transaction is valid when dependencies are met) the set of transactions will contain both TRUE and FALSE: it means in the generated test case there can be all valid transactions but also part of them as well. This is 1-tuple coverage:

With generated test cases (tc3 is missing as it consists of FALSE values only and generator wisely skips such test cases):

Now, when creation process of test cases is automated it is very easy to increase the coverage. This is 2-tuple coverage:

With generated test cases:

Running the testcases

It is time to run the test cases. The generated test cases are just pasted into JUnit class:

And the class is run as shown here:

If curious, you can view all the code HERE under automatic-tc-generation-from-diagram branch.

Sum up

Even if not perfect this is a solution to automatically generate test cases from state transition diagram. Together with automatic test case generation for combinatorial problems described by decision tables it is very solid approach to quickly achieve optimal coverage and thus assure quality in the application under test.

by Grzegorz

More about the coverage – let’s get it right and fast automatically

Background

I was writing once about TCases software which allows to put pair-wise testing into practice. I said it was a giant leap towards the right coverage. In this short article I would like to go much further. This is not going to be another leap, it is going to be a flight 🙂

The problem

When dealing with a problem in QA practice it has often combinatorial nature. There are many possible combinations of either data, actions or other “inputs” which constitute the testing space. The testing space always tends to be infinite or at least extremely large as not only must one take into consideration the fact all the inputs have to be used in test runtime but also all the relations between them. Such a relations can only be tested when specific combinations of inputs will be used. Thus, the main problem to solve is how to choose those combinations. Well, this one is already solved here.

When testing combinations are known next problem arises quickly. There are always many combinations to test. There is huge work to do to create automated tests out of generated combinations not to mention manual testing. And there is even more work to do when system under test changes as the tests have to be adjusted accordingly. Let’s solve this problem.

The solution

Automation is the process which started quite long time ago. We are not testing manually, we have set of automated tests to run. We can use them on all the testing levels: small, medium and large tests are automated and used in development process continuously. However before we can run the test we have to write it. It takes time as it is still manual process nowadays. It is now time to move on and start creating test cases automatically!

It looks more less like this:

manual_design_time

old process

It is important to move to such a process:

automatic_design_time

new process

When speaking about combinatorial testing problem we have TCases at hand which utilizes pair-wise testing concept and generates the set of testcases to be created. After that, such a raw testcases need to be translated into executable test cases:

tc_generation_flow

automatic test cases design time

Example

There are domain specific languages in use at present. I wrote shortly about it here already. I am going to use internal domain language that is a language crafted from Java itself.

Let’s assume, we want to test the behaviour of Notepad++ application in terms of new document settings with regards to format, encoding and default language. These settings are available at: Settings->Preferences->New Document.

I am going to use all the available formats and 5 languages and encodings for the sake of simplicity (in reality, one needs to use all of them in the testing space of course).

We have 3 variables: format, encoding and default language. Each variable contains information related to testing space that is which values are possible and at the same time it contains information related to domain language (Has attributes). Domain language contains two kinds of information: the one related to GIVEN and WHEN clauses (command) and the one related to THEN clause (expected).

The testing space looks like this (input.xml):

So, for example when “format_isChecked” variable will have FORMAT_WINDOWS value, it will mean:

translated later into:

All the hashes and asterisks are used as placeholders only. It is much easier to write dsl related info when using external domain language like Cucumber. Dealing with internal one like in this example (Java) is more problematic because of braces and dots.

Anyway, each variable value will be translated into GIVEN/WHEN and THEN sentences and each variable value which belong to the same testcase will be concatenated with each other to create a single dsl test case.

To achieve this, we need to create output xml file with raw testcase which are not executable yet (they are manually executable though). I use 2-tuple coverage:

After TCases is run we get (output.xml):

The coverage is pair-wise (2-tuple) only but we already have 35 test cases! We are not afraid of high amount of required test cases anymore.

Now, we just need the last step and generate executable test cases. Here we go with the generator:

After generator is run with output.xml file we get this:

Which is the set of executable test cases. Now, just paste it into JUnit file:

 

Things you might want to see

If you want to see how the runtime looks like here it is:

And if you are interested in the code, you can get it from HERE. Use automatic_tc_generation branch.

As you can see we do not have to pay any attention to test cases anymore. We just focus on testing space that is variables which influence the behaviour of the system. The rest happens automatically.

Sum up

I showed here practical example for extremely heavy tests as you can see on the movie. I also used internal domain language which is not perfect for testcase generation. But this is valid approach for all testing levels. Component tests, integration tests, system tests – all of them can take advantage. If you have any testing framework which can have tests expressed in some kind of nice language, it is easy to write the generator. Once generator is in place, you can have hundreds of test cases to deliver optimal coverage.

This is a very important approach in my opinion and it really makes the difference when you do QA job. When system changes for some reason, you just need to concentrate on testing space to adjust it to new variables. Then, you just hit generate button. Documentation of the coverage is in place as well: it is clear what coverage is used. It is clear how much it can be increased or decreased.

Don’t write test cases, generate them – it means stop jumping and start to fly.

 

 

 

by Grzegorz

Groovy – universal scripting tool

I recently had a need of create some script in Windows environment. I knew about Groovy for some time so I decided to try it. In my opinion this is very strong alternative for Python and for Perl. Because of the fact it uses Java virtual machine, it can be easily launched on every platform where Java is supported. It may also be merged with Java classes so any code you have there somewhere can be used in Groovy script. But this is not all. The very interesting features I would like to show in this post are: ability to create nice domain specific language, handling of XML files (and other form of storing the data) and handling GUI interfaces via Swing library.

Domain specific language

I have already spoke few words about DSL HERE. Groovy has a nice ability to easily create your own language.

This application does… well I do not have to describe what it does – it is self explanatory:

The output of this application is:

It is not only the domain language we can construct but also short code inside. There are 4 files there and let me just show you filtering functionality for files:

You can see the full code HERE.

 

XML handling

This is the best and easiest way to process XML I have seen so far. I extend the functionality of the previous application so that it opens config.model.xml file (which is part of Notepad++ distribution) and then displays GUIconfig element’s attributes. Take a look, we have DSL part:

The output:

And the internals:

 

And that’s it. Finally there is an easy way of processing XML files. This is very frequent task to deal with such a format data is stored…

In line 8 file is parsed into variable which has the name as the root node of the XML document. Then, in line 10, one uses child element’s names to build a path. After iterating with “each” keyword we assert @name attribute and if it matches we iterate over attributes. Short, simple and clear.

It is important to mention,
there is the same way of working with JSON

(and that was a short poetry, did you notice?)

The full code is HERE.

And XML file I was using looks like this:

 

GUI

This is also easy to create simple GUI interface to have more rich tool by your hand. In this example I create a small application which will show the values for given elements and all its attributes. This also have auto completion/suggestion feature for more friendly user interface. The default behaviour is to open config.model.xml file and find all attributes of GUIConfigs.GUIConfig element with specific “name” attribute. File path, element path and attribute name are customizable.

The whole application is only in 1 file, so if you copy paste you could run it immediately:

The first part is the logic:

  • buildNodePath builds a node path out of user input; this shows how to navigate through xml document when user is to provide node names during runtime
  • quickList displays the attributes which match string entered by user (“App” will show //GUIConfigs/GUIConfig[@name=”AppPosition”] is available and when confirmed it will display the value and all the attributes of //GUIConfigs/GUIConfig element)
  • autoComplete populates dropdown
  • getAttributeSummaryOfElement displays all attributes and the value of the given element

The second part is the layout done by Swing in Groovy. There is very clear way of positioning items on the screen by using html like trs and tds.

You can view the code HERE.

The screenshot of the application (wordpad on the right side):

XML_explorer

The movie which shows the runtime:

Nice tool

This is nice tool worth reading about it more. There are small drawbacks from my point of view especially when mixing things with Java code. I often find myself writing some Javish Groovy or Groovish Java… still this is clearly an advantage as you can append existing Java code to your script. I think I like XML/JSON processing most, this is extremly useful. I also am impressed by details like this one:

It gives “c” in the output. Isn’t the simplicity impressing in Groovy ?

 

by Grzegorz

Get the right coverage

Test design techniques

Let’s think about quality control basics for a moment. In my opinion this is the most important thing to be able to design the right test case. No matter if this is automated or manual we need to get the confidence software we are working on has a very low probability of still unrevealed functional defects in the area covered by our test cases. We do not have to reinvent the wheel as we have already test design techniques there in place to help us achieve this goal. Because it is basics of the basics for QA engineer you know all of them and apply them in practice aren’t you… ? I can tell you basing on my interview experience in reality the majority of QA engineers heard about some of them but also majority is not applying them in practice (as described here). If you are by any chance in this notorious majority, I hope you read all this no to be part of this group of people anymore.

Pair wise testing

I would like to concentrate in this article on the most advanced test design technique in my opinion, or at least most interesting from my point of view which is pair-wise testing. The purpose of this technique is to deal with situation when number of combinations we have to test is too large. Because we have combinations almost everywhere this is extremly important thing to know. The of combinatorial testing problems are for example:

  • application setting page with many switches (we need to know if some combination of settings we choose doesn’t influence any of them – Notepad++ preferences),
  • software designed for many platforms (the combination is here array of operating systems – UNIX, mobile, Windows – and external software combinations – database vendors, database providers),
  • aplication which has REST or SOAP web service interface (number of available combination of input data – application accepts POST message in XML format, some of the elements are mandatory, some of them are optional)

The idea behind pair-wise technique is to focus on pairs instead of all combinations.
For example, let’s imagine we have 3 inputs where each of them accepts one letter at a time. 1. input accepts only letters (A,B), 2. (A,B) and 3. (A,B,C). We can easily write all combinations for such model (2x2x3=12 combinations):

1 => (A,B)
2 => (A,B)
3 => (A,B,C)

full coverage – all combinations
no 1 2 3
1 A A A
2 A A B
3 A A C
4 A B A
5 A B B
6 A B C
7 B A A
8 B A B
9 B A C
10 B B A
11 B B B
12 B B C

Of course, in such a case we do not need any special approach, we can test all of them. But let’s think of a situation each combination takes 1 week to execute or else that we have 3 inputs where range A-Z is accepted or when each input accepts more then one letter.
We can decrease the coverage for 100% (all combinations) to all pairs. Please notice 100% coverage here means actually all triplets. We are actually moving from all triplets to all pairs now:

Let’s enumerate all pair combinations as we are interested now only in pairs:

pairs listing
no 1 2 3
1 A A –
2 A B –
3 B A –
4 B B –
5 A – A
6 A – B
7 A – C
8 B – A
9 B – B
10 B – C
11 – A A
12 – A B
13 – A C
14 – B A
15 – B B
16 – B C

Let’s choose the subset of combinations which will have all the pairs listed above. Consider this:

reducing number of combinations
no Comb. 1 2 3 comment
1 AAA AA_ _AA A_A we don’t need this, we have these pairs in AAC, BAA and ABA
2 AAB AA_ _AB A_B we don’t need this, we have these pairs in AAC, BAB and ABB
3 AAC AA_ _AC A_C
4 ABA AB_ _BA A_A
5 ABB AB_ _BB A_B
6 ABC AB_ _BC A_C
7 BAA BA_ _AA B_A
8 BAB BA_ _AB B_B
9 BAC BA_ _AC B_C we don’t need this, we have these pairs in BAB, AAC and BBC
10 BBA BB_ _BA B_A we don’t need this, we have these pairs in BBC, ABA and BAA
11 BBB BB_ _BB B_B we don’t need this, we have these pairs in BBC, ABB and BAB
12 BBC BB_ _BC B_C

So, we can use now just 7 combinations out of 12 originally:

all pairs coverage
no Comb. 1 2 3 comment
1 AAC AA _AC A_C
2 ABA AB _BA A_A
3 ABB AB _BB A_B
4 ABC AB _BC A_C
5 BAA BA _AA B_A
6 BAB BA _AB B_B
7 BBC BB _BC B_C

Can we reduce number of combinations more? Yes, we can move from all pairs coverage to single value which would mean that we want to use every possible value for each input at least once and we do not care about any combinations at the same time:

single value coverage
no 1 2 3
1 A A A
2 B B B
3 A B C

In this set of 3 combinations input 1 uses A and B, 2 uses A and B and 3 uses A, B and C which is all that we need.
As you can see we have a nice theory, but we are not going to compute things manually, are we ?

TCases

Let’s use the software for the example from the previous section.
It is named TCases and it is located HERE. I will not be explaining the usage as there is excellent help on the page there (I tell you it is really excellent). It is enough to just say we need input file which is modelling the input and generator file which allows us to set actual coverage. The input file for the example shown above looks like this:

After we say we want to have single value coverage (which is called 1-tuple coverage):

We get the same result as we did manually:

It doesn’t matter 3. test case is different as the only thing which matters there is to use C for the 3. input.

Now, let’s repeat all pairs coverage (named 2-tuple coverage):

The result is:

There is slight difference as number of test cases here is 8 while it was 7 when done manually. This is because TCases by default doesn’t guarantee minimal set of pair combinations. We need to explicitly ask for it by using tcases-reducer (I must say I overlooked it initially – many thanks to Kerry Kimbrough for helping me with this). Looking at this result we can see we need to exclude AAA from this set as the pairs from AAA are also present in test cases id: 2 (AA_), 4 (_AA) and 7 (A_A). Let’s see what happens.
After running tcases-reducer generator file is modified:

After running tcases with new generator the result is surprising:

Wait a minute, we have only 6 combinations, how is this possible?
Let’s combine manual table I created previously with this one:

all pairs coverage
no Comb. 1 2 3 AUTO COMB 1 2 3
1 AAC AA_ _AC A_C  AAC  AA_  _AC  A_C
2 ABA AB_ _BA A_A  ABA  AB  _BA  A_A
3 ABB AB_ _BB A_B AAB AA_  _AB  A_B
4 ABC AB_ _BC A_C BBB  BB_  B_B  _BB
5 BAA BA_ _AA B_A  BAA  BA_  B_A  _AA
6 BAB BA_ _AB B_B BBC BB_  B_C  _BC
7 BBC BB_ _BC B_C

It turns out, it is important what is the sequence in which we are choosing a combination to be left out.
TCases found a way to have only 6 combinations and to cover all pairs ! Different combinations are marked green. As I pointed out there are 16 pairs total: my manual combination set was redundant by 5 pairs and computed combinations are redundant only by 2 pairs (take a look at yellow cells).

And finally let’s change the generator to use triplets which means 100% coverage in our example:

Result:

So it must be the same which is 12 combinations.
As you can see this application is working really nicely.
But what about real usage of this technique in real life testing problems? Let’s apply it!

Practice

Rarely can I find such a superb software like this. I am not sure if you are seeing the power of this tool yet. Let’s use it in practical example.
Let’s assume testing of Notepad++ “recent file history” is required.
The setting looks like this:

recent files history

recent files history

The input model for this testing problem can look like this:

Take a look at customSize_isSetTo – this is dependent variable which is related to customize_maximum_length value of configuration_isSetTo variable.
Now, the simplest test suite with 1-tuple coverage looks like this:

Custom size is set to NA when maximum length is not customized in the test case. This is really important to process dependent variables.
And 2-tuple coverage:

We can know decide if we have enough resources to process 1-tuple or 2-tuple coverage. We need to concentrate only on proper model input to have confidence we are achieving the right coverage. What is also important we have documented the way of creating test cases.
It is a giant leap towards the right coverage.

by Grzegorz

Statewise testing (GUI automation part IV)

Introduction

You must have heard about state transition tables. This is great test design technique to let us achieve the right coverage when testing with GUI application. We analyze the application from its states perspective to be able to adjust the coverage we can achieve according to resources we have available. In this part about GUI automation I will show how easy we can adapt the testing framework to use it.

Details

We have a framework at this point available which has a DSL layer and it looks very nice (as shown HERE). What we can test at this point (from test design technique point of view) is actually one test case decision table like this:

action TC 1
GIVEN().tab().isOpened(). X
WHEN().tab().isWritten(text). text = “text1”
AND().tab().contentIsCopied(). X
THEN().clipboardContent().shouldBe(equalTo(text)); X

However we need to be able to look on the application from state transition point of view, to be able to act basing on the state transition diagram. We would like to have an automation which can execute state transition diagram by walking the paths we choose.

Let’s look at the example of one – it focuses on the feature LTR/RTL which is text direction left to right and vice versa in Notepad++.

text direction feature in Notepad

text direction feature in Notepad

State transition diagram with text direction focus

State transition diagram with text direction focus

As you can see there is one main state which is one tab selection in Notepad++ (there is always some tab selected). There are also 2 substates which indicate if RTL or LTR is used. So it means when we start using the application we always start with 1 tab selected and LTR text direction.  Then, we can change the text direction or do any other transitions like createNewTab, selectTab or deleteTab. I do not want to write here all the state transition diagram details (I hope to create separate post on test design techniques some day) – let’s only say the basic coverage for this technique should be to execute each state and each transition at least once.
We would like to be able to transform this diagram into Java code and introduce one extra layer on top of we already have in the Sikuli testing framework and then run it.

Let’s do it!

Every of 3 states will be represented as separate class in Java:

Now we can navigate through the state transition diagram using IntelliJ suggestions and we can make any path we wish. As the result we can adjust the coverage to our needs.
The example test case based on the state transition diagram looks like this:

What we test here is we navigate through all the states and transitions at least once and at some point we execute assertTabIsWritable. By doing so, we test if the tab currently selected in the test flow is writable or not. Under the hood we use here Sikuli based testing framework which is described in previous posts.

As you can see we are using the existing framework to check if tab is writable. We are focusing only on this feature. We of course could be checking much more: like when executing createNewTab we could be checking if extra tab was created etc. but I skipped it for simplicity sake.

The video goes here:

You may also view the source code which is
HERE under GUI_automation_part4 branch.

Summary

This is just an example of the automation of tests which are basing on state transition diagram. The power of this approach is related to this test design technique. Firstly, if a requirement is described in this way we are ready for automatic testing. Secondly, in my opinion when such a diagrams which decribe specific funcionality exist in the project, it is possible to use them for example for purpose of retesting. If there is a new feature F1 in functionality A1 it is possible to have automated test for the A1 functionality by executing its state transition diagrams to check if nothing is spoiled there by F1.

The great GUI automation summary

So this would be all I wanted to present about GUI automation. This framework in my opinion is extremely flexible. This could be used for any application – not only GUI. This is just a matter of substitution of Sikuli with other thing like some web service self made framework for testing web service application. One could simplify it from programmatic point of view, one could add tests which would be testing framework itself (unit tests), or else improve the DSL layer. Still even such simple version is usable and shows the potential.
I hope that now the tank comparison from the 1. part (HERE) is more clear 🙂

by Grzegorz

Start speaking your own language (GUI automation III)

Introduction

At this point we KNOW HOW TO AUTOMATE and we know HOW TO AUTOMATE TESTING. Still although the way the tests are created seems to be very flexible, their content remains unclear especially for non developers. They look more like a code and they may require extra documentation to clear them out. What should we do to make it easy to understand for every party in the project, and speed up writing the tests at the same time? The advantage is at this point we have a custom framework so we can improve it any way we wish. Let’s move towards behaviour driven development then.

Behaviour driven development

You can read more about it HERE. In general it means focusing on behaviour of the system by describing it with help of domain specific language (dsl) that is a language which is on one hand easy to understand for everyone because of its similiarity to English and on the other hand is tightly related to the application itself. The main advantages in practice are: improved overall communication (we don’t have to ask what business analyst had on his mind, nor what is the specific test doing), automation speed up (the behaviour described in domain specific language can be directly executable, which means just after it is created it can be run by QA engineer or tester), documentation improvement (we do not have to create aditional documentation as the described behaviour of the system becomes documentation itself).
As you can read in Wikipedia the popular template for the dsl are GIVEN, WHEN, THEN words. Thinking about it from a perspective of automation engineer one can say GIVEN may be considered as a setup, WHEN is a body of a test and THEN is a assertion. Let’s use it in practice!

Notepad specific language

For interacting with Notepad we need specific language. There is no such a language yet so let’s create Notepad Specific Language – NSL.
We need words which are related to Notepad GUI entities like document, tab, bookmark and actions like open, write, close. Let’s look at the example below:

So if we follow the pattern from Wikipedia, we can say:

It looks much better now, doesn’t it ?

Test report can also speak NSL

After the test is run we need to review results. Often happens we need to know what actions were taken at specific point in time. Let’s look at the test ouput we get when appropriate DSL logging is present.

As we can see not only the testcase itself becomes very clear but also the logging done in NSL level when analyzed with application logs allows mapping each action done on system test level to the sequence of events which take place on single service and class levels (component level) at the given point in time. Of course this is easy now to do the reverse thing and tweak logging a little bit to allow copy paste of log output to allow instant replay of the logged actions. So if we just change the logging messages a bit so that all of them are Java methods, we will be able to copy paste them and execute instantly. Consider this:

 

More about test report

This is nice thing we can see the test run log in NSL, but this is even better to have the screenshots. This is good test proof and very good way to see what went wrong in case test produce error instead of pass or fail, especially when it is hard to tell from application logs what is the reason. We are using custom framework so we can have screenshots anyway we like. Let’s do it then.

In this example screenshot is made in BDD layer of THEN class just before hamcrest based assertion is made. Screenshot information is added to the log messages. Screenshot itself is made by Sikuli method capture which is using java.awt.Robot under the hood. The class is one of the 3 main mechanisms which drive Sikuli as you can learn HERE.

As you can see in the example, there is parameter passed to the method which allows for capturing not only the full screen but also any region which is used by Sikuli. As for previous posts I share the code I was telling you about HERE under GUI_automation_part3 branch.

The movie goes here:

Summary

There are new nice things I presented you here. BDD aproach constitutes efficient solution for quality control. However, we can do more, we can do better. As usual… I will cover this in my next part about GUI automation.

by Grzegorz

How to automate testing of GUI application ? (GUI automation II)

Let’s start testing

In the last ARTICLE I showed how to automate actions for Notepad++. It is time to apply this to get the real value for quality control of the application: test automation.
For this purpose we need something which will allow us to follow automation good practices namely create small independent tests with setups, teardowns and reports.

Details

Let’s look at JUnit with hamcrest matchers. This is universal solution used very often by developers for their unit testing.
The example test looks like this:

We have few interesting things here: the TestCases class has 1 test in this example. I call such a class a test set as it can contains many tests. We have a test set then with 1 test, with setup (@BeforeClass) and teardown (@AfterClass). Those are applied only once so Notepad++ will be launched at the beginning of all tests and will be closed after all tests finish.

JUnit also has a possibility to use @Before and @After decorators to allow some piece of code to fire before and after every test. Hence I try to avoid it to keep the tests as clear as possible: please take a look at finally clause. It allows to create very clear teardown without the decorator.
Hamcrest matchers are also very useful addition. I prefer them to standard JUnit assertions for both writing and logging clarity. I marked them yellow in the code above. The meaning is clear and when executing the test the following information is displayed when result is pass:

and the following for fail:

If you are interested, the full code is available HERE under GUI_automation_part2 branch.

Look and feel

I use IntelliJ Community edition to run the tests. Please take a look:


What next ?

This is definately not all we can do in area of testing automation. We have Java in use here so there is much, much more. I will cover it in my next articles.

by Grzegorz

How to automate GUI application ? (GUI automation I)

World of tanks

Is there something more diffcult to automate than functional system testing of GUI based application which is not web based one? There are few things I guess, for sure this would be system integration for GUI based applications which interact with each other. Anyway most of automation tasks are easier: command line interface, web service interface, web interface. For all these interfaces we can automaticaly interact with the system instantly; JMeter, Selenium can help with web service and web, it is very easy to programmatically read CLI as well.I faced the problem of GUI system testing automation when I once jumped into the project when there was an application with client side created in Java running on Windows. At first I reached for Quick Test Professional because it was available in the company instantly by a mean of a floating licence. I heard much about Mercury/HP product so I was sure this is good choice.My impressions? Well… would you like to go for a weekend trip with a tank? You will get to the destination point producing incredible noise and will consume like 300 litres of fuel. It will be hot inside and no room for more than 3 people I suppose. It will be extremely expensive ride. But you will eventually get there. And believe it or not, people do it – they drive tanks every day! Still, one advantage is clear: it can do most of the tasks you can think of in automation testing area, this is really universal vehicle. I almost forgot: you also need to be Visual Basic fan. You are, aren’t you ?

Go flexible

I just thought something more suitable has to exists and came across Sikuli. It looked promising at the beginning and after few months I still couldn’t find any significant disadvantages that would force me to change the tool to other one. It is working and producing results. The conclusion is, it is worth at least to try!

Actually there are 2 versions of Sikuli which are developed independently of each other: http://www.sikulix.com/ (http://www.sikuli.org/)  and http://slides.sikuli.org/  (present in Maven repository). When I refer to Sikuli I always means the first one.

The architecture of the solution looks like this. As you can see you can both use Jython and Java to write scripts. However the main effort of the page is focused on Jython API. I discourage you from staying with this API. There is also bundled editor which allows you to rapidly see how it works. Again, do not use this for extended period of time unless you just check image recognition (it is really good for the latter if one needs to adjust the pattern similarity). Try the editor, write simple Jython scripts and leave it. This is really a flexible solution but in my opinion you have to switch to Java.

Jython vs Java with Sikuli

Why is that, I discourage from using Jython? It is a nice object oriented scripting language interpreted to byte code which is run on Java virtual machine. Well, here we go: “interpreted”. Every single run of the test requires interpretation. And it takes time which we would like to use for something more useful than waiting. Let alone my regression suite I was using at work was starting nearly 20 seconds! One can also say Jython is more readable than Java. I can agree it is cleaner but there is a price you pay which in my opinion is just too high: you loose possibility to integrate seamlessly with your Java project (if this is your technology), you will make developers and managers be more reluctant to start using the solution, you will have problems when trying to use some BDD framework on top of it. When your test become more complicated you will have to reach out to Java libraries anyway like I had to do when I was writing test case which included activities over ssh. There are minor issues with Jython as well like for example running Sikuli on secondary display while in Java it works perfect. Anyway, if you know Jython only and have Java nightmares stick to Jython, otherwise go for Java. The flexibility which comes along is awesome in my opinion.

Simple example

Let’s start with the simple example. This is fully functional example demonstrating the usage of Sikuli which will run if you have Sikuli and Notepad++ available on your computer. In this example Notepad++ is started, cursor is moved to the center of Notepad++ window and after 1 second pause text is written to the tab. I marked lines which are related to Sikuli libraries with yellow colour.

Diving into details

The main idea that drives Sikuli is this is image based concept. That is one needs many small screenshots of the application to let this testing framework run. Thus it operates more on user level of abstraction and it is closer to the human perception.

The other important point is it is not a “record-replay” tool. You do not record anything, you just write tests like you would write any regular application. By that mean you are test driven developmnent (TDD) and (behaviour driven development) BDD ready!
Let us look closer at how we can apply Sikuli to automate popular GUI Notepad++ application in Windows.
I am going to use version 6.8.3 in Windows 7 Home Premium version.

The main concept

The main concept can be illustrated by this drawing:

idea

The framework concept

The idea behind the framework is to be able to use meaningful “pieces” which use Sikuli in the background to simulate any action user can do in the application. They should have names close to english language, should present consistent grammar and possibly not have any parameters so that no free text is entered by the user.

 

As you can see it is very clear which user actions are simulated.

The main idea to use Sikuli is that when it starts, the framework should recognize the main window region and smaller regions so that images can be found quickly as well as to provide maintainability of the scope and context for actions in the code properly. Each region is handled by controller

Class overview

From the code point of view we should remember about reasonable design to allow easy development of the framework as well as its maintanance.

If you are interested in details please take a look yourself at the source code. I am sharing it here under GUI_automation_part1 branch.

Look and feel

Please watch the movie which shows the actual run. One can see red rectangles there which show the regions Sikuli is using.

Short summary

In the article I showed you how to simulate user actions on GUI which is not web based one. It was much about Java however this is the price we have to pay to have the exact testing framework we need. In the real project a help from developer could be required to make the framework correct from coding point of view.
Please note I was just showing the possibility of simulating user actions. I was calling the framework “testing framework” few times but I did not say anything about how to write automated test cases, nor about how to run them. This would be too much for the single article. Still I plan to continue the subject and come back to this in the near future. Stay tuned…