Theory – friendly requirements
Speaking about quality assurance, there are well known patterns available of how to approach a project. We start with requirements (functional and non-functional), we think how to describe them, we consider behaviour driven development, specification by example and other techniques to achieve clear communication with customer. We create test cases using test design techniques to achieve the right coverage. We have some schedule to fit in, we can plan our actions.
This is all very useful for the project start (new feature start), this is very nice to have all these in the project. I call it proactive approach – I can act before any defect is planted to the code. I can set my defense even before anything was started. I have time.
Reality – angry user
How many times were you assigned to the project where things are set up in the right way? I think in most cases we are thrown in the middle of the sophisticated project which has many problems and where end users or client is complaining about many things. And they want you to act fast. How to act effectively in such a difficult environment?
I say – act reactively first. Group all the defects reported in the production and think of the most efficient actions so that when a defect is solved, not only it is retested, but also whole area of the bug is secured from quality point of view in the project as well. Such an approach improves project in places which are most important for the end user. When it is done, the user calms down and you can move balance to proactive approach, which will improve the project in the long term.
Let me give you some examples. They are real one.
User is complaining about not seeing data in the application in some places. It turns out, there is a problem with database connection as nice db guys forgot to tell your team they changed the connection string over the weekend. Well, it seems like it is not a defect for us, is it ? Other user is complaning of not being able to send anything from the application to some other one. It turns out, there is hardware malfunction of the remote application, again it doesn’t seem to be our problem. Finally the third one cannot save the data to the database – this time it seems we have typo in connection string in the production so let’s fix it.
At this point it is important to notice these are all configuration related issues and we can improve this area in the project by introducing configuration testing. We must be missing it by now. This can be easily added as component (unit/small) testing. All the configurations we have in the code base should be tested during each build to catch configuration strings’ typos as well as to be aware of the status of external systems to know in turn if the remote system is having problem or it was reconfigured for some reason. On top of it we can build configuration tool which would read production configuration and do the quick configuration check if in doubt if the production issues is related to connection problems or not.
Few users are complaining the application works very slowly. All the data is loading slowly and it causes business problem for them. It turns out, there is a defect in which timeout configuration setting – although set in configuration file – is not applied in the runtime which degrades the performance significantly.
This problem may be solved by introducing simple automatic comparison testing, where in the test environment a comparison of various configuration settings with runtime settings as displayed in the log files is made.
There are few defects with major priority reported, which are related to the core functionality of the application.
Well, it is easy one – actually the basics. The goal is to create the pyramid like set of test cases (many component tests, less integration tests and few system tests) which will cover this functionality as fast as possible to prevent defects from plaguing this area which is most important for end user.
Users again are complaining about performance, the description of the issue they give is very general and doesn’t help to narrow down the area where defect is hidden.
Log analysis is needed to find out how application characteristic in production looks like. It may occur, logging is not properly implemented and much information is missing or the other way around logging gives too much information. In such case developer team should start improving this to make log analysis easier in the future.
Developers are spending much time supporting the application as support team is asking them for help twice a day.
This is bad situation which slows down the project very much. Developers get frustrated as they cannot concentrate on 1 thing during work day. Project work is slowed down as instead of improving things or implementing new features devs are talking for hours with support or end users.
The best reaction in my opinion is to push as much activities to support as possible and also start working on things which will allow more things to be pushed away soon. Well, easy to say, harder to do… but possible. Again, we need to group problems which are reported by support:
– what does this button do? how to do this or that in application? – this is application knowledge question and should not be asked at all to developers; knowledge base should be started or even better expert system should be built to allow support to get the needed answers on their own
– the app is not working, there is no data in that window – if it turns out it is external system which doesn’t work, let support guys use configuration check tool (create it immediately if it is not available); they should be able to tell by themselves if there is problem within the app or else it is just external connection problem
– the app is not working, some issue appears – support guys should be able to understand what is the sequence of events which preceeded the issue; if they are not able to say anything maybe they need more clear logging, some memory usage history or more information in log files to be able to understand the state of the system quickly and also to immediately deliver the needed information to fix the defect for developer.
The proactive reality
I mentioned in the last section in my opinion we should use proactive approach anyway. However, it isn’t hard to think of a project which has problems with requirements. We can run into trouble here as well. What can we do not to rely on production feedback only and start pin pointing the defects before they reach end user? If we do not want to base on field feedback and requirements are not there or maybe are not complete at the same time, a very good method of improving proactive approach is comparison testing. I show it on the diagram – we have 2 red arrows showing missing or incomplete communication for requirements and feedback:
Testing by comparison is a powerful technique.
We compare previous and present version of the system or single component. Previous version can be yesterday build or last one which was released to production depending on our needs.
What can we be comparing actually?
– performance – this would be so called benchmarking to see if newest version is faster or slower
– functional behaviour – any discrepancies indicate an issue which in turn might mean a defect
– logs – differences (more logging, less logging, more warnings, errors etc.) may indicate a defect
– memory consumption – differences between consecutive test runs may indicate a memory leak defect (notice that we can observe it in comparison testing, even if no out of memory exception is thrown)
In my opinion it is good if we can act proactively but if the application causes problems for end users we should start with field defects analysis – reactive approach. We need to use production feedback in a smart way to get the large effect from small piece of information and small effort as well. After this, we should move to proactive part which often may be not so simple to handle in reality, but comparison testing improves this approach in significant way.
We are proactive unless production issues are reported again which make us react in a smart way again. In time, the application becomes more stable and reliable and only then can we finally do exactly what standard QA theory says.