Vast majority of testers are convinced, that testing of software product is a search of bugs. I am strongly convinced, that the main idea of testing process is to skip as less as possible bugs, than to find as much as possible bugs. At first glance, it seems, that there is not too much difference between these two ideas. But there is.
The testing process of large-scale software products is different from the same process for small software projects. Below I describe, what is the difference between two notions, mentioned above. Also, I share some good practices for testing of large-scale software projects.
What is the search of bugs?
What can be the main objective of testing in the case of search of bugs? Main objective is finding as much as possible bugs. And it is quite logical. For testers it is always good to find bugs, because bugs is visual representation of work done by them.
Which parts of software product will be tested in this case? First of all, the least stable parts of software will be tested. Frequently, these parts are less stable than other parts, because they less prioritize, but in this case it is not important, because the number of bugs is the main objective anyway.
What happens if there is bug, which cannot be easily reproduced? Reproducing of these bugs costs work time and it is better to find three bugs, which can be easier reproduced than current one, than spent huge amount of time, trying to reproduce the bug, even do not knowing whether it will be successful.
Which kind of tests will be done first? Certainly, the most non-standard, like division to zero, usage of different non-appropriate types of variable in certain fields, attempts to upload executable files instead of photo and etc. In the case of search of bugs, software testing process frequently starts not from testing of basic features of software, but attempts to find problematic moments in less standard part of software product, which can probably increase the number of founded bug.
I personally, cannot agree, that above-mentioned process has too much in common
with real testing of large-scale software product.
What is testing?
What is the main objective of testing? Main objective – skip as less as possible bugs, which are important for users. Less bugs are skipped, less customers complains – the more is the efficiency of testing process.
Which parts of software product will be tested in this case? In this case, testers will try to start with the most important parts and features of software for the end user. Even, if it is known that these parts are stable and worked successfully in previous release (for instance, in the case, when we have new release of existing software product, which has certain amount of new features, realized in it). Anyway, there can be problem with existing functionality, and in most of the cases, this functionality is not tested properly.
What happens if there is bug, which cannot be easily reproduced? If testers faced difficulties with reproducing certain bugs or misunderstanding of business-processes of end users or lack and technical requirements for project, it is important to research what is right and which way of is proper. In this case, testers can spend larger amount of time to reproduce and register bug in bug tracking system, but it will be more efficient, because it gives an understanding of architecture and details of software project to tester, which is very important in testing large-scale software products.
Which kind of tests will be done first? In this case basic functionality will be tested first. For example, testing of the most common case with standard parameters. And, only after completion of testing of basic features, new features must be tested. In my practice, I faced situations, in which after finding important bug in basic features, in the stage of problem solving, business-process was changed and hence, product is changed itself. In this case, testing of non-standard scenarios was not efficient and meant a loss of time.
Results of testing and search of bugs
If are considered the short period of time of testing and test of small software product, “search of bugs” approach can be efficient, because large number of bugs registered very fast.
However, if the project is large and testing process continuing long time, this approach is extremely non-efficient. Below are the proofs of this assumption:
• The percentage of missed bugs increasing very fast, because of the lack of knowledge of architecture of the software project.
• Developers try to understand the reasons of weird bugs which are appeared in very strange scenarios and which are very hard to reproduce.
• In production version of software there are very simple but very important bugs, which are easily seen by users.
• The amount of found bugs decrease dramatically.
How to change the approach?
In order to make testing of large-scale software projects will be efficient there are couple of simple rules, which must be followed during the whole testing process:
1. Good analysis of product and documentation of tests
All test-cases must be fully-documented, which made information more shareable in terms of using together by specialists from different departments (such as business analysts, developer and etc.) as well as in terms of sharing knowledge with new and existing team members.
2. Assessment of results and efficiency of testing process
Efficiency of testing must be considered and be measured during testing process. Also, the missed bugs and the reasons of missed bugs need to be discussed. Furthermore, obtaining data about other parameters, like coverage of functionality and code by tests and level of satisfaction of users also very important for large software projects.
3. Discussion of objectives of testing with team
There is no single goal of testing appropriate for any type of software products. Each project has its own goals in testing and clear understanding of these objectives is vitally important for teams of testers. Therefore, discussion of objectives of testing in teams is very useful practice.
4. Understanding of users and business-processes
There were software projects in my practice, when testers in team do not know too much about aims of products, its business model, end users, their environment and qualification. However, proper definition of all these parameters increases efficiency of testing and decreases probability of missed bugs. Understanding of business-processes of software gives opportunity for testers to understand the criticality of found bugs.
Summarizing all above-mentioned, I want to point out, that efficiency of testing of large-scale software products strictly depends on understanding the ideology of testing software and architecture of software tested.