When to stop testing
There are many activities in life which once started, have no criteria to end. For instance, a kid trying to memorize the spellings of palindrome words will be trapped in infinite loop :). Take the word “madam”. While cramming the word, he/she would consider the last m as the first m and will start over and over again….. madammadammadammadammadammadam….
Same is the case with software testing, once started will have no definite stop point. The reason is, it is an effort to enhance quality and achieve perfection. As you know, the perfection is only a dream. We can dream of perfection but an attempt to achieve it will keep us in infinite loop of attempts. So generally speaking, there is no end to testing. The testing stops when the time is over or the budget is exhausted. But still we need to find out some criteria to stop testing. In lay man terms, the testing should stop, when every feature has been thoroughly tested and all the possible hidden bugs has been found and fixed. but this is very vague and subjective statement. We need some concrete measurement way. I am listing few techniques to stop testing but again these are not hard and fast rules…It really depends upon the situation.
Setting and Achieving the goals.
We can set the goals of testing by explicitly witting the test cases in advances. Our goal would be to execute all the test cases before rolling the product into production. For instance,w may set out goals to achieve white box goals like /* make every list item as link to other pages*/
- statement coverage
- branch/decision coverage
- condition coverage
- condition-decision coverage
- multiple condition coverage
- Path coverage
- Basis Path CoverageOr the black box testing goals like
- Equivalence class partitioning (ECP)
- Boundary Value Analysis (BVA)
- state transition based testing
- use case based testing
- pairwise testing etc…
So we can write test cases in advance by following some techniques as few of them are mentioned above. But there is some problem with this approach as well. We, humans are generally goal oriented. We set a goal and then put all the efforts to achieve that. You can find the proof of this human nature on road in the mo. Every one would be keen to reach office on time and in case of road accidents, there would be few who would stop. It does not mean that we don’;t have sympathy for others. We do. But the problem is, we find it very difficult to deviate from the defined path.
Setting some threshold of bugs
You can set some threshold of bug count. Let’s say, for an application, you are getting 50 bugs per day in first week. In next week, the count drops to 30 bug per day. Let’s say, your threshold is less than 5 bugs per day. When you achieve that threshold, you can stop testing.But in this technique, we are really dependent on the testing team. if your testing team is competent enough then this technique may work. otherwise, the drop in bug count may also be due to laziness or incompetency of your testing team.
Comparing the marginal cost of bug
The concept of marginal cost comparison is very interesting. For mobile manufacturing company, which has produced 10,000 mobiles of some XYZ model. The cost to produce one mobile would be minimal. Because there would be more scrap left out than what would be required. but for testing firm, if you have found 1000 bugs in a software, the cost of finding one more bug would really high. It is like hunting the mosquitoes in your room, when you have killed nearly all of them. So first of all, we would calculate the marginal cost of the bug and then we would compare that cost with cost of shipping the product with bug. if the cost of retaining the bug in the product is more than the cost of finding that bug. The we would give up the testing hassle for that poor petty bug.
you can understand the concept by some real life analogies. The prisoners in jails are kept under tight security but the security cost of the prisoners should not exceed the cost or worth of the prisoners. Let it put in some other way. you have gold of worth .5 million. you want to save it from robbery and theft. How much can you invest for its security. Definitely not more than its worth. So this is bottom line. if the cost of retaining the bug is less than the cost of finding and fixing the bug, we will stop testing.
It is a decision that is made by consensus. If the members of testing team, agree to roll it to production as all of them have gut feeling that the product has attained reasonable and acceptable quality, we should stop testing. So, it is decision that does not follow any technical rule, rather it is made democratically respecting the remarks of all the testing team members.
They say, there are two rules to deal and reconcile with your boss.
- The boss is always right.
- if you think, your boss is a stupid non technical person and you are right, refer to rule 1.
So in any scenario, the boss is always right. There is a rationale behind this logic. That is, your boss has more exposure than what you have. He better knows the money matters of a company. if the company is going through crisis and does not have even budget to pay salaries to the employees. how can it prolong testing. or on the other way round, if the client of a particular project is underpaying you, how can you work for peanuts. So sometimes, the boss’ decision may look stupid, and the team may be thinking that boss is shipping the product full of bugs. But he knows and understands the gravity of the situation better than his subordinates.