Software testing is a relatively new field and has changed considerably in past few years. Computer science or programming courses in the universities do not teach software testing properly, and when I moved from development to testing in 2001, I was confused about it. I tried to learn from internet, books, forums and was not impressed with the information I got. I even did my certification (CSTE, if you are interested) but that wasn’t very useful either. During that time, I came across many interesting theories/concepts, and after working in the industry, I know they are not genuine and are myths. Unfortunately, some of these myths are still in practice and widespread.
In this post, I will explore some of the most prevalent software testing myths, why they are myths and what wrong these myths are doing to our profession?
1. Testers are Gatekeepers Of Quality – Nothing should be released to production before test organization/testers give their approval.
In many organizations, testers in the test team fight for this right. It makes test team empowered and to be honest, when I started my career I did think this is the right way. In reality, this view is extremely dangerous, for the team and product both. The test team is an information provider and provides information to stakeholders. It is up to them to act on the information test team provides.
When testers act as the gatekeeper, they become responsible for quality in the product. It gives them feeling that other than test team, no one else is concern about quality. It also increases pressure and sometimes creates a situation wherein testers are afraid to release the product because there might be one last remaining defect in the system that is not uncovered.
2. Complete testing is possible – If you plan correctly, it is possible to test software completely, identify and fix all the defects.
Many organizations feel that it is possible to test software completely and fix all the defects. Nothing can be further from the truth. No matter how much you test, complete testing is an illusion. Applications are becoming more and more sophisticated, and the possibility of testing all the features, under all the conditions are rare. When management is trapped in this belief, test team will become responsible for every defect. Also, if test team attempts to do complete testing, they will become the bottleneck. In reality, almost all the products have defects. The only difference is what kind of defects they have and how frequent is their occurrence. If you try hard, I am sure you can find defects in almost any software you use. Complete testing is not the solution for this.
3. Best Practices – Improving quality is simple & straight forward, just follow the best practices.
Best practices, standards, and processes are still a big myth. Not all the standards, procedures, and best practices work all the time. Sometimes they work and sometimes they don’t. There is nothing wrong in the practice as such, problem is in not identifying the context and problem before applying practices. Practices are practices, what makes them right or wrong is whether they are applied after considering the context or not. Using best practices is like applying hammer, if you do not consider the size of the nail and try to use the same hammer for all the nails, sometimes it will work and sometimes it will not. When test team starts implementing industry’s best practices without considering their project, timeline, skills, technology, environment, organization structure and many other aspects, they get frustrated because they do not get results they expected.
4. Certifications will make you better tester – So go and get CSTE, ISTQB…., etc. to become better tester / get a promotion.
When I started my career as a tester, I was in service industry and certifications were/are considered safe. There was a valid reason for that, because if you need more clients than boasting about a number of certified test professionals will increase their confidence. But from what I have seen, certification exams are very shallow in nature and does not reflect whether person who is getting the certification is a good tester or not. Certifications, in their current format, can be acquired by anyone who is prepared to study for a couple of weeks and it is highly unlikely that someone will become a good tester in a couple of weeks time. Certifications in its current format have created unnecessary pressure in the testing community to get certified, just because of peer pressure and client demand rather than as a benchmark for knowledge.
5. Test Automation is Silver Bullet – If something can be automated and you can automate – automate it.
Now do not get me wrong, I am a big fan of automation, but only where it add value. I have seen many engineering hours wasted on developing automation or frameworks for automation which is hardly used. Automation, without considering its ROI and effectiveness is just waste of time. Dr. James in his recent post has highlighted it nicely and made an excellent point that manual/automated testing should be considered only after good test design. This mentality of thinking test automation as a silver bullet, like many other myths is dangerous for our profession because of many reasons. Management sometimes can become extremely focused on the automation rather than improve quality. Remember, more automation will not improve quality. The right level of automation combined with required exploratory testing and good test design will certainly improve it.
6. Testing is the demonstration of Zero defect – Testing is completed for this product, and test team has signed off the product. This product does not have any defect now.
Whoever claim this, is, of course, wrong. It is impossible to claim that any product is defect free. Even after spending thousands of hours in testing, there will be one more combination which was not tested, one condition which was missed and for all we know that might surface in the production environment. If as a tester/manager you believe that zero defect is a possibility, you will feel responsible for any error which is uncovered in production. On the other hand, if you modify the sentence and say that for the combinations I have tried, environment and data I have used and scenarios I tested, there was no defect according to my understanding of the product.Also, the goal of testing is to uncover defects. As a tester, you can only find defects; you can not claim that this product is defect free.
7. All measurements are useful – Keep track of a number of test cases, how many of them are executed, how much automation is present, defect count.. and any other numbers you can think of.
When I started my career, we started preparing reports on the lines of – how many test cases are written, how many of them are executed, how many of them are automated. How many defects were found and so on. Week after week, we would send these reports without realizing that if additional information is not provided along with numbers, they do not convey any meaning. If these numbers become the primary consideration for management, quality will suffer. For example, if the number of defects is important, test team will start filing each and every issue, if the number of rejected defects / duplicate defects become necessary, test team will start spending a lot more time on errors before filing them or maybe will not file at all. Any measurement program should be approached with caution and should always provide clear meaning / summary for all the numbers.
8. Stable requirement and documentation are essential for any project. BTW development team is crap.
With Agile development, this myth is slowly going away, and we have realized that changes are inevitable. Rather than fighting changes, we now embrace them. It was different when I started and probably still in many organizations, changes are not welcome, requirements are similar to contractual obligation and documentation is the first thing test team ask for. Development and test-team work in their silos and communication between them are limited to finger pointing. It is impossible to have quality software coming out from such environment. Development and test team should work together to improve quality.
9. Limited time and resources are the main reasons for missed defects – As a test group, we are always pressed for time and hardly have any resources. We could have caught these errors, only if we had more time/resources.
I am sure many of us have heard this, and some of us even raised this as an issue, including me. It is true that time and resources are limited, but how many times defects are missed because of unavailability of resources and how many time defects are missed because of not utilizing resources properly and faulty test strategy and design. I have seen it many times that resources spend time in creating work which is not adding value in any way. It could be in the form of writing detailed test plans, test cases, writing automation suite which becomes shelf-ware, munching numbers to prepare a report for management and so on. Availability of time and resources is important, but also it is more important to have solid test strategy and design, prepared with application/project under test in mind.
10. Anyone can become a tester; they should have the ability to read and follow instructions, that’s it.Testing is not a creative job and does not require any special training or skills and that’s why there are not many professional testers around.
This is one of the most damaging myths of all, and to some extent, this is because of some of the practices we have seen in our industry. Manual scripted testing is probably closest to an unskilled job, which require minimal skill and probably very basic training. Everything else apart from that, right from test design, to test execution to automation is highly skilled and creative job and can be done effectively, only if you are skilled. Not considering testing as a skilled profession has done more harm to the testing community than any other myth. This myth is going away with the recognition of testing as a separate skill, exploratory testing practices, Agile and sensible test automation but still there is a long way to go.
This was my list of myths and is by no means is a complete list. Do leave your comments if you have observed/come across myths which are not covered here or if you do not agree with anything I said.